[
https://issues.apache.org/jira/browse/TAJO-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730515#comment-14730515
]
ASF GitHub Bot commented on TAJO-1493:
--------------------------------------
Github user hyunsik commented on a diff in the pull request:
https://github.com/apache/tajo/pull/624#discussion_r38731587
--- Diff:
tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
---
@@ -1027,6 +1081,136 @@ public GetTablePartitionsResponse
getAllPartitions(RpcController controller, Nul
}
@Override
+ public GetPartitionsResponse getPartitionsByAlgebra(RpcController
controller,
+ PartitionsByAlgebraProto request) throws ServiceException {
+ String dbName = request.getDatabaseName();
+ String tbName = request.getTableName();
+
+ try {
+ // linked meta data do not support partition.
+ // So, the request that wants to get partitions in this db will be
failed.
+ if (linkedMetadataManager.existsDatabase(dbName)) {
+ return
GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build();
+ }
+ } catch (Throwable t) {
+ printStackTraceIfError(LOG, t);
+ return GetPartitionsResponse.newBuilder()
+ .setState(returnError(t))
+ .build();
+ }
+
+ if (metaDictionary.isSystemDatabase(dbName)) {
+ return
GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build();
+ }
+
+ rlock.lock();
+ try {
+ boolean contain;
+
+ contain = store.existDatabase(dbName);
+ if (contain) {
+ contain = store.existTable(dbName, tbName);
+ if (contain) {
+
+ if (store.existPartitionMethod(dbName, tbName)) {
+ GetPartitionsResponse.Builder builder =
GetPartitionsResponse.newBuilder();
+ List<PartitionDescProto> partitions =
store.getPartitionsByAlgebra(request);
+ builder.addAllPartition(partitions);
+ builder.setState(OK);
+ return builder.build();
+ } else {
+ return GetPartitionsResponse.newBuilder()
+ .setState(errUndefinedPartitionMethod(tbName))
+ .build();
+ }
+ } else {
+ return GetPartitionsResponse.newBuilder()
+ .setState(errUndefinedTable(tbName))
+ .build();
+ }
+ } else {
+ return GetPartitionsResponse.newBuilder()
+ .setState(errUndefinedDatabase(dbName))
+ .build();
+ }
+ } catch (Throwable t) {
+ printStackTraceIfError(LOG, t);
+
+ return GetPartitionsResponse.newBuilder()
+ .setState(returnError(t))
+ .build();
+
+ } finally {
+ rlock.unlock();
+ }
+ }
+
+ @Override
+ public GetPartitionsResponse getPartitionsByDirectSql(RpcController
controller,
+
PartitionsByDirectSqlProto request) throws ServiceException {
+ String dbName = request.getDatabaseName();
+ String tbName = request.getTableName();
+
+ try {
+ // linked meta data do not support partition.
+ // So, the request that wants to get partitions in this db will be
failed.
+ if (linkedMetadataManager.existsDatabase(dbName)) {
+ return
GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build();
+ }
+ } catch (Throwable t) {
+ printStackTraceIfError(LOG, t);
+ return GetPartitionsResponse.newBuilder()
+ .setState(returnError(t))
+ .build();
+ }
+
+ if (metaDictionary.isSystemDatabase(dbName)) {
+ return
GetPartitionsResponse.newBuilder().setState(errUndefinedPartitionMethod(tbName)).build();
+ }
+
+ rlock.lock();
+ try {
+ boolean contain;
+
+ contain = store.existDatabase(dbName);
+ if (contain) {
--- End diff --
These tests can be improved like other methods. Please refer to
existsDatabase() method.
> Add a method to get partition directories with filter conditions.
> -----------------------------------------------------------------
>
> Key: TAJO-1493
> URL: https://issues.apache.org/jira/browse/TAJO-1493
> Project: Tajo
> Issue Type: Sub-task
> Components: Catalog
> Reporter: Jaehwa Jung
> Assignee: Jaehwa Jung
> Fix For: 0.11.0, 0.12.0
>
> Attachments: TAJO-1493.patch, TAJO-1493_2.patch, TAJO-1493_3.patch
>
>
> Currently, PartitionedTableRewriter take a look into partition directories
> for rewriting filter conditions. It get all sub directories of table path
> because catalog doesn’t provide partition directories. But if there are lots
> of sub directories on HDFS, such as, more than 10,000 directories, it might
> be cause overload to NameNode. Thus, CatalogStore need to provide partition
> directories for specified filter conditions. I designed new method to
> CatalogStore as follows:
> * method name: getPartitionsWithConditionFilters
> * first parameter: database name
> * second parameter: table name
> * third parameter: where clause (included target column name and partition
> value)
> * return values:
> List<org.apache.tajo.catalog.proto.CatalogProtos.TablePartitionProto>
> * description: It scan right partition directories on CatalogStore with where
> caluse.
> For examples, users set parameters as following:
> ** first parameter: default
> ** second parameter: table1
> ** third parameter: COLUMN_NAME = 'col1' AND PARTITION_VALUE = '3
> In the previous cases, this method will create select clause as follows.
> {code:xml}
> SELECT DISTINCT A.PATH
> FROM PARTITIONS A, (
> SELECT B.PARTITION_ID
> FROM PARTITION_KEYS B
> WHERE B.PARTITION_ID > 0
> AND (
> COLUMN_NAME = 'col1' AND PARTITION_VALUE = '3'
> )
> ) B
> WHERE A.PARTITION_ID > 0
> AND A.TID = ${table_id}
> AND A.PARTITION_ID = B.PARTITION_ID
> {code}
> At the first time, I considered to use EvalNode instead of where clause. But
> I can’t use it because of recursive related problems between tajo-catalog
> module and tajo-plan module. So, I’ll implement utility class to convert
> EvalNode to SQL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
