[ https://issues.apache.org/jira/browse/TAJO-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15023620#comment-15023620 ]
ASF GitHub Bot commented on TAJO-1952: -------------------------------------- Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/846#discussion_r45691070 --- Diff: tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java --- @@ -42,13 +45,13 @@ import org.apache.tajo.storage.Tuple; import org.apache.tajo.storage.VTuple; import org.apache.tajo.util.StringUtils; +import org.apache.tajo.util.TUtil; --- End diff -- unused > Implement PartitionedFileFragment > --------------------------------- > > Key: TAJO-1952 > URL: https://issues.apache.org/jira/browse/TAJO-1952 > Project: Tajo > Issue Type: Improvement > Components: Planner/Optimizer, Storage > Reporter: Jaehwa Jung > Assignee: Jaehwa Jung > Fix For: 0.12.0, 0.11.1 > > Attachments: TAJO-1952.patch > > > Currently, PartitionedTableScanNode contains the list of partitions and it > seems to me that the list has some problems as following: > 1. Duplicate Informs: Task contains Fragment which specify target directory > or target file for scanning. A path of partition lists already would write to > Fragment. > 2. Network Resource: When scanning lost of partition, it will occupy network > resource, for example, several hundred kilobytes or more. It looks like an > unnecessary resource because Fragment already has the path of partitions. > I want to improve above problems by implementing new Fragment called > PartitionedFileFragment. Currently, I'm planning the implementation as > following: > * PartitionedFileFragment will borrow FileFragment and it contains the > partition path and the partition key values. > * Remove the path array of partitions from PartitionedTableScanNode. > * Implement a method for getting filtered partition directories in > FileTableSpace. > * Implement a method for making PartitionedFileFragment array. > * Before making splits, call above method and use it for making splits. -- This message was sent by Atlassian JIRA (v6.3.4#6332)