[jira] [Updated] (HIVE-4246) Implement predicate pushdown for ORC
[ https://issues.apache.org/jira/browse/HIVE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-4246: - Resolution: Fixed Fix Version/s: 0.12.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks Owen! Implement predicate pushdown for ORC Key: HIVE-4246 URL: https://issues.apache.org/jira/browse/HIVE-4246 Project: Hive Issue Type: New Feature Components: File Formats Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.12.0 Attachments: HIVE-4246.D11415.1.patch, HIVE-4246.D11415.2.patch, HIVE-4246.D11415.3.patch, HIVE-4246.D11415.3.patch, HIVE-4246.D11415.4.patch, HIVE-4246.D11415.5.patch, HIVE-4246.D11415.6.patch By using the push down predicates from the table scan operator, ORC can skip over 10,000 rows at a time that won't satisfy the predicate. This will help a lot, especially if the file is sorted by the column that is used in the predicate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4246) Implement predicate pushdown for ORC
[ https://issues.apache.org/jira/browse/HIVE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4246: -- Attachment: HIVE-4246.D11415.5.patch omalley updated the revision HIVE-4246 [jira] Implement predicate pushdown for ORC. updated expected test results Reviewers: hagleitn, JIRA REVISION DETAIL https://reviews.facebook.net/D11415 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D11415?vs=37767id=37875#toc BRANCH h-4246 ARCANIST PROJECT hive AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/BitFieldReader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthByteReader.java ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgument.java ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitFieldReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitPack.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInStream.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestIntegerCompressionReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRecordReaderImpl.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthByteReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthIntegerReader.java ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java ql/src/test/results/compiler/plan/case_sensitivity.q.xml ql/src/test/results/compiler/plan/cast1.q.xml ql/src/test/results/compiler/plan/groupby1.q.xml ql/src/test/results/compiler/plan/groupby2.q.xml ql/src/test/results/compiler/plan/groupby3.q.xml ql/src/test/results/compiler/plan/groupby4.q.xml ql/src/test/results/compiler/plan/groupby5.q.xml ql/src/test/results/compiler/plan/groupby6.q.xml ql/src/test/results/compiler/plan/input1.q.xml ql/src/test/results/compiler/plan/input2.q.xml ql/src/test/results/compiler/plan/input20.q.xml ql/src/test/results/compiler/plan/input3.q.xml ql/src/test/results/compiler/plan/input4.q.xml ql/src/test/results/compiler/plan/input5.q.xml ql/src/test/results/compiler/plan/input6.q.xml ql/src/test/results/compiler/plan/input7.q.xml ql/src/test/results/compiler/plan/input8.q.xml ql/src/test/results/compiler/plan/input9.q.xml ql/src/test/results/compiler/plan/input_part1.q.xml ql/src/test/results/compiler/plan/input_testsequencefile.q.xml ql/src/test/results/compiler/plan/input_testxpath.q.xml ql/src/test/results/compiler/plan/input_testxpath2.q.xml ql/src/test/results/compiler/plan/join1.q.xml ql/src/test/results/compiler/plan/join2.q.xml ql/src/test/results/compiler/plan/join3.q.xml ql/src/test/results/compiler/plan/join4.q.xml ql/src/test/results/compiler/plan/join5.q.xml ql/src/test/results/compiler/plan/join6.q.xml ql/src/test/results/compiler/plan/join7.q.xml ql/src/test/results/compiler/plan/join8.q.xml ql/src/test/results/compiler/plan/sample1.q.xml ql/src/test/results/compiler/plan/sample2.q.xml ql/src/test/results/compiler/plan/sample3.q.xml ql/src/test/results/compiler/plan/sample4.q.xml ql/src/test/results/compiler/plan/sample5.q.xml ql/src/test/results/compiler/plan/sample6.q.xml ql/src/test/results/compiler/plan/sample7.q.xml ql/src/test/results/compiler/plan/subq.q.xml ql/src/test/results/compiler/plan/udf1.q.xml ql/src/test/results/compiler/plan/udf4.q.xml ql/src/test/results/compiler/plan/udf6.q.xml ql/src/test/results/compiler/plan/udf_case.q.xml ql/src/test/results/compiler/plan/udf_when.q.xml ql/src/test/results/compiler/plan/union.q.xml serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java To: JIRA, hagleitn, omalley Cc: hagleitn Implement predicate pushdown for ORC Key: HIVE-4246 URL: https://issues.apache.org/jira/browse/HIVE-4246 Project: Hive Issue Type: New Feature Components: File Formats Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-4246.D11415.1.patch, HIVE-4246.D11415.2.patch, HIVE-4246.D11415.3.patch, HIVE-4246.D11415.3.patch, HIVE-4246.D11415.4.patch, HIVE-4246.D11415.5.patch By using the push down
[jira] [Updated] (HIVE-4246) Implement predicate pushdown for ORC
[ https://issues.apache.org/jira/browse/HIVE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4246: -- Attachment: HIVE-4246.D11415.6.patch omalley updated the revision HIVE-4246 [jira] Implement predicate pushdown for ORC. resubmitting for jenkins Reviewers: hagleitn, JIRA REVISION DETAIL https://reviews.facebook.net/D11415 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D11415?vs=37875id=37941#toc BRANCH h-4246 ARCANIST PROJECT hive AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/BitFieldReader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthByteReader.java ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgument.java ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitFieldReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitPack.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInStream.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestIntegerCompressionReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRecordReaderImpl.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthByteReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthIntegerReader.java ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java ql/src/test/results/compiler/plan/case_sensitivity.q.xml ql/src/test/results/compiler/plan/cast1.q.xml ql/src/test/results/compiler/plan/groupby1.q.xml ql/src/test/results/compiler/plan/groupby2.q.xml ql/src/test/results/compiler/plan/groupby3.q.xml ql/src/test/results/compiler/plan/groupby4.q.xml ql/src/test/results/compiler/plan/groupby5.q.xml ql/src/test/results/compiler/plan/groupby6.q.xml ql/src/test/results/compiler/plan/input1.q.xml ql/src/test/results/compiler/plan/input2.q.xml ql/src/test/results/compiler/plan/input20.q.xml ql/src/test/results/compiler/plan/input3.q.xml ql/src/test/results/compiler/plan/input4.q.xml ql/src/test/results/compiler/plan/input5.q.xml ql/src/test/results/compiler/plan/input6.q.xml ql/src/test/results/compiler/plan/input7.q.xml ql/src/test/results/compiler/plan/input8.q.xml ql/src/test/results/compiler/plan/input9.q.xml ql/src/test/results/compiler/plan/input_part1.q.xml ql/src/test/results/compiler/plan/input_testsequencefile.q.xml ql/src/test/results/compiler/plan/input_testxpath.q.xml ql/src/test/results/compiler/plan/input_testxpath2.q.xml ql/src/test/results/compiler/plan/join1.q.xml ql/src/test/results/compiler/plan/join2.q.xml ql/src/test/results/compiler/plan/join3.q.xml ql/src/test/results/compiler/plan/join4.q.xml ql/src/test/results/compiler/plan/join5.q.xml ql/src/test/results/compiler/plan/join6.q.xml ql/src/test/results/compiler/plan/join7.q.xml ql/src/test/results/compiler/plan/join8.q.xml ql/src/test/results/compiler/plan/sample1.q.xml ql/src/test/results/compiler/plan/sample2.q.xml ql/src/test/results/compiler/plan/sample3.q.xml ql/src/test/results/compiler/plan/sample4.q.xml ql/src/test/results/compiler/plan/sample5.q.xml ql/src/test/results/compiler/plan/sample6.q.xml ql/src/test/results/compiler/plan/sample7.q.xml ql/src/test/results/compiler/plan/subq.q.xml ql/src/test/results/compiler/plan/udf1.q.xml ql/src/test/results/compiler/plan/udf4.q.xml ql/src/test/results/compiler/plan/udf6.q.xml ql/src/test/results/compiler/plan/udf_case.q.xml ql/src/test/results/compiler/plan/udf_when.q.xml ql/src/test/results/compiler/plan/union.q.xml serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java To: JIRA, hagleitn, omalley Cc: hagleitn Implement predicate pushdown for ORC Key: HIVE-4246 URL: https://issues.apache.org/jira/browse/HIVE-4246 Project: Hive Issue Type: New Feature Components: File Formats Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-4246.D11415.1.patch, HIVE-4246.D11415.2.patch, HIVE-4246.D11415.3.patch, HIVE-4246.D11415.3.patch, HIVE-4246.D11415.4.patch, HIVE-4246.D11415.5.patch, HIVE-4246.D11415.6.patch By using
[jira] [Updated] (HIVE-4246) Implement predicate pushdown for ORC
[ https://issues.apache.org/jira/browse/HIVE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4246: -- Attachment: HIVE-4246.D11415.4.patch omalley updated the revision HIVE-4246 [jira] Implement predicate pushdown for ORC. updated the parse trees Reviewers: hagleitn, JIRA REVISION DETAIL https://reviews.facebook.net/D11415 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D11415?vs=37635id=37767#toc BRANCH h-4246 ARCANIST PROJECT hive AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/BitFieldReader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthByteReader.java ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgument.java ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitFieldReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitPack.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInStream.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestIntegerCompressionReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRecordReaderImpl.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthByteReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthIntegerReader.java ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java ql/src/test/results/compiler/plan/case_sensitivity.q.xml ql/src/test/results/compiler/plan/cast1.q.xml ql/src/test/results/compiler/plan/groupby1.q.xml ql/src/test/results/compiler/plan/groupby2.q.xml ql/src/test/results/compiler/plan/groupby3.q.xml ql/src/test/results/compiler/plan/groupby4.q.xml ql/src/test/results/compiler/plan/groupby5.q.xml ql/src/test/results/compiler/plan/groupby6.q.xml ql/src/test/results/compiler/plan/input1.q.xml ql/src/test/results/compiler/plan/input2.q.xml ql/src/test/results/compiler/plan/input20.q.xml ql/src/test/results/compiler/plan/input3.q.xml ql/src/test/results/compiler/plan/input4.q.xml ql/src/test/results/compiler/plan/input5.q.xml ql/src/test/results/compiler/plan/input6.q.xml ql/src/test/results/compiler/plan/input7.q.xml ql/src/test/results/compiler/plan/input8.q.xml ql/src/test/results/compiler/plan/input9.q.xml ql/src/test/results/compiler/plan/input_part1.q.xml ql/src/test/results/compiler/plan/input_testsequencefile.q.xml ql/src/test/results/compiler/plan/input_testxpath.q.xml ql/src/test/results/compiler/plan/input_testxpath2.q.xml ql/src/test/results/compiler/plan/join1.q.xml ql/src/test/results/compiler/plan/join2.q.xml ql/src/test/results/compiler/plan/join3.q.xml ql/src/test/results/compiler/plan/join4.q.xml ql/src/test/results/compiler/plan/join5.q.xml ql/src/test/results/compiler/plan/join6.q.xml ql/src/test/results/compiler/plan/join7.q.xml ql/src/test/results/compiler/plan/join8.q.xml ql/src/test/results/compiler/plan/sample1.q.xml ql/src/test/results/compiler/plan/sample2.q.xml ql/src/test/results/compiler/plan/sample3.q.xml ql/src/test/results/compiler/plan/sample4.q.xml ql/src/test/results/compiler/plan/sample5.q.xml ql/src/test/results/compiler/plan/sample6.q.xml ql/src/test/results/compiler/plan/sample7.q.xml ql/src/test/results/compiler/plan/subq.q.xml ql/src/test/results/compiler/plan/udf1.q.xml ql/src/test/results/compiler/plan/udf4.q.xml ql/src/test/results/compiler/plan/udf6.q.xml ql/src/test/results/compiler/plan/udf_case.q.xml ql/src/test/results/compiler/plan/udf_when.q.xml ql/src/test/results/compiler/plan/union.q.xml serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java To: JIRA, hagleitn, omalley Cc: hagleitn Implement predicate pushdown for ORC Key: HIVE-4246 URL: https://issues.apache.org/jira/browse/HIVE-4246 Project: Hive Issue Type: New Feature Components: File Formats Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-4246.D11415.1.patch, HIVE-4246.D11415.2.patch, HIVE-4246.D11415.3.patch, HIVE-4246.D11415.3.patch, HIVE-4246.D11415.4.patch By using the push down predicates from the table scan operator
[jira] [Updated] (HIVE-4246) Implement predicate pushdown for ORC
[ https://issues.apache.org/jira/browse/HIVE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4246: -- Attachment: HIVE-4246.D11415.3.patch omalley updated the revision HIVE-4246 [jira] Implement predicate pushdown for ORC. Rebased the patch to the current trunk. Reviewers: hagleitn, JIRA REVISION DETAIL https://reviews.facebook.net/D11415 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D11415?vs=37533id=37635#toc BRANCH h-4246 ARCANIST PROJECT hive AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/BitFieldReader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthByteReader.java ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgument.java ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitFieldReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitPack.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInStream.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestIntegerCompressionReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRecordReaderImpl.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthByteReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthIntegerReader.java ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java To: JIRA, hagleitn, omalley Cc: hagleitn Implement predicate pushdown for ORC Key: HIVE-4246 URL: https://issues.apache.org/jira/browse/HIVE-4246 Project: Hive Issue Type: New Feature Components: File Formats Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-4246.D11415.1.patch, HIVE-4246.D11415.2.patch, HIVE-4246.D11415.3.patch, HIVE-4246.D11415.3.patch By using the push down predicates from the table scan operator, ORC can skip over 10,000 rows at a time that won't satisfy the predicate. This will help a lot, especially if the file is sorted by the column that is used in the predicate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4246) Implement predicate pushdown for ORC
[ https://issues.apache.org/jira/browse/HIVE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4246: -- Attachment: HIVE-4246.D11415.3.patch omalley updated the revision HIVE-4246 [jira] Implement predicate pushdown for ORC. Rebased the patch to the current trunk. Reviewers: hagleitn, JIRA REVISION DETAIL https://reviews.facebook.net/D11415 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D11415?vs=37533id=37635#toc BRANCH h-4246 ARCANIST PROJECT hive AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/BitFieldReader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthByteReader.java ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgument.java ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitFieldReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitPack.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInStream.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestIntegerCompressionReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRecordReaderImpl.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthByteReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthIntegerReader.java ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java To: JIRA, hagleitn, omalley Cc: hagleitn Implement predicate pushdown for ORC Key: HIVE-4246 URL: https://issues.apache.org/jira/browse/HIVE-4246 Project: Hive Issue Type: New Feature Components: File Formats Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-4246.D11415.1.patch, HIVE-4246.D11415.2.patch, HIVE-4246.D11415.3.patch, HIVE-4246.D11415.3.patch By using the push down predicates from the table scan operator, ORC can skip over 10,000 rows at a time that won't satisfy the predicate. This will help a lot, especially if the file is sorted by the column that is used in the predicate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4246) Implement predicate pushdown for ORC
[ https://issues.apache.org/jira/browse/HIVE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4246: -- Attachment: HIVE-4246.D11415.2.patch omalley updated the revision HIVE-4246 [jira] Implement predicate pushdown for ORC. Addressed Gunther's comments. Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D11415 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D11415?vs=34977id=37533#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/BitFieldReader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthByteReader.java ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitFieldReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInStream.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRecordReaderImpl.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthByteReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthIntegerReader.java ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java To: JIRA, omalley Cc: hagleitn Implement predicate pushdown for ORC Key: HIVE-4246 URL: https://issues.apache.org/jira/browse/HIVE-4246 Project: Hive Issue Type: New Feature Components: File Formats Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-4246.D11415.1.patch, HIVE-4246.D11415.2.patch By using the push down predicates from the table scan operator, ORC can skip over 10,000 rows at a time that won't satisfy the predicate. This will help a lot, especially if the file is sorted by the column that is used in the predicate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4246) Implement predicate pushdown for ORC
[ https://issues.apache.org/jira/browse/HIVE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4246: -- Attachment: HIVE-4246.D11415.1.patch omalley requested code review of HIVE-4246 [jira] Implement predicate pushdown for ORC. Reviewers: JIRA Record Reader changes By using the push down predicates from the table scan operator, ORC can skip over 10,000 rows at a time that won't satisfy the predicate. This will help a lot, especially if the file is sorted by the column that is used in the predicate. TEST PLAN unit tests and cluster test REVISION DETAIL https://reviews.facebook.net/D11415 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/BitFieldReader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthByteReader.java ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgument.java ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitFieldReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInStream.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRecordReaderImpl.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthByteReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthIntegerReader.java ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/27075/ To: JIRA, omalley Implement predicate pushdown for ORC Key: HIVE-4246 URL: https://issues.apache.org/jira/browse/HIVE-4246 Project: Hive Issue Type: New Feature Components: File Formats Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-4246.D11415.1.patch By using the push down predicates from the table scan operator, ORC can skip over 10,000 rows at a time that won't satisfy the predicate. This will help a lot, especially if the file is sorted by the column that is used in the predicate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4246) Implement predicate pushdown for ORC
[ https://issues.apache.org/jira/browse/HIVE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-4246: Status: Patch Available (was: Open) This patch: * Adds the column names for the required columns * Uses the SearchArgument interface added in HIVE-4579 * Updates the ORC reader to skip over sets of rows that aren't useful. * Extends InStream to read from multiple sets of byte buffers * Updates the ORC reader to skip over ignored rows after each next Implement predicate pushdown for ORC Key: HIVE-4246 URL: https://issues.apache.org/jira/browse/HIVE-4246 Project: Hive Issue Type: New Feature Components: File Formats Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-4246.D11415.1.patch By using the push down predicates from the table scan operator, ORC can skip over 10,000 rows at a time that won't satisfy the predicate. This will help a lot, especially if the file is sorted by the column that is used in the predicate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
