[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285451#comment-14285451 ] Lefty Leverenz commented on HIVE-5283: -- Doc note: Vectorization parameters are now documented in the wiki, including *hive.vectorized.execution.enabled* (added by this patch). * [Configuration Properties -- Vectorization | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Vectorization] Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: 0.13.0 Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch, HIVE-5283.4.patch, alltypesorc The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13803389#comment-13803389 ] Thejas M Nair commented on HIVE-5283: - Added fix version of 0.13 in addition to vectorization-branch for these 106 jiras (fixed+ fix-version=vectorization-branch ). Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: 0.13.0 Attachments: alltypesorc, HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch, HIVE-5283.4.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13802228#comment-13802228 ] Thejas M Nair commented on HIVE-5283: - Now that vectorization changes are merged into trunk, I think we should *add* fix-version of 0.13 to the jiras. That way the release notes for 0.13 will include these changes as well. I will go ahead and make that change to the jiras. Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: 0.13.0 Attachments: alltypesorc, HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch, HIVE-5283.4.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782760#comment-13782760 ] Hudson commented on HIVE-5283: -- FAILURE: Integrated in Hive-trunk-hadoop2 #467 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/467/]) HIVE-5283 : Merge vectorization branch to trunk (Jitendra Nath Pandey via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1527858) * /hive/trunk * /hive/trunk/ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java * /hive/trunk/ant/src/org/apache/hadoop/hive/ant/GenVectorTestCode.java * /hive/trunk/build-common.xml * /hive/trunk/build.xml * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/data/files/alltypesorc * /hive/trunk/ql/build.xml * /hive/trunk/ql/src/gen/vectorization * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapper.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/vector * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CommonRCFileInputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/VectorizedRCFileInputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/VectorizedRCFileRecordReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/BitFieldReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/DynamicByteArray.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/IntegerReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthByteReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerReaderV2.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcSerde.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFHex.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/vector * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedORCReader.java * /hive/trunk/ql/src/test/queries/clientpositive/vectorization_short_regress.q * /hive/trunk/ql/src/test/queries/clientpositive/vectorized_rcfile_columnar.q * /hive/trunk/ql/src/test/results/clientpositive/add_part_exist.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter1.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter2.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter3.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter4.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter5.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter_index.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter_rename_partition.q.out * /hive/trunk/ql/src/test/results/clientpositive/describe_table_json.q.out * /hive/trunk/ql/src/test/results/clientpositive/index_creation.q.out * /hive/trunk/ql/src/test/results/clientpositive/input2.q.out * /hive/trunk/ql/src/test/results/clientpositive/input3.q.out * /hive/trunk/ql/src/test/results/clientpositive/input4.q.out * /hive/trunk/ql/src/test/results/clientpositive/plan_json.q.out * /hive/trunk/ql/src
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782897#comment-13782897 ] Hudson commented on HIVE-5283: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #121 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/121/]) HIVE-5283 : Merge vectorization branch to trunk (Jitendra Nath Pandey via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1527858) * /hive/trunk * /hive/trunk/ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java * /hive/trunk/ant/src/org/apache/hadoop/hive/ant/GenVectorTestCode.java * /hive/trunk/build-common.xml * /hive/trunk/build.xml * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/data/files/alltypesorc * /hive/trunk/ql/build.xml * /hive/trunk/ql/src/gen/vectorization * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapper.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/vector * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CommonRCFileInputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/VectorizedRCFileInputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/VectorizedRCFileRecordReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/BitFieldReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/DynamicByteArray.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/IntegerReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthByteReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerReaderV2.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcSerde.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFHex.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/vector * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedORCReader.java * /hive/trunk/ql/src/test/queries/clientpositive/vectorization_short_regress.q * /hive/trunk/ql/src/test/queries/clientpositive/vectorized_rcfile_columnar.q * /hive/trunk/ql/src/test/results/clientpositive/add_part_exist.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter1.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter2.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter3.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter4.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter5.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter_index.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter_rename_partition.q.out * /hive/trunk/ql/src/test/results/clientpositive/describe_table_json.q.out * /hive/trunk/ql/src/test/results/clientpositive/index_creation.q.out * /hive/trunk/ql/src/test/results/clientpositive/input2.q.out * /hive/trunk/ql/src/test/results/clientpositive/input3.q.out * /hive/trunk/ql/src/test/results/clientpositive/input4.q.out * /hive/trunk/ql/src/test/results/clientpositive/plan_json.q.out * /hive
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782990#comment-13782990 ] Hudson commented on HIVE-5283: -- SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #187 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/187/]) HIVE-5283 : Merge vectorization branch to trunk (Jitendra Nath Pandey via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1527858) * /hive/trunk * /hive/trunk/ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java * /hive/trunk/ant/src/org/apache/hadoop/hive/ant/GenVectorTestCode.java * /hive/trunk/build-common.xml * /hive/trunk/build.xml * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/data/files/alltypesorc * /hive/trunk/ql/build.xml * /hive/trunk/ql/src/gen/vectorization * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapper.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/vector * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CommonRCFileInputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/VectorizedRCFileInputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/VectorizedRCFileRecordReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/BitFieldReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/DynamicByteArray.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/IntegerReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthByteReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerReaderV2.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcSerde.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFHex.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/vector * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedORCReader.java * /hive/trunk/ql/src/test/queries/clientpositive/vectorization_short_regress.q * /hive/trunk/ql/src/test/queries/clientpositive/vectorized_rcfile_columnar.q * /hive/trunk/ql/src/test/results/clientpositive/add_part_exist.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter1.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter2.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter3.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter4.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter5.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter_index.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter_rename_partition.q.out * /hive/trunk/ql/src/test/results/clientpositive/describe_table_json.q.out * /hive/trunk/ql/src/test/results/clientpositive/index_creation.q.out * /hive/trunk/ql/src/test/results/clientpositive/input2.q.out * /hive/trunk/ql/src/test/results/clientpositive/input3.q.out * /hive/trunk/ql/src/test/results/clientpositive/input4.q.out * /hive/trunk/ql/src/test/results/clientpositive/plan_json.q.out * /hive
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13781981#comment-13781981 ] Jitendra Nath Pandey commented on HIVE-5283: Thanks for the review [~brocknoland]. VectorizedRowBatch is accessed and mutated by vectorized expressions in the inner loops. The members are exposed as public to avoid method call overheads. I will file jiras to address rest of the comments. Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: alltypesorc, HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch, HIVE-5283.4.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13781987#comment-13781987 ] Brock Noland commented on HIVE-5283: Hey thanks for your feedback. Is there evidence showing that the method calls are not inlined by JIT? Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: alltypesorc, HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch, HIVE-5283.4.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782077#comment-13782077 ] Jitendra Nath Pandey commented on HIVE-5283: Inlining would really depend on JIT implementation and in most cases only for final classes and methods. Although, VectorizedRowBatch is not polymorphic for now, but it may evolve in future, and at that point changing the vectorized expressions could become a big deal. We could add comments or annotations in the class to indicate that these variables should be accessed only in restricted packages. Would that be reasonable? Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: alltypesorc, HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch, HIVE-5283.4.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782096#comment-13782096 ] Brock Noland commented on HIVE-5283: bq. Inlining would really depend on JIT implementation and Is there a JVM we are targeting that doesn't have an advanced JIT implementation? bq. in most cases only for final classes and methods. Where are you getting this? I am guessing the state of of the art JIT has improved dramatically since Brian Goetz wrote this article in 2002: http://www.ibm.com/developerworks/java/library/j-jtp1029/index.html Where he says: The common perception is that declaring classes or methods final makes it easier for the compiler to inline method calls, but this perception is incorrect (or at the very least, greatly overstated). So the reality is that while final might be a useful hint to a dumb run-time optimizer that doesn't perform any global dependency analysis, its use doesn't actually enable very many compile-time optimizations, and is not needed by a smart JIT to perform run-time optimizations. Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: alltypesorc, HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch, HIVE-5283.4.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782195#comment-13782195 ] Jitendra Nath Pandey commented on HIVE-5283: I have filed following 3 jiras for the follow up work: HIVE-5397, HIVE-5398 and HIVE-5399. HIVE-5397 is aimed for the discussion/work related to public members in VectorizedRowBatch. Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: alltypesorc, HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch, HIVE-5283.4.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5283: --- Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Jitendra! Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: 0.13.0 Attachments: alltypesorc, HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch, HIVE-5283.4.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782602#comment-13782602 ] Hudson commented on HIVE-5283: -- FAILURE: Integrated in Hive-trunk-h0.21 #2369 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2369/]) HIVE-5283 : Merge vectorization branch to trunk (Jitendra Nath Pandey via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1527858) * /hive/trunk * /hive/trunk/ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java * /hive/trunk/ant/src/org/apache/hadoop/hive/ant/GenVectorTestCode.java * /hive/trunk/build-common.xml * /hive/trunk/build.xml * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/data/files/alltypesorc * /hive/trunk/ql/build.xml * /hive/trunk/ql/src/gen/vectorization * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapper.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/vector * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CommonRCFileInputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/VectorizedRCFileInputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/VectorizedRCFileRecordReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/BitFieldReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/DynamicByteArray.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/IntegerReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthByteReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerReader.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerReaderV2.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcSerde.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFHex.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/vector * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedORCReader.java * /hive/trunk/ql/src/test/queries/clientpositive/vectorization_short_regress.q * /hive/trunk/ql/src/test/queries/clientpositive/vectorized_rcfile_columnar.q * /hive/trunk/ql/src/test/results/clientpositive/add_part_exist.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter1.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter2.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter3.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter4.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter5.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter_index.q.out * /hive/trunk/ql/src/test/results/clientpositive/alter_rename_partition.q.out * /hive/trunk/ql/src/test/results/clientpositive/describe_table_json.q.out * /hive/trunk/ql/src/test/results/clientpositive/index_creation.q.out * /hive/trunk/ql/src/test/results/clientpositive/input2.q.out * /hive/trunk/ql/src/test/results/clientpositive/input3.q.out * /hive/trunk/ql/src/test/results/clientpositive/input4.q.out * /hive/trunk/ql/src/test/results/clientpositive/plan_json.q.out * /hive/trunk/ql/src
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13781495#comment-13781495 ] Carl Steinbach commented on HIVE-5283: -- +1 the latest patch looks good to me. Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: alltypesorc, HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch, HIVE-5283.4.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[
https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13781503#comment-13781503
]
Brock Noland commented on HIVE-5283:
Great work guys!
Can you address these three issues in follow on jiras?
We should never be blindly converting throwable to an exception.
{noformat}
+} catch (Throwable e) {
+ throw new HiveException(e);
+}
{noformat}
This is non-standard:
{noformat}
+ transient private long value;
{noformat}
Unless we have a very good reason we shouldn't be exposing the members of
VectorizedRowBatch publicly.
Merge vectorization branch to trunk
---
Key: HIVE-5283
URL: https://issues.apache.org/jira/browse/HIVE-5283
Project: Hive
Issue Type: Bug
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey
Attachments: alltypesorc, HIVE-5283.1.patch, HIVE-5283.2.patch,
HIVE-5283.3.patch, HIVE-5283.4.patch
The purpose of this jira is to upload vectorization patch, run tests etc. The
actual work will continue under HIVE-4160 umbrella jira.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13780578#comment-13780578 ] Ashutosh Chauhan commented on HIVE-5283: +1 I have overseen much of this work so I am familiar with this work. But it won't harm to get extra pair of eyes on it. So, it will be good if other committers also chime in. Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: alltypesorc, HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch, HIVE-5283.4.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13779242#comment-13779242 ] Jitendra Nath Pandey commented on HIVE-5283: I ran tests in my setup. All tests pass. [~cwsteinbach] Please take a look, the patch addresses your feedback comments. Other committers are also requested to take a look. Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: alltypesorc, HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch, HIVE-5283.4.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[
https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777252#comment-13777252
]
Hive QA commented on HIVE-5283:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12604927/HIVE-5283.3.patch
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/884/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/884/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-884/source-prep.txt
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java'
Reverted
'service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf build hcatalog/build hcatalog/core/build
hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build
hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build
hcatalog/hcatalog-pig-adapter/build common/src/gen
+ svn update
Abeeline/src/test/org/apache/hive/beeline/src/test/TestSchemaTool.java
Ubeeline/src/java/org/apache/hive/beeline/BeeLineOpts.java
Abeeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java
Abeeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java
Ubeeline/src/java/org/apache/hive/beeline/Commands.java
Ubeeline/src/java/org/apache/hive/beeline/BeeLine.java
Ubuild.xml
Umetastore/scripts/upgrade/derby/014-HIVE-3764.derby.sql
Umetastore/scripts/upgrade/mysql/014-HIVE-3764.mysql.sql
Umetastore/scripts/upgrade/oracle/014-HIVE-3764.oracle.sql
Umetastore/scripts/upgrade/postgres/014-HIVE-3764.postgres.sql
Abin/schematool
Abin/ext/schemaTool.sh
Fetching external item into 'hcatalog/src/test/e2e/harness'
Updated external to revision 1526125.
Updated to revision 1526125.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0 to p2
+ exit 1
'
{noformat}
This message is automatically generated.
Merge vectorization branch to trunk
---
Key: HIVE-5283
URL: https://issues.apache.org/jira/browse/HIVE-5283
Project: Hive
Issue Type: Bug
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey
Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch
The purpose of this jira is to upload vectorization patch, run tests etc. The
actual work will continue under HIVE-4160 umbrella jira.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-5283: --- Status: Open (was: Patch Available) Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch, HIVE-5283.4.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-5283: --- Status: Patch Available (was: Open) Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch, HIVE-5283.4.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778185#comment-13778185 ] Jitendra Nath Pandey commented on HIVE-5283: bq. Are we really going to apply a patch, or are we going to SVN merge? I would rather see a merge so we do not lose the commit history on the branch. I am fine with either approach to get this work into trunk. Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch, HIVE-5283.4.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 51 tests failed
{noformat}
This message is automatically generated.
Merge vectorization branch to trunk
---
Key: HIVE-5283
URL: https://issues.apache.org/jira/browse/HIVE-5283
Project: Hive
Issue Type: Bug
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey
Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch,
HIVE-5283.4.patch
The purpose of this jira is to upload vectorization patch, run tests etc. The
actual work will continue under HIVE-4160 umbrella jira.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-5283: --- Attachment: alltypesorc The tests are failing because the patch contains a binary file that patch doesn't handle correctly. Uploaded the binary file. I will run the tests manually and provide the results. Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: alltypesorc, HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch, HIVE-5283.4.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request 14130: Merge vectorization branch to trunk
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14130/ --- (Updated Sept. 25, 2013, 12:07 a.m.) Review request for hive and Ashutosh Chauhan. Changes --- Updated patch, addressing the review comments and rebased to the latest state of the trunk. Bugs: HIVE-5283 https://issues.apache.org/jira/browse/HIVE-5283 Repository: hive-git Description --- Merge vectorization branch to trunk. Diffs (updated) - ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java PRE-CREATION ant/src/org/apache/hadoop/hive/ant/GenVectorTestCode.java PRE-CREATION build-common.xml 5a16bff build.xml cf75b3d common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e971644 conf/hive-default.xml.template 1ee756c data/files/alltypesorc PRE-CREATION ql/build.xml ac2b7b0 ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterColumnCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterScalarCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringColumnCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringScalarCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestClass.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnColumnFilterVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnColumnOperationVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnScalarFilterVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnScalarOperationVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvg.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMax.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMaxString.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFSum.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFVar.txt PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java b7d45c5 ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java bcee201 ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java d2c981d ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java e498327 ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapper.java c303b30 ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 3b15667 ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 85a22b7 ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 1f763a4 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java c6cecb5 ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java bd9d0bb ql/src/java/org/apache/hadoop/hive/ql/exec/vector/BytesColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/DoubleColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/LongColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/TimestampUtils.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAggregationBufferBatch.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAggregationBufferRow.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorFileSinkOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorFilterOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorGroupByOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorHashKeyWrapper.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorHashKeyWrapperBatch.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorReduceSinkOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSelectOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java PRE-CREATION
Re: Review Request 14130: Merge vectorization branch to trunk
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14130/ --- (Updated Sept. 25, 2013, 12:08 a.m.) Review request for hive and Ashutosh Chauhan. Changes --- Updated patch, addressing the review comments and rebased to the latest state of the trunk. Bugs: HIVE-5283 https://issues.apache.org/jira/browse/HIVE-5283 Repository: hive-git Description --- Merge vectorization branch to trunk. Diffs - ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java PRE-CREATION ant/src/org/apache/hadoop/hive/ant/GenVectorTestCode.java PRE-CREATION build-common.xml 5a16bff build.xml cf75b3d common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e971644 conf/hive-default.xml.template 1ee756c data/files/alltypesorc PRE-CREATION ql/build.xml ac2b7b0 ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterColumnCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterScalarCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringColumnCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringScalarCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestClass.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnColumnFilterVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnColumnOperationVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnScalarFilterVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnScalarOperationVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvg.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMax.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMaxString.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFSum.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFVar.txt PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java b7d45c5 ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java bcee201 ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java d2c981d ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java e498327 ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapper.java c303b30 ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 3b15667 ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 85a22b7 ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 1f763a4 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java c6cecb5 ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java bd9d0bb ql/src/java/org/apache/hadoop/hive/ql/exec/vector/BytesColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/DoubleColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/LongColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/TimestampUtils.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAggregationBufferBatch.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAggregationBufferRow.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorFileSinkOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorFilterOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorGroupByOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorHashKeyWrapper.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorHashKeyWrapperBatch.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorReduceSinkOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSelectOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java PRE-CREATION ql/src/java
[jira] [Updated] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-5283: --- Status: Patch Available (was: Open) Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-5283: --- Attachment: HIVE-5283.3.patch Updated patch, addressing the review comments. Updated on review board as well. Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777072#comment-13777072 ] Edward Capriolo commented on HIVE-5283: --- Are we really going to apply a patch, or are we going to SVN merge? I would rather see a merge so we do not lose the commit history on the branch. Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch, HIVE-5283.3.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request 14130: Merge vectorization branch to trunk
, 5:51 p.m.) Review request for hive and Ashutosh Chauhan. Bugs: HIVE-5283 https://issues.apache.org/jira/browse/HIVE-5283 Repository: hive-git Description --- Merge vectorization branch to trunk. Diffs - .gitignore c0e9b3c build-common.xml ee219a9 common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c5a8ff3 metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 15a2a81 ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterColumnCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterScalarCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringColumnCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringScalarCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestClass.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnColumnFilterVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnColumnOperationVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnScalarFilterVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnScalarOperationVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvg.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMax.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMaxString.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFSum.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFVar.txt PRE-CREATION ql/src/gen/vectorization/org/apache/hadoop/hive/ql/exec/vector/gen/CodeGen.java PRE-CREATION ql/src/gen/vectorization/org/apache/hadoop/hive/ql/exec/vector/gen/TestCodeGen.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 393ef57 ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java d2265e2 ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java bcee201 ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java d2c981d ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java e498327 ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapper.java c303b30 ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 3b15667 ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 85a22b7 ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 869417f ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 2ece97e ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 4cc7129 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/BytesColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/DoubleColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/LongColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/TimestampUtils.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAggregationBufferBatch.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAggregationBufferRow.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorFileSinkOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorFilterOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorGroupByOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorHashKeyWrapper.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorHashKeyWrapperBatch.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorReduceSinkOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSelectOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedBatchUtil.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec
Re: Review Request 14130: Merge vectorization branch to trunk
On Sept. 16, 2013, 12:10 a.m., Carl Steinbach wrote: ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java, line 111 https://reviews.apache.org/r/14130/diff/1/?file=351926#file351926line111 AllVectorTypesRecord seems to add an additional level of indirection without providing any real benefit. I'd recommend following convention and just hardcoding it for now. Addressed in HIVE-5309. - Jitendra --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14130/#review26120 --- On Sept. 13, 2013, 5:51 p.m., Jitendra Pandey wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14130/ --- (Updated Sept. 13, 2013, 5:51 p.m.) Review request for hive and Ashutosh Chauhan. Bugs: HIVE-5283 https://issues.apache.org/jira/browse/HIVE-5283 Repository: hive-git Description --- Merge vectorization branch to trunk. Diffs - .gitignore c0e9b3c build-common.xml ee219a9 common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c5a8ff3 metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 15a2a81 ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterColumnCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterScalarCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringColumnCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringScalarCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestClass.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnColumnFilterVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnColumnOperationVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnScalarFilterVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnScalarOperationVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvg.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMax.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMaxString.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFSum.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFVar.txt PRE-CREATION ql/src/gen/vectorization/org/apache/hadoop/hive/ql/exec/vector/gen/CodeGen.java PRE-CREATION ql/src/gen/vectorization/org/apache/hadoop/hive/ql/exec/vector/gen/TestCodeGen.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 393ef57 ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java d2265e2 ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java bcee201 ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java d2c981d ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java e498327 ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapper.java c303b30 ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 3b15667 ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 85a22b7 ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 869417f ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 2ece97e ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 4cc7129 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/BytesColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/DoubleColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/LongColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/TimestampUtils.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAggregationBufferBatch.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector
Re: Review Request 14130: Merge vectorization branch to trunk
On Sept. 16, 2013, 12:02 a.m., Carl Steinbach wrote: Jitendra Pandey wrote: I have filed jiras HIVE-5308, HIVE-5309, HIVE-5314 to address some of the review comments. I will upload an updated patch as soon as those jiras are committed. The individual comments are answered inline. Thanks for addressing these issues. On Sept. 16, 2013, 12:02 a.m., Carl Steinbach wrote: ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt, line 1 https://reviews.apache.org/r/14130/diff/1/?file=351623#file351623line1 We currently use Apache Velocity to generate test code at compile-time (e.g. TestCliDriver, ...). I realize that the templating code in CodeGen and TestCodeGen is pretty simple, but was wondering if it might be better from a build and maintenance standpoint to use Velocity instead. Also, is it possible to select a less generic file suffix for the template files, e.g. *.t or *.tmpl? Jitendra Pandey wrote: I have added a patch on HIVE-5308, that removes the generated code for both expressions and tests and instead the code is generated as an ant-task during the build. The patch doesn't however uses Velocity, do you think its ok to move to velocity as a follow up task post merge? I will take care of template file suffixes as part of move to velocity because I believe velocity requires its own suffix. I haven't looked at it closely enough to be able to say that using Velocity is a superior approach. I think it's something worth investigating in the future, but agree with you that there's no need to do it right away. On Sept. 16, 2013, 12:02 a.m., Carl Steinbach wrote: ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt, line 27 https://reviews.apache.org/r/14130/diff/1/?file=351623#file351623line27 In addition to the name (and preferably path) of the template I think this comment should also include the name and path of the code generator, and a warning that it should not be modified by hand. Jitendra Pandey wrote: The generated code will not be committed anymore. The code will be generated in the build directory and a clean will remove it. Awesome! On Sept. 16, 2013, 12:02 a.m., Carl Steinbach wrote: ql/src/gen/vectorization/org/apache/hadoop/hive/ql/exec/vector/gen/TestCodeGen.java, line 1 https://reviews.apache.org/r/14130/diff/1/?file=351645#file351645line1 Maybe this should go in ql/src/test/gen. Thoughts? Jitendra Pandey wrote: I have renamed this file to GenVectorTestCode and moved it to ant/src/org/apache/hadoop/hive/ant/ so that it runs along with other ant-tasks. Is that ok? The jira with this change is HIVE-5308. Sounds good. On Sept. 16, 2013, 12:02 a.m., Carl Steinbach wrote: ql/src/test/queries/clientpositive/vectorization_0.q, line 20 https://reviews.apache.org/r/14130/diff/1/?file=351959#file351959line20 What is the expected behavior when vectorized.execution=enabled and the source table is not reading ORC formatted data? I think it's worth adding some additional tests (either positive or negative) to lock down this behavior. Jitendra Pandey wrote: Any query that vectorization cannot execute must fall back to non-vector mode. If the input format doesn't provide vectorized input the query must still succeed. I have added a test in my patch on HIVE-5309 that runs a query on rc-file table with vectorization turned on. The query executes successfully. Sounds good. - Carl --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14130/#review26119 --- On Sept. 13, 2013, 5:51 p.m., Jitendra Pandey wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14130/ --- (Updated Sept. 13, 2013, 5:51 p.m.) Review request for hive and Ashutosh Chauhan. Bugs: HIVE-5283 https://issues.apache.org/jira/browse/HIVE-5283 Repository: hive-git Description --- Merge vectorization branch to trunk. Diffs - .gitignore c0e9b3c build-common.xml ee219a9 common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c5a8ff3 metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 15a2a81 ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterColumnCompareColumn.txt
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771592#comment-13771592 ] Carl Steinbach commented on HIVE-5283: -- [~anthony.murphy] Thanks for the very detailed explanation. Clearly a lot of thought has gone into writing these tests. Now we just need to find a way of conveying this information to people down the road. I recommend doing the following: concatenate all of these tests into a single qfile named vectorization_short_regress.q and include the information from above in a comment at the top of the file. It would also be great if you could include a short comment per query so folks have an easy way of telling them apart. Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770196#comment-13770196 ] Tony Murphy commented on HIVE-5283: --- [~cwsteinbach]Thanks for the questions and I would definitely appreciate some feedback on how to appropriately document the test strategy I used here. In regards to you question about magic numbers in the queries, the values of effectively random, but they are important. If you look at [ql/src/test/org/apache/hadoop/hive/ql/exec/vector/util/OrcFileGenerator.java|https://reviews.apache.org/r/14130/diff/?page=17#337] which is the data generation class you'll see that those values are specified in the initializeFixedPointValues for each data type. When I created the queries I used those values where I needed scalar values to ensure that when the queries executed their predicates would be filtering on values that are guaranteed to exist. Beyond those values, all the other data in the alltypesorc file is random, but there is a specific pattern to the data that is important for coverage. In orc and subsequently vectorization there are a number of optimizations for certain data patterns: AllValues, NoNulls, RepeatingValue, RepeatingNull. The data in alltypesorc is generated such that each column has exactly 3 batches of each data pattern. This gives us coverage for the vector expression optimizations and ensure the metadata in appropriately set on the row batch object which are reused across batches. For the queries themselves in order to efficiently cover as much of the new vectorization functionality as I could I used a number of different techniques to create the vectorization_*.q test suites, primarily equivalence classes, and pairwise combinations. First I divided the search space into a number of dimensions such as type, aggregate function, filter operation, arithmetic operation, etc. The types were explored as equivalence classes of long, double, time, string, and bool. Also, rather than creating a very large number of small queries the resulting vectors were grouped by compatible dimensions to reduce the number of queries. It wouldn't be to much work to add comments into the .q files that summarize the coverage they provide based on the vectors used to create each scenario. Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-5283: - Status: Open (was: Patch Available) Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request 14130: Merge vectorization branch to trunk
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14130/#review26119 --- .gitignore https://reviews.apache.org/r/14130/#comment50977 I think we should follow established convention of checking these file instead of generating them since it serves as a useful canary for catching accidental changes to the ORC format. common/src/java/org/apache/hadoop/hive/conf/HiveConf.java https://reviews.apache.org/r/14130/#comment50978 Please add this to hive-default.xml.template along with a description. metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java https://reviews.apache.org/r/14130/#comment50979 Necessary? ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt https://reviews.apache.org/r/14130/#comment50980 We currently use Apache Velocity to generate test code at compile-time (e.g. TestCliDriver, ...). I realize that the templating code in CodeGen and TestCodeGen is pretty simple, but was wondering if it might be better from a build and maintenance standpoint to use Velocity instead. Also, is it possible to select a less generic file suffix for the template files, e.g. *.t or *.tmpl? ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt https://reviews.apache.org/r/14130/#comment50981 In addition to the name (and preferably path) of the template I think this comment should also include the name and path of the code generator, and a warning that it should not be modified by hand. ql/src/gen/vectorization/org/apache/hadoop/hive/ql/exec/vector/gen/TestCodeGen.java https://reviews.apache.org/r/14130/#comment50982 Maybe this should go in ql/src/test/gen. Thoughts? ql/src/test/queries/clientpositive/vectorization_0.q https://reviews.apache.org/r/14130/#comment50984 There are a lot of magic numbers in these these new test files. Do they have any special meaning or are they effectively random? ql/src/test/queries/clientpositive/vectorization_0.q https://reviews.apache.org/r/14130/#comment50983 What is the expected behavior when vectorized.execution=enabled and the source table is not reading ORC formatted data? I think it's worth adding some additional tests (either positive or negative) to lock down this behavior. - Carl Steinbach On Sept. 13, 2013, 5:51 p.m., Jitendra Pandey wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14130/ --- (Updated Sept. 13, 2013, 5:51 p.m.) Review request for hive and Ashutosh Chauhan. Bugs: HIVE-5283 https://issues.apache.org/jira/browse/HIVE-5283 Repository: hive-git Description --- Merge vectorization branch to trunk. Diffs - .gitignore c0e9b3c build-common.xml ee219a9 common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c5a8ff3 metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 15a2a81 ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterColumnCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterScalarCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringColumnCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringScalarCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestClass.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnColumnFilterVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnColumnOperationVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnScalarFilterVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnScalarOperationVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvg.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMax.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMaxString.txt
Re: Review Request 14130: Merge vectorization branch to trunk
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14130/#review26120 --- ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java https://reviews.apache.org/r/14130/#comment50985 AllVectorTypesRecord seems to add an additional level of indirection without providing any real benefit. I'd recommend following convention and just hardcoding it for now. - Carl Steinbach On Sept. 13, 2013, 5:51 p.m., Jitendra Pandey wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14130/ --- (Updated Sept. 13, 2013, 5:51 p.m.) Review request for hive and Ashutosh Chauhan. Bugs: HIVE-5283 https://issues.apache.org/jira/browse/HIVE-5283 Repository: hive-git Description --- Merge vectorization branch to trunk. Diffs - .gitignore c0e9b3c build-common.xml ee219a9 common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c5a8ff3 metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 15a2a81 ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterColumnCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterScalarCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringColumnCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringScalarCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestClass.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnColumnFilterVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnColumnOperationVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnScalarFilterVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnScalarOperationVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvg.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMax.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMaxString.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFSum.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFVar.txt PRE-CREATION ql/src/gen/vectorization/org/apache/hadoop/hive/ql/exec/vector/gen/CodeGen.java PRE-CREATION ql/src/gen/vectorization/org/apache/hadoop/hive/ql/exec/vector/gen/TestCodeGen.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 393ef57 ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java d2265e2 ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java bcee201 ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java d2c981d ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java e498327 ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapper.java c303b30 ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 3b15667 ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 85a22b7 ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 869417f ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 2ece97e ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 4cc7129 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/BytesColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/DoubleColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/LongColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/TimestampUtils.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAggregationBufferBatch.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAggregationBufferRow.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorFileSinkOperator.java PRE
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767957#comment-13767957 ] Carl Steinbach commented on HIVE-5283: -- I added some more comments on RB. I wanted to note here since the site seems overwhelmed by the size of this patch and I have my doubts that they're actually going to get reposted here. Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-5283: --- Attachment: HIVE-5283.2.patch The previous patch didn't apply cleanly, here is the updated patch. Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-5283: --- Status: Patch Available (was: Open) Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-5283: --- Status: Open (was: Patch Available) Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13766303#comment-13766303 ] Carl Steinbach commented on HIVE-5283: -- [~jnp] Can you please post a review request on phabricator or reviewboard? Also, is the plan to commit this patch or to merge the vectorization branch into trunk? Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request 14130: Merge vectorization branch to trunk
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14130/ --- Review request for hive and Ashutosh Chauhan. Bugs: HIVE-5283 https://issues.apache.org/jira/browse/HIVE-5283 Repository: hive-git Description --- Merge vectorization branch to trunk. Diffs - .gitignore c0e9b3c build-common.xml ee219a9 common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c5a8ff3 metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 15a2a81 ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterColumnCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterScalarCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringColumnCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringColumnCompareScalar.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/FilterStringScalarCompareColumn.txt PRE-CREATION ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestClass.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnColumnFilterVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnColumnOperationVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnScalarFilterVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/TestTemplates/TestColumnScalarOperationVectorExpressionEvaluation.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvg.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMax.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMaxString.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFSum.txt PRE-CREATION ql/src/gen/vectorization/UDAFTemplates/VectorUDAFVar.txt PRE-CREATION ql/src/gen/vectorization/org/apache/hadoop/hive/ql/exec/vector/gen/CodeGen.java PRE-CREATION ql/src/gen/vectorization/org/apache/hadoop/hive/ql/exec/vector/gen/TestCodeGen.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 393ef57 ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java d2265e2 ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java bcee201 ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java d2c981d ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java e498327 ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapper.java c303b30 ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 3b15667 ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 85a22b7 ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 869417f ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 2ece97e ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 4cc7129 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/BytesColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/DoubleColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/LongColumnVector.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/TimestampUtils.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAggregationBufferBatch.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAggregationBufferRow.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorFileSinkOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorFilterOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorGroupByOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorHashKeyWrapper.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorHashKeyWrapperBatch.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorReduceSinkOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSelectOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13766733#comment-13766733 ] Jitendra Nath Pandey commented on HIVE-5283: Added review request on the reviewboard: https://reviews.apache.org/r/14130/ bq. is the plan to commit this patch or to merge the vectorization branch into trunk? I am fine with either approach. I have uploaded this patch for jenkins runs and to have all the changes in a single place. But, if change in by-laws takes too long, it might make sense to just get it committed as a patch. Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[
https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13766783#comment-13766783
]
Jitendra Nath Pandey commented on HIVE-5283:
[~brocknoland] The tests are not getting executed for this patch and the
console output shows following exception in hudson. Could you please take a
look? I verified that the patch applies cleanly to trunk.
{code}
ERROR: Failed to archive artifacts: hive/build/test-results/TEST-*.xml
hudson.util.IOException2: hudson.util.IOException2: Failed to extract
/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/test-results/TEST-*.xml
at hudson.FilePath.readFromTar(FilePath.java:2050)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1962)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:137)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:782)
at
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:754)
at hudson.model.Build$BuildExecution.post2(Build.java:183)
at
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:707)
at hudson.model.Run.execute(Run.java:1628)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:246)
Caused by: java.io.IOException
at
hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:175)
at hudson.util.HeadBufferingStream.read(HeadBufferingStream.java:61)
at
com.jcraft.jzlib.InflaterInputStream.fill(InflaterInputStream.java:175)
at
com.jcraft.jzlib.InflaterInputStream.read(InflaterInputStream.java:106)
at org.apache.tools.tar.TarBuffer.readBlock(TarBuffer.java:257)
at org.apache.tools.tar.TarBuffer.readRecord(TarBuffer.java:223)
at
hudson.org.apache.tools.tar.TarInputStream.read(TarInputStream.java:345)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1025)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:999)
at hudson.util.IOUtils.copy(IOUtils.java:37)
at hudson.FilePath.readFromTar(FilePath.java:2040)
... 11 more
at hudson.FilePath.copyRecursiveTo(FilePath.java:1969)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:137)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:782)
at
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:754)
at hudson.model.Build$BuildExecution.post2(Build.java:183)
at
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:707)
at hudson.model.Run.execute(Run.java:1628)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:246)
Caused by: java.util.concurrent.ExecutionException:
hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected
termination of the channel
at hudson.remoting.Request$1.get(Request.java:278)
at hudson.remoting.Request$1.get(Request.java:210)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1965)
... 10 more
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException:
Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:299)
at hudson.remoting.Channel.terminate(Channel.java:774)
at
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
Caused by: java.io.IOException: Unexpected termination of the channel
at
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at
java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1316)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at hudson.remoting.Command.readFrom(Command.java:92)
at
hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
at
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
{code}
Merge vectorization branch to trunk
---
Key: HIVE-5283
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13766865#comment-13766865 ] Brock Noland commented on HIVE-5283: Not sure what happened. I would re-run. But there is quite a queue at present so you might want to run locally. Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
[
https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13766873#comment-13766873
]
Hive QA commented on HIVE-5283:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12602928/HIVE-5283.2.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 3965 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_plan_json
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/719/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/719/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
Merge vectorization branch to trunk
---
Key: HIVE-5283
URL: https://issues.apache.org/jira/browse/HIVE-5283
Project: Hive
Issue Type: Bug
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey
Attachments: HIVE-5283.1.patch, HIVE-5283.2.patch
The purpose of this jira is to upload vectorization patch, run tests etc. The
actual work will continue under HIVE-4160 umbrella jira.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5283) Merge vectorization branch to trunk
Jitendra Nath Pandey created HIVE-5283: -- Summary: Merge vectorization branch to trunk Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5283) Merge vectorization branch to trunk
hcatalog/core/src/main/java/org/apache/hive/hcatalog/security/StorageDelegationAuthorizationProvider.java
Uhcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java
D
hcatalog/core/src/test/java/org/apache/hive/hcatalog/cli/DummyStorageHandler.java
Uhcatalog/storage-handlers/hbase/pom.xml
A
hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHiveHBaseTableOutputFormat.java
A
hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHiveHBaseStorageHandler.java
A
hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestPigHBaseStorageHandler.java
U
hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/SkeletonHBaseTest.java
U
hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHBaseInputFormat.java
A
hcatalog/storage-handlers/hbase/src/test/org/apache/hcatalog/hbase/TestHCatHBaseInputFormat.java
U
hcatalog/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/snapshot/RevisionManager.java
U
hcatalog/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseHCatStorageHandler.java
Uhcatalog/build-support/ant/checkstyle.xml
Uql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
Aql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java
Aql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughRecordWriter.java
Uql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java
Uql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java
Uql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
Uql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java
Uql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
Uql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java
Uql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
Fetching external item into 'hcatalog/src/test/e2e/harness'
Updated external to revision 1522681.
Updated to revision 1522681.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0 to p2
+ exit 1
'
{noformat}
This message is automatically generated.
Merge vectorization branch to trunk
---
Key: HIVE-5283
URL: https://issues.apache.org/jira/browse/HIVE-5283
Project: Hive
Issue Type: Bug
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey
Attachments: HIVE-5283.1.patch
The purpose of this jira is to upload vectorization patch, run tests etc. The
actual work will continue under HIVE-4160 umbrella jira.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-5283: --- Status: Patch Available (was: Open) Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5283) Merge vectorization branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-5283: --- Attachment: HIVE-5283.1.patch Merge vectorization branch to trunk --- Key: HIVE-5283 URL: https://issues.apache.org/jira/browse/HIVE-5283 Project: Hive Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5283.1.patch The purpose of this jira is to upload vectorization patch, run tests etc. The actual work will continue under HIVE-4160 umbrella jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Merge vectorization branch to trunk
On Mon, Sep 9, 2013 at 6:51 PM, Edward Capriolo edlinuxg...@gmail.comwrote: Can we vote to change the by-laws and then svn merge the branch? Seems better then making a big patch and losing the commit history. Yes, we can do that, but amending the bylaws requires a 2/3 majority vote from active PMC members and a six day voting period. There are a couple other feature branches that people are working on right now, so it probably makes sense to bite the bullet and get this over with. I'll work on getting a proposal ready. Thanks. Carl
Re: Merge vectorization branch to trunk
Hi Brock, I will merge latest trunk to the branch and run all the tests again, and provide you the run time difference. Trunk has moved ahead since the last merge, therefore test results are a bit stale right now. What new modules or dependencies are added? No new dependencies have been added. Can the feature be disabled if required? Yes, there is boolean hive.vectorized.execution.enabled flag. If it is false, vectorization optimization is completely ignored. By default, we intend to keep it false for now, so that users who don't care about vectorization don't have to change anything. On Fri, Sep 6, 2013 at 2:24 PM, Brock Noland br...@cloudera.com wrote: Hi, First of all I'd like to say thanks for your hard work! I also agree that this should go in post 0.12 branch. I have a couple questions: Do all the tests pass? How long does ant test test take in comparison with trunk? What new modules or dependencies are added? Can the feature be disabled if required? Brock On Fri, Sep 6, 2013 at 3:44 PM, Jitendra Pandey jiten...@hortonworks.com wrote: Hi Folks, Vectorized query execution work has been under development for around 6 months now. We have made good progress in making basic datatypes and queries to work in vectorized mode and also we have laid sufficient groundwork so that any future development can happen directly on trunk. This is a community effort involving several developers including Eric Hanson, Remus Rusanu, Sarvesh Sakalanaga https://issues.apache.org/jira/secure/ViewProfile.jspa?name=sarvesh.sn, Gopal V, Teddy Choi, Tony Murphy, Timothy Chen, Prashanth J and myself. I believe vectorization branch is now ready to be merged to the trunk. I propose to merge it immediately after the hive-0.12 branch is cut, so that we get sufficient time before the next release. thanks, jitendra -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Merge vectorization branch to trunk
First of all I'd like to say thanks for your hard work! I also agree that this should go in post 0.12 branch. Ditto. As far as merging this into trunk goes, we're currently hamstrung by the fact that the Hive Bylaws don't include any provisions covering development work done on feature branches. Consequently, the only option currently available for getting this merged into trunk is to treat it like a regular patch, i.e. file a JIRA ticket, post a patch for review, get a +1 vote and then wait 24 hours before doing the actual commit. I think we should consider amending the bylaws to include the feature branch provisions that are currently found in the Hadoop project bylaws: http://hadoop.apache.org/bylaws.html Thanks. Carl
Re: Merge vectorization branch to trunk
Can we vote to change the by-laws and then svn merge the branch? Seems better then making a big patch and losing the commit history. On Mon, Sep 9, 2013 at 9:44 PM, Carl Steinbach cwsteinb...@gmail.comwrote: First of all I'd like to say thanks for your hard work! I also agree that this should go in post 0.12 branch. Ditto. As far as merging this into trunk goes, we're currently hamstrung by the fact that the Hive Bylaws don't include any provisions covering development work done on feature branches. Consequently, the only option currently available for getting this merged into trunk is to treat it like a regular patch, i.e. file a JIRA ticket, post a patch for review, get a +1 vote and then wait 24 hours before doing the actual commit. I think we should consider amending the bylaws to include the feature branch provisions that are currently found in the Hadoop project bylaws: http://hadoop.apache.org/bylaws.html Thanks. Carl
Merge vectorization branch to trunk
Hi Folks, Vectorized query execution work has been under development for around 6 months now. We have made good progress in making basic datatypes and queries to work in vectorized mode and also we have laid sufficient groundwork so that any future development can happen directly on trunk. This is a community effort involving several developers including Eric Hanson, Remus Rusanu, Sarvesh Sakalanagahttps://issues.apache.org/jira/secure/ViewProfile.jspa?name=sarvesh.sn, Gopal V, Teddy Choi, Tony Murphy, Timothy Chen, Prashanth J and myself. I believe vectorization branch is now ready to be merged to the trunk. I propose to merge it immediately after the hive-0.12 branch is cut, so that we get sufficient time before the next release. thanks, jitendra -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
