[jira] [Commented] (HIVE-8226) Vectorize dynamic partitioning in VectorFileSinkOperator
[ https://issues.apache.org/jira/browse/HIVE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151450#comment-14151450 ] Prasanth J commented on HIVE-8226: -- [~mmccline] Can you rebase the patch against current trunk? I see failure when I tried to commit this patch. There is diff in golden file when I ran dynpart_sort_opt_vectorization.q test. Also patch did not apply cleanly on trunk. Also is this going into branch-0.14 as well? If so please check with [~vikram.dixit] and make changes to Affects and Fix versions accordingly. Vectorize dynamic partitioning in VectorFileSinkOperator Key: HIVE-8226 URL: https://issues.apache.org/jira/browse/HIVE-8226 Project: Hive Issue Type: Bug Components: Tez, Vectorization Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Attachments: HIVE-8226.01.patch, HIVE-8226.02.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8226) Vectorize dynamic partitioning in VectorFileSinkOperator
[ https://issues.apache.org/jira/browse/HIVE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151468#comment-14151468 ] Matt McCline commented on HIVE-8226: Yes, I rebased and re-ran the dynpart_sort_opt_vectorization.q and found a few stages now vectorize... Perhaps I didn't create patch #2 correctly. Anyway, submitted patch #3. Vectorize dynamic partitioning in VectorFileSinkOperator Key: HIVE-8226 URL: https://issues.apache.org/jira/browse/HIVE-8226 Project: Hive Issue Type: Bug Components: Tez, Vectorization Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Attachments: HIVE-8226.01.patch, HIVE-8226.02.patch, HIVE-8226.03.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8226) Vectorize dynamic partitioning in VectorFileSinkOperator
[ https://issues.apache.org/jira/browse/HIVE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151472#comment-14151472 ] Matt McCline commented on HIVE-8226: ~pjayachandran I added you to e-mail I sent to Gunther about branch-0.14 Vectorize dynamic partitioning in VectorFileSinkOperator Key: HIVE-8226 URL: https://issues.apache.org/jira/browse/HIVE-8226 Project: Hive Issue Type: Bug Components: Tez, Vectorization Affects Versions: 0.14.0 Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8226.01.patch, HIVE-8226.02.patch, HIVE-8226.03.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8226) Vectorize dynamic partitioning in VectorFileSinkOperator
[
https://issues.apache.org/jira/browse/HIVE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151944#comment-14151944
]
Hive QA commented on HIVE-8226:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12671756/HIVE-8226.03.patch
{color:green}SUCCESS:{color} +1 6363 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1038/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1038/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1038/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12671756
Vectorize dynamic partitioning in VectorFileSinkOperator
Key: HIVE-8226
URL: https://issues.apache.org/jira/browse/HIVE-8226
Project: Hive
Issue Type: Bug
Components: Tez, Vectorization
Affects Versions: 0.14.0
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
Fix For: 0.14.0
Attachments: HIVE-8226.01.patch, HIVE-8226.02.patch,
HIVE-8226.03.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8226) Vectorize dynamic partitioning in VectorFileSinkOperator
[ https://issues.apache.org/jira/browse/HIVE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152015#comment-14152015 ] Prasanth J commented on HIVE-8226: -- Committed patch to trunk. I will wait for [~vikram.dixit] to weigh this for branch-0.14 commit. Vectorize dynamic partitioning in VectorFileSinkOperator Key: HIVE-8226 URL: https://issues.apache.org/jira/browse/HIVE-8226 Project: Hive Issue Type: Bug Components: Tez, Vectorization Affects Versions: 0.14.0 Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8226.01.patch, HIVE-8226.02.patch, HIVE-8226.03.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8226) Vectorize dynamic partitioning in VectorFileSinkOperator
[ https://issues.apache.org/jira/browse/HIVE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152019#comment-14152019 ] Vikram Dixit K commented on HIVE-8226: -- +1 for 0.14 Vectorize dynamic partitioning in VectorFileSinkOperator Key: HIVE-8226 URL: https://issues.apache.org/jira/browse/HIVE-8226 Project: Hive Issue Type: Bug Components: Tez, Vectorization Affects Versions: 0.14.0 Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8226.01.patch, HIVE-8226.02.patch, HIVE-8226.03.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8226) Vectorize dynamic partitioning in VectorFileSinkOperator
[
https://issues.apache.org/jira/browse/HIVE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14150884#comment-14150884
]
Hive QA commented on HIVE-8226:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12671583/HIVE-8226.02.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6360 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1020/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1020/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1020/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12671583
Vectorize dynamic partitioning in VectorFileSinkOperator
Key: HIVE-8226
URL: https://issues.apache.org/jira/browse/HIVE-8226
Project: Hive
Issue Type: Bug
Components: Tez, Vectorization
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
Attachments: HIVE-8226.01.patch, HIVE-8226.02.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8226) Vectorize dynamic partitioning in VectorFileSinkOperator
[
https://issues.apache.org/jira/browse/HIVE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14150145#comment-14150145
]
Hive QA commented on HIVE-8226:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12671378/HIVE-8226.01.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1002/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1002/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1002/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-1002/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted
'hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java'
Reverted 'hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java'
Reverted
'hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeHelper.java'
Reverted
'hbase-handler/src/java/org/apache/hadoop/hive/hbase/DefaultHBaseKeyFactory.java'
Reverted
'hbase-handler/src/java/org/apache/hadoop/hive/hbase/struct/AvroHBaseValueFactory.java'
Reverted
'hbase-handler/src/java/org/apache/hadoop/hive/hbase/struct/HBaseValueFactory.java'
Reverted
'hbase-handler/src/java/org/apache/hadoop/hive/hbase/struct/DefaultHBaseValueFactory.java'
Reverted 'hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java'
Reverted
'hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target
shims/0.20S/target shims/0.23/target shims/aggregator/target
shims/common/target shims/common-secure/target packaging/target
hbase-handler/target
hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestStructSerializer.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/struct/HBaseStructValue.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/struct/StructHBaseValueFactory.java
testutils/target jdbc/target metastore/target itests/target
itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target
itests/hive-unit-hadoop2/target itests/hive-minikdc/target
itests/hive-unit/target itests/custom-serde/target itests/util/target
hcatalog/target hcatalog/core/target hcatalog/streaming/target
hcatalog/server-extensions/target hcatalog/webhcat/svr/target
hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target
accumulo-handler/target hwi/target common/target common/src/gen contrib/target
service/target serde/target beeline/target odbc/target cli/target
ql/dependency-reduced-pom.xml ql/target
+ svn update
Uitests/src/test/resources/testconfiguration.properties
Uql/src/test/queries/clientpositive/vector_char_simple.q
Uql/src/test/queries/clientpositive/vector_varchar_simple.q
Uql/src/test/results/clientpositive/vector_char_simple.q.out
Uql/src/test/results/clientpositive/vector_varchar_simple.q.out
Uql/src/test/results/clientpositive/tez/vector_varchar_simple.q.out
Uql/src/test/results/clientpositive/tez/vector_char_simple.q.out
U
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatchCtx.java
Uql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java
U
[jira] [Commented] (HIVE-8226) Vectorize dynamic partitioning in VectorFileSinkOperator
[ https://issues.apache.org/jira/browse/HIVE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14150218#comment-14150218 ] Prasanth J commented on HIVE-8226: -- LGTM, +1. Pending test run. Minor thing: Did you run dynpart_sort_opt_vectorization.q for TestCliDriver. The diff is shown only for tez. Any reason? Vectorize dynamic partitioning in VectorFileSinkOperator Key: HIVE-8226 URL: https://issues.apache.org/jira/browse/HIVE-8226 Project: Hive Issue Type: Bug Components: Tez, Vectorization Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Attachments: HIVE-8226.01.patch, HIVE-8226.02.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8226) Vectorize dynamic partitioning in VectorFileSinkOperator
[ https://issues.apache.org/jira/browse/HIVE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14150219#comment-14150219 ] Matt McCline commented on HIVE-8226: It turns out the explain plans and query results match exactly because we don't vectorize the reduce-side except for Tez. Thanks for the review! Vectorize dynamic partitioning in VectorFileSinkOperator Key: HIVE-8226 URL: https://issues.apache.org/jira/browse/HIVE-8226 Project: Hive Issue Type: Bug Components: Tez, Vectorization Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Attachments: HIVE-8226.01.patch, HIVE-8226.02.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8226) Vectorize dynamic partitioning in VectorFileSinkOperator
[ https://issues.apache.org/jira/browse/HIVE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14150225#comment-14150225 ] Prasanth J commented on HIVE-8226: -- I see. Makes sense. You might want this to be committed before HIVE-8151 so that I can test with VectorFileSink. Vectorize dynamic partitioning in VectorFileSinkOperator Key: HIVE-8226 URL: https://issues.apache.org/jira/browse/HIVE-8226 Project: Hive Issue Type: Bug Components: Tez, Vectorization Reporter: Matt McCline Assignee: Matt McCline Priority: Critical Attachments: HIVE-8226.01.patch, HIVE-8226.02.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
