[jira] [Commented] (HIVE-11529) Loop optimization for SIMD in IfExprLong*Long*
[
https://issues.apache.org/jira/browse/HIVE-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948424#comment-15948424
]
Teddy Choi commented on HIVE-11529:
---
The test failure
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
is not reproducible in my laptop. It just succeeds. I'm trying to find out
more.
> Loop optimization for SIMD in IfExprLong*Long*
> --
>
> Key: HIVE-11529
> URL: https://issues.apache.org/jira/browse/HIVE-11529
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
> Attachments: HIVE-11529.2.patch, HIVE-11529.3.patch,
> HIVE-11529.4.patch, HIVE-11529.5.patch, HIVE-11529.6.patch, HIVE-11529.patch
>
>
> HIVE-10238 showed that ?: operator can be optimized with bitwise operators
> when it only handles LongColumnVectors. There are more similar cases as
> following.
> IfExprLongColumnLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> vector2[i] : arg3Scalar);{code}
> IfExprLongScalarLongColumn: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : vector3[i]);{code}
> IfExprLongScalarLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : arg3Scalar);{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11529) Loop optimization for SIMD in IfExprLong*Long*
[
https://issues.apache.org/jira/browse/HIVE-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948346#comment-15948346
]
Hive QA commented on HIVE-11529:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12861144/HIVE-11529.6.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10540 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=141)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=172)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4457/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4457/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4457/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12861144 - PreCommit-HIVE-Build
> Loop optimization for SIMD in IfExprLong*Long*
> --
>
> Key: HIVE-11529
> URL: https://issues.apache.org/jira/browse/HIVE-11529
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
> Attachments: HIVE-11529.2.patch, HIVE-11529.3.patch,
> HIVE-11529.4.patch, HIVE-11529.5.patch, HIVE-11529.6.patch, HIVE-11529.patch
>
>
> HIVE-10238 showed that ?: operator can be optimized with bitwise operators
> when it only handles LongColumnVectors. There are more similar cases as
> following.
> IfExprLongColumnLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> vector2[i] : arg3Scalar);{code}
> IfExprLongScalarLongColumn: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : vector3[i]);{code}
> IfExprLongScalarLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : arg3Scalar);{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11529) Loop optimization for SIMD in IfExprLong*Long*
[
https://issues.apache.org/jira/browse/HIVE-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948267#comment-15948267
]
Teddy Choi commented on HIVE-11529:
---
[~Ferd], I fixed remaining bugs.
> Loop optimization for SIMD in IfExprLong*Long*
> --
>
> Key: HIVE-11529
> URL: https://issues.apache.org/jira/browse/HIVE-11529
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
> Attachments: HIVE-11529.2.patch, HIVE-11529.3.patch,
> HIVE-11529.4.patch, HIVE-11529.5.patch, HIVE-11529.6.patch, HIVE-11529.patch
>
>
> HIVE-10238 showed that ?: operator can be optimized with bitwise operators
> when it only handles LongColumnVectors. There are more similar cases as
> following.
> IfExprLongColumnLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> vector2[i] : arg3Scalar);{code}
> IfExprLongScalarLongColumn: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : vector3[i]);{code}
> IfExprLongScalarLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : arg3Scalar);{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11529) Loop optimization for SIMD in IfExprLong*Long*
[
https://issues.apache.org/jira/browse/HIVE-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948163#comment-15948163
]
Ferdinand Xu commented on HIVE-11529:
-
Could you check why the result for vectorized_case.q is changed? Thanks.
> Loop optimization for SIMD in IfExprLong*Long*
> --
>
> Key: HIVE-11529
> URL: https://issues.apache.org/jira/browse/HIVE-11529
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
> Attachments: HIVE-11529.2.patch, HIVE-11529.3.patch,
> HIVE-11529.4.patch, HIVE-11529.5.patch, HIVE-11529.patch
>
>
> HIVE-10238 showed that ?: operator can be optimized with bitwise operators
> when it only handles LongColumnVectors. There are more similar cases as
> following.
> IfExprLongColumnLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> vector2[i] : arg3Scalar);{code}
> IfExprLongScalarLongColumn: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : vector3[i]);{code}
> IfExprLongScalarLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : arg3Scalar);{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11529) Loop optimization for SIMD in IfExprLong*Long*
[
https://issues.apache.org/jira/browse/HIVE-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947628#comment-15947628
]
Hive QA commented on HIVE-11529:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12861019/HIVE-11529.5.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10540 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_case]
(batchId=54)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=141)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_case]
(batchId=151)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_case]
(batchId=120)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=172)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4445/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4445/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4445/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12861019 - PreCommit-HIVE-Build
> Loop optimization for SIMD in IfExprLong*Long*
> --
>
> Key: HIVE-11529
> URL: https://issues.apache.org/jira/browse/HIVE-11529
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
> Attachments: HIVE-11529.2.patch, HIVE-11529.3.patch,
> HIVE-11529.4.patch, HIVE-11529.5.patch, HIVE-11529.patch
>
>
> HIVE-10238 showed that ?: operator can be optimized with bitwise operators
> when it only handles LongColumnVectors. There are more similar cases as
> following.
> IfExprLongColumnLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> vector2[i] : arg3Scalar);{code}
> IfExprLongScalarLongColumn: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : vector3[i]);{code}
> IfExprLongScalarLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : arg3Scalar);{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11529) Loop optimization for SIMD in IfExprLong*Long*
[
https://issues.apache.org/jira/browse/HIVE-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946973#comment-15946973
]
Teddy Choi commented on HIVE-11529:
---
I made small changes and uploaded it. This fifth patch added more integration
tests. I'm waiting for its test results.
> Loop optimization for SIMD in IfExprLong*Long*
> --
>
> Key: HIVE-11529
> URL: https://issues.apache.org/jira/browse/HIVE-11529
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
> Attachments: HIVE-11529.2.patch, HIVE-11529.3.patch,
> HIVE-11529.4.patch, HIVE-11529.5.patch, HIVE-11529.patch
>
>
> HIVE-10238 showed that ?: operator can be optimized with bitwise operators
> when it only handles LongColumnVectors. There are more similar cases as
> following.
> IfExprLongColumnLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> vector2[i] : arg3Scalar);{code}
> IfExprLongScalarLongColumn: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : vector3[i]);{code}
> IfExprLongScalarLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : arg3Scalar);{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11529) Loop optimization for SIMD in IfExprLong*Long*
[
https://issues.apache.org/jira/browse/HIVE-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946152#comment-15946152
]
Carter Shanklin commented on HIVE-11529:
[~teddy.choi] did this ever make it in?
> Loop optimization for SIMD in IfExprLong*Long*
> --
>
> Key: HIVE-11529
> URL: https://issues.apache.org/jira/browse/HIVE-11529
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
> Attachments: HIVE-11529.2.patch, HIVE-11529.3.patch,
> HIVE-11529.4.patch, HIVE-11529.patch
>
>
> HIVE-10238 showed that ?: operator can be optimized with bitwise operators
> when it only handles LongColumnVectors. There are more similar cases as
> following.
> IfExprLongColumnLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> vector2[i] : arg3Scalar);{code}
> IfExprLongScalarLongColumn: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : vector3[i]);{code}
> IfExprLongScalarLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : arg3Scalar);{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11529) Loop optimization for SIMD in IfExprLong*Long*
[
https://issues.apache.org/jira/browse/HIVE-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855068#comment-15855068
]
Ferdinand Xu commented on HIVE-11529:
-
LGTM +1
> Loop optimization for SIMD in IfExprLong*Long*
> --
>
> Key: HIVE-11529
> URL: https://issues.apache.org/jira/browse/HIVE-11529
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
> Attachments: HIVE-11529.2.patch, HIVE-11529.3.patch,
> HIVE-11529.4.patch, HIVE-11529.patch
>
>
> HIVE-10238 showed that ?: operator can be optimized with bitwise operators
> when it only handles LongColumnVectors. There are more similar cases as
> following.
> IfExprLongColumnLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> vector2[i] : arg3Scalar);{code}
> IfExprLongScalarLongColumn: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : vector3[i]);{code}
> IfExprLongScalarLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : arg3Scalar);{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11529) Loop optimization for SIMD in IfExprLong*Long*
[
https://issues.apache.org/jira/browse/HIVE-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854101#comment-15854101
]
Hive QA commented on HIVE-11529:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12851155/HIVE-11529.4.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10211 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=117)
[join39.q,bucketsortoptimize_insert_7.q,vector_distinct_2.q,bucketmapjoin10.q,join11.q,union13.q,auto_sortmerge_join_16.q,windowing.q,union_remove_3.q,skewjoinopt7.q,stats7.q,annotate_stats_join.q,multi_insert_lateral_view.q,ptf_streaming.q,join_1to1.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_when_case]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_if_expr]
(batchId=10)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_char_simple]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_varchar_simple]
(batchId=153)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testWaitQueuePreemption
(batchId=277)
org.apache.hadoop.hive.ql.exec.vector.TestVectorizationContext.testIfConditionalExprs
(batchId=259)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3395/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3395/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3395/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12851155 - PreCommit-HIVE-Build
> Loop optimization for SIMD in IfExprLong*Long*
> --
>
> Key: HIVE-11529
> URL: https://issues.apache.org/jira/browse/HIVE-11529
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
> Attachments: HIVE-11529.2.patch, HIVE-11529.3.patch,
> HIVE-11529.4.patch, HIVE-11529.patch
>
>
> HIVE-10238 showed that ?: operator can be optimized with bitwise operators
> when it only handles LongColumnVectors. There are more similar cases as
> following.
> IfExprLongColumnLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> vector2[i] : arg3Scalar);{code}
> IfExprLongScalarLongColumn: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : vector3[i]);{code}
> IfExprLongScalarLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : arg3Scalar);{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11529) Loop optimization for SIMD in IfExprLong*Long*
[
https://issues.apache.org/jira/browse/HIVE-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853832#comment-15853832
]
Teddy Choi commented on HIVE-11529:
---
It brought up to 4x performance.
Before:
{noformat}
o.a.h.b.v.VectorizedLogicBench.IfExprLongColumnLongColumnBench.bench
avgt2 2201086483.000 ± NaN ns/op
o.a.h.b.v.VectorizedLogicBench.IfExprLongColumnLongScalarBench.bench
avgt2 1361940192.000 ± NaN ns/op
o.a.h.b.v.VectorizedLogicBench.IfExprLongColumnRepeatingLongColumnBench.bench
avgt2 2965654720.500 ± NaN ns/op
o.a.h.b.v.VectorizedLogicBench.IfExprLongScalarLongColumnBench.bench
avgt2 1252793715.000 ± NaN ns/op
o.a.h.b.v.VectorizedLogicBench.IfExprLongScalarLongScalarBench.bench
avgt2 525885607.500 ± NaN ns/op
o.a.h.b.v.VectorizedLogicBench.IfExprLongScalarRepeatingLongColumnBench.bench
avgt2 937117696.500 ± NaN ns/op
o.a.h.b.v.VectorizedLogicBench.IfExprRepeatingLongColumnLongColumnBench.bench
avgt2 2970268531.500 ± NaN ns/op
o.a.h.b.v.VectorizedLogicBench.IfExprRepeatingLongColumnLongScalarBench.bench
avgt2 1258292159.500 ± NaN ns/op
o.a.h.b.v.VectorizedLogicBench.IfExprRepeatingLongColumnRepeatingLongColumnBench.bench
avgt2 2028832459.000 ± NaN ns/op
{noformat}
After:
{noformat}
o.a.h.b.v.VectorizedLogicBench.IfExprLongColumnLongColumnBench.bench
avgt2 1905568359.500 ± NaN ns/op
o.a.h.b.v.VectorizedLogicBench.IfExprLongColumnLongScalarBench.bench
avgt2 311806588.500 ± NaN ns/op V
o.a.h.b.v.VectorizedLogicBench.IfExprLongColumnRepeatingLongColumnBench.bench
avgt2 2950095981.000 ± NaN ns/op
o.a.h.b.v.VectorizedLogicBench.IfExprLongScalarLongColumnBench.bench
avgt2 309752757.500 ± NaN ns/op V
o.a.h.b.v.VectorizedLogicBench.IfExprLongScalarLongScalarBench.bench
avgt2 227286994.000 ± NaN ns/op V
o.a.h.b.v.VectorizedLogicBench.IfExprLongScalarRepeatingLongColumnBench.bench
avgt2 380720489.500 ± NaN ns/op V
o.a.h.b.v.VectorizedLogicBench.IfExprRepeatingLongColumnLongColumnBench.bench
avgt2 2974550657.500 ± NaN ns/op
o.a.h.b.v.VectorizedLogicBench.IfExprRepeatingLongColumnLongScalarBench.bench
avgt2 385519086.000 ± NaN ns/op V
o.a.h.b.v.VectorizedLogicBench.IfExprRepeatingLongColumnRepeatingLongColumnBench.bench
avgt2 2079411174.500 ± NaN ns/op
{noformat}
> Loop optimization for SIMD in IfExprLong*Long*
> --
>
> Key: HIVE-11529
> URL: https://issues.apache.org/jira/browse/HIVE-11529
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
> Attachments: HIVE-11529.2.patch, HIVE-11529.3.patch,
> HIVE-11529.4.patch, HIVE-11529.patch
>
>
> HIVE-10238 showed that ?: operator can be optimized with bitwise operators
> when it only handles LongColumnVectors. There are more similar cases as
> following.
> IfExprLongColumnLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> vector2[i] : arg3Scalar);{code}
> IfExprLongScalarLongColumn: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : vector3[i]);{code}
> IfExprLongScalarLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : arg3Scalar);{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11529) Loop optimization for SIMD in IfExprLong*Long*
[
https://issues.apache.org/jira/browse/HIVE-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853275#comment-15853275
]
Hive QA commented on HIVE-11529:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12851085/HIVE-11529.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10211 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=125)
[table_access_keys_stats.q,bucketmapjoin11.q,auto_join4.q,mapjoin_decimal.q,join34.q,nullgroup.q,mergejoins_mixed.q,sort.q,stats8.q,auto_join28.q,join17.q,union17.q,skewjoinopt11.q,groupby1_map.q,load_dyn_part11.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_when_case]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_if_expr]
(batchId=10)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[hybridgrace_hashjoin_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=223)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testWaitQueuePreemption
(batchId=277)
org.apache.hadoop.hive.ql.exec.vector.TestVectorizationContext.testIfConditionalExprs
(batchId=259)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3388/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3388/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3388/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12851085 - PreCommit-HIVE-Build
> Loop optimization for SIMD in IfExprLong*Long*
> --
>
> Key: HIVE-11529
> URL: https://issues.apache.org/jira/browse/HIVE-11529
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
> Attachments: HIVE-11529.2.patch, HIVE-11529.3.patch, HIVE-11529.patch
>
>
> HIVE-10238 showed that ?: operator can be optimized with bitwise operators
> when it only handles LongColumnVectors. There are more similar cases as
> following.
> IfExprLongColumnLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> vector2[i] : arg3Scalar);{code}
> IfExprLongScalarLongColumn: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : vector3[i]);{code}
> IfExprLongScalarLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : arg3Scalar);{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11529) Loop optimization for SIMD in IfExprLong*Long*
[
https://issues.apache.org/jira/browse/HIVE-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991740#comment-14991740
]
Teddy Choi commented on HIVE-11529:
---
The performance increase is not consistent. I will investigate deeper and
update this patch.
> Loop optimization for SIMD in IfExprLong*Long*
> --
>
> Key: HIVE-11529
> URL: https://issues.apache.org/jira/browse/HIVE-11529
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
> Attachments: HIVE-11529.2.patch, HIVE-11529.patch
>
>
> HIVE-10238 showed that ?: operator can be optimized with bitwise operators
> when it only handles LongColumnVectors. There are more similar cases as
> following.
> IfExprLongColumnLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> vector2[i] : arg3Scalar);{code}
> IfExprLongScalarLongColumn: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : vector3[i]);{code}
> IfExprLongScalarLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : arg3Scalar);{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11529) Loop optimization for SIMD in IfExprLong*Long*
[
https://issues.apache.org/jira/browse/HIVE-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990001#comment-14990001
]
Hive QA commented on HIVE-11529:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12770543/HIVE-11529.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9768 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_tez_union_with_udf
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_with_udf
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5917/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5917/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5917/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12770543 - PreCommit-HIVE-TRUNK-Build
> Loop optimization for SIMD in IfExprLong*Long*
> --
>
> Key: HIVE-11529
> URL: https://issues.apache.org/jira/browse/HIVE-11529
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
> Attachments: HIVE-11529.2.patch, HIVE-11529.patch
>
>
> HIVE-10238 showed that ?: operator can be optimized with bitwise operators
> when it only handles LongColumnVectors. There are more similar cases as
> following.
> IfExprLongColumnLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> vector2[i] : arg3Scalar);{code}
> IfExprLongScalarLongColumn: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : vector3[i]);{code}
> IfExprLongScalarLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : arg3Scalar);{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11529) Loop optimization for SIMD in IfExprLong*Long*
[
https://issues.apache.org/jira/browse/HIVE-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14966108#comment-14966108
]
Teddy Choi commented on HIVE-11529:
---
Review request on https://reviews.apache.org/r/39503/
> Loop optimization for SIMD in IfExprLong*Long*
> --
>
> Key: HIVE-11529
> URL: https://issues.apache.org/jira/browse/HIVE-11529
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
> Labels: optimi
> Attachments: HIVE-11529.patch
>
>
> HIVE-10238 showed that ?: operator can be optimized with bitwise operators
> when it only handles LongColumnVectors. There are more similar cases as
> following.
> IfExprLongColumnLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> vector2[i] : arg3Scalar);{code}
> IfExprLongScalarLongColumn: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : vector3[i]);{code}
> IfExprLongScalarLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : arg3Scalar);{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11529) Loop optimization for SIMD in IfExprLong*Long*
[
https://issues.apache.org/jira/browse/HIVE-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14966290#comment-14966290
]
Hive QA commented on HIVE-11529:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12767710/HIVE-11529.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9657 tests executed
*Failed tests:*
{noformat}
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarDataNucleusUnCaching
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5717/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5717/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5717/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12767710 - PreCommit-HIVE-TRUNK-Build
> Loop optimization for SIMD in IfExprLong*Long*
> --
>
> Key: HIVE-11529
> URL: https://issues.apache.org/jira/browse/HIVE-11529
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Minor
> Attachments: HIVE-11529.patch
>
>
> HIVE-10238 showed that ?: operator can be optimized with bitwise operators
> when it only handles LongColumnVectors. There are more similar cases as
> following.
> IfExprLongColumnLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> vector2[i] : arg3Scalar);{code}
> IfExprLongScalarLongColumn: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : vector3[i]);{code}
> IfExprLongScalarLongScalar: {code}outputVector[i] = (vector1[i] == 1 ?
> arg2Scalar : arg3Scalar);{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
