[jira] [Commented] (HIVE-13637) Fold CASE into NVL when CBO optimized the plan
[
https://issues.apache.org/jira/browse/HIVE-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273086#comment-15273086
]
Hive QA commented on HIVE-13637:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12802079/HIVE-13637.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10010 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-join1.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/188/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/188/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-188/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12802079 - PreCommit-HIVE-MASTER-Build
> Fold CASE into NVL when CBO optimized the plan
> --
>
> Key: HIVE-13637
> URL: https://issues.apache.org/jira/browse/HIVE-13637
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13637.02.patch, HIVE-13637.patch, HIVE-13637.patch
>
>
> After HIVE-13068 goes in, folding CASE into NVL got disabled when CBO has
> optimized the plan, as it was done by ConstantPropagate in Hive. We need to
> enable it back.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13637) Fold CASE into NVL when CBO optimized the plan
[
https://issues.apache.org/jira/browse/HIVE-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15272921#comment-15272921
]
Jesus Camacho Rodriguez commented on HIVE-13637:
{noformat}
Test Name DurationAge
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket3 26 sec
1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez2
1 min 27 sec1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge9
21 sec 1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_load_dyn_part2
1.4 sec 1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_15
1.1 sec 1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
1.7 sec 1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_decimal_date
0.78 sec1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union_multiinsert
1 min 38 sec1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_sort_skew_1_23
1 min 49 sec1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats201.6 sec
1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_13
0.92 sec1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_19
0.83 sec1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_21
0.72 sec1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_reorder2
0.9 sec 1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_script_env_var1
0.77 sec1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32 6.5 sec 2
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
87 ms 2
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefaultEmptyString
10 sec 2
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
3 sec 2
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
5 sec 3
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
10 sec 4
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
40 sec 6
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
4.1 sec 6
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl
1 min 15 sec15
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
3 sec 19
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
11 sec 20
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
2 min 11 sec21
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_selectindate21 sec
25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avrocountemptytbl
10 sec 25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null 35 sec
25
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
20 sec 25
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
1 min 30 sec25
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore
2 sec 25
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore
0.5 sec 25
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
3.3 sec 25
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin 2 min 23 sec25
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
8.1 sec 25
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
1 min 8 sec 25
{noformat}
> Fold CASE into NVL when CBO optimized the plan
> --
>
> Key: HIVE-13637
> URL: https://issues.apache.org/jira/browse/HIVE-13637
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13637.02.patch, HIVE-13637.patch, HIVE-13637.patch
>
>
> After HIVE-13068 goes in, folding CASE into NVL got disabled when CBO has
> optimized the plan, as it was done by ConstantPropagate in Hive. We need to
> enable it back.
--
This message was sent by Atlassian JIRA
[jira] [Commented] (HIVE-13637) Fold CASE into NVL when CBO optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265960#comment-15265960 ] Ashutosh Chauhan commented on HIVE-13637: - +1 > Fold CASE into NVL when CBO optimized the plan > -- > > Key: HIVE-13637 > URL: https://issues.apache.org/jira/browse/HIVE-13637 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13637.patch, HIVE-13637.patch > > > After HIVE-13068 goes in, folding CASE into NVL got disabled when CBO has > optimized the plan, as it was done by ConstantPropagate in Hive. We need to > enable it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13637) Fold CASE into NVL when CBO optimized the plan
[
https://issues.apache.org/jira/browse/HIVE-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265758#comment-15265758
]
Hive QA commented on HIVE-13637:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801431/HIVE-13637.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/141/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/141/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-141/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-llap-tez ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/llap-tez/target/hive-llap-tez-2.1.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-llap-tez ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-llap-tez
---
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/llap-tez/target/hive-llap-tez-2.1.0-SNAPSHOT.jar
to
/home/hiveptest/.m2/repository/org/apache/hive/hive-llap-tez/2.1.0-SNAPSHOT/hive-llap-tez-2.1.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/llap-tez/pom.xml to
/home/hiveptest/.m2/repository/org/apache/hive/hive-llap-tez/2.1.0-SNAPSHOT/hive-llap-tez-2.1.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Spark Remote Client 2.1.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ spark-client ---
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/spark-client/target
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/spark-client (includes =
[datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
spark-client ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-client
---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
spark-client ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ spark-client ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ spark-client
---
[INFO] Compiling 28 source files to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/classes
[WARNING]
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientUtilities.java:
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientUtilities.java
uses or overrides a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientUtilities.java:
Recompile with -Xlint:deprecation for details.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/rpc/RpcDispatcher.java:
Some input files use unchecked or unsafe operations.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/rpc/RpcDispatcher.java:
Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
spark-client ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ spark-client ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf
[copy] Copying 15 files to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
spark-client ---
[INFO] Compiling 5 source files to
[jira] [Commented] (HIVE-13637) Fold CASE into NVL when CBO optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263859#comment-15263859 ] Jesus Camacho Rodriguez commented on HIVE-13637: I was also doubting... Problem is that Calcite does not support NVL natively, so we might miss folding optimization opportunities if we convert CASE into NVL. I could also create an additional rule to transform it at the end of query planning, but it seems an overkill. I have just uploaded a new patch that integrates the conversion in the parser instead of the ASTConverter, and thus integrates better with the work done in HIVE-13068 (it will need that patch to go in first though). > Fold CASE into NVL when CBO optimized the plan > -- > > Key: HIVE-13637 > URL: https://issues.apache.org/jira/browse/HIVE-13637 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13637.patch, HIVE-13637.patch > > > After HIVE-13068 goes in, folding CASE into NVL got disabled when CBO has > optimized the plan, as it was done by ConstantPropagate in Hive. We need to > enable it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13637) Fold CASE into NVL when CBO optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263126#comment-15263126 ] Ashutosh Chauhan commented on HIVE-13637: - Wondering if it were better to do this on Calcite tree instead? May be HiveRexUtil::simplify() > Fold CASE into NVL when CBO optimized the plan > -- > > Key: HIVE-13637 > URL: https://issues.apache.org/jira/browse/HIVE-13637 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13637.patch > > > After HIVE-13068 goes in, folding CASE into NVL got disabled when CBO has > optimized the plan, as it was done by ConstantPropagate in Hive. We need to > enable it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
