[jira] [Commented] (HIVE-16029) COLLECT_SET and COLLECT_LIST does not return NULL in the result
[ https://issues.apache.org/jira/browse/HIVE-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020747#comment-16020747 ] Eric Lin commented on HIVE-16029: - Hi [~appodictic], I have modified test so that it passes, please help to review the code at https://reviews.apache.org/r/57009/. Thanks > COLLECT_SET and COLLECT_LIST does not return NULL in the result > --- > > Key: HIVE-16029 > URL: https://issues.apache.org/jira/browse/HIVE-16029 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Eric Lin >Assignee: Eric Lin >Priority: Minor > Attachments: HIVE-16029.2.patch, HIVE-16029.3.patch, HIVE-16029.patch > > > See the test case below: > {code} > 0: jdbc:hive2://localhost:1/default> select * from collect_set_test; > +-+ > | collect_set_test.a | > +-+ > | 1 | > | 2 | > | NULL| > | 4 | > | NULL| > +-+ > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,4] | > +---+ > {code} > The correct result should be: > {code} > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,null,4] | > +---+ > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16029) COLLECT_SET and COLLECT_LIST does not return NULL in the result
[ https://issues.apache.org/jira/browse/HIVE-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019525#comment-16019525 ] Hive QA commented on HIVE-16029: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12869250/HIVE-16029.3.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 10742 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5381/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5381/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5381/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12869250 - PreCommit-HIVE-Build > COLLECT_SET and COLLECT_LIST does not return NULL in the result > --- > > Key: HIVE-16029 > URL: https://issues.apache.org/jira/browse/HIVE-16029 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Eric Lin >Assignee: Eric Lin >Priority: Minor > Attachments: HIVE-16029.2.patch, HIVE-16029.3.patch, HIVE-16029.patch > > > See the test case below: > {code} > 0: jdbc:hive2://localhost:1/default> select * from collect_set_test; > +-+ > | collect_set_test.a | > +-+ > | 1 | > | 2 | > | NULL| > | 4 | > | NULL| > +-+ > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,4] | > +---+ > {code} > The correct result should be: > {code} > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,null,4] | > +---+ > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16029) COLLECT_SET and COLLECT_LIST does not return NULL in the result
[ https://issues.apache.org/jira/browse/HIVE-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974206#comment-15974206 ] Eric Lin commented on HIVE-16029: - Hi [~appodictic], Thanks for the suggestion. I am trying to run the test for TestCliDriver using below command under directory itests/qtest by following the documentation https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ#HiveDeveloperFAQ-Testing: {code} mvn test -Dtest=TestCliDriver {code} However, it kept failing with below error: {code} [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test (default-test) on project hive-it-qfile: ExecutionException: java.lang.RuntimeException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was /bin/sh -c cd /hadoop/code/hive/itests/qtest && /hadoop/jdk1.8.0_91/jre/bin/java -Xmx1024m -XX:MaxPermSize=256M -jar /hadoop/code/hive/itests/qtest/target/surefire/surefirebooter7738443094919274008.jar /hadoop/code/hive/itests/qtest/target/surefire/surefire4160478088421683107tmp /hadoop/code/hive/itests/qtest/target/surefire/surefire_05453129517537389906tmp [ERROR] -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test (default-test) on project hive-it-qfile: ExecutionException at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:414) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:357) Caused by: org.apache.maven.plugin.MojoFailureException: ExecutionException at org.apache.maven.plugin.surefire.SurefirePlugin.assertNoException(SurefirePlugin.java:262) at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:252) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:854) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:722) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) ... 19 more Caused by: org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:343) at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:178) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:990) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:824) ... 22 more Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called? Command was /bin/sh -c cd /hadoop/code/hive/itests/qtest && /hadoop/jdk1.8.0_91/jre/bin/java -Xmx1024m -XX:MaxPermSize=256M -jar /hadoop/code/hive/itests/qtest/target/surefire/surefirebooter7738443094919274008.jar /hadoop/code/hive/itests/qtest/target/surefire/surefire4160478088421683107tmp
[jira] [Commented] (HIVE-16029) COLLECT_SET and COLLECT_LIST does not return NULL in the result
[ https://issues.apache.org/jira/browse/HIVE-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969952#comment-15969952 ] Edward Capriolo commented on HIVE-16029: Code looks look, but some of the q test files run the explain command: https://builds.apache.org/job/PreCommit-HIVE-Build/4704/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_udaf_collect_set_/ You need to update the .q.out files so they do not fil > COLLECT_SET and COLLECT_LIST does not return NULL in the result > --- > > Key: HIVE-16029 > URL: https://issues.apache.org/jira/browse/HIVE-16029 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Eric Lin >Assignee: Eric Lin >Priority: Minor > Attachments: HIVE-16029.2.patch, HIVE-16029.patch > > > See the test case below: > {code} > 0: jdbc:hive2://localhost:1/default> select * from collect_set_test; > +-+ > | collect_set_test.a | > +-+ > | 1 | > | 2 | > | NULL| > | 4 | > | NULL| > +-+ > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,4] | > +---+ > {code} > The correct result should be: > {code} > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,null,4] | > +---+ > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16029) COLLECT_SET and COLLECT_LIST does not return NULL in the result
[ https://issues.apache.org/jira/browse/HIVE-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969949#comment-15969949 ] Hive QA commented on HIVE-16029: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12863552/HIVE-16029.2.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10579 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_collect_set] (batchId=8) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_order_null] (batchId=27) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[udaf_collect_set_2] (batchId=155) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=143) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[udaf_collect_set] (batchId=102) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4704/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4704/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4704/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12863552 - PreCommit-HIVE-Build > COLLECT_SET and COLLECT_LIST does not return NULL in the result > --- > > Key: HIVE-16029 > URL: https://issues.apache.org/jira/browse/HIVE-16029 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Eric Lin >Assignee: Eric Lin >Priority: Minor > Attachments: HIVE-16029.2.patch, HIVE-16029.patch > > > See the test case below: > {code} > 0: jdbc:hive2://localhost:1/default> select * from collect_set_test; > +-+ > | collect_set_test.a | > +-+ > | 1 | > | 2 | > | NULL| > | 4 | > | NULL| > +-+ > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,4] | > +---+ > {code} > The correct result should be: > {code} > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,null,4] | > +---+ > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16029) COLLECT_SET and COLLECT_LIST does not return NULL in the result
[ https://issues.apache.org/jira/browse/HIVE-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969931#comment-15969931 ] Eric Lin commented on HIVE-16029: - Review is also updated: https://reviews.apache.org/r/57009/. Please help to review and see if there is any other changes required. > COLLECT_SET and COLLECT_LIST does not return NULL in the result > --- > > Key: HIVE-16029 > URL: https://issues.apache.org/jira/browse/HIVE-16029 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Eric Lin >Assignee: Eric Lin >Priority: Minor > Attachments: HIVE-16029.2.patch, HIVE-16029.patch > > > See the test case below: > {code} > 0: jdbc:hive2://localhost:1/default> select * from collect_set_test; > +-+ > | collect_set_test.a | > +-+ > | 1 | > | 2 | > | NULL| > | 4 | > | NULL| > +-+ > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,4] | > +---+ > {code} > The correct result should be: > {code} > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,null,4] | > +---+ > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16029) COLLECT_SET and COLLECT_LIST does not return NULL in the result
[ https://issues.apache.org/jira/browse/HIVE-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884595#comment-15884595 ] Dudu Markovitz commented on HIVE-16029: --- Not a bug. NULL values are being ignored by all aggregate functions. > COLLECT_SET and COLLECT_LIST does not return NULL in the result > --- > > Key: HIVE-16029 > URL: https://issues.apache.org/jira/browse/HIVE-16029 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Eric Lin >Assignee: Eric Lin >Priority: Minor > Attachments: HIVE-16029.patch > > > See the test case below: > {code} > 0: jdbc:hive2://localhost:1/default> select * from collect_set_test; > +-+ > | collect_set_test.a | > +-+ > | 1 | > | 2 | > | NULL| > | 4 | > | NULL| > +-+ > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,4] | > +---+ > {code} > The correct result should be: > {code} > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,null,4] | > +---+ > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16029) COLLECT_SET and COLLECT_LIST does not return NULL in the result
[ https://issues.apache.org/jira/browse/HIVE-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884347#comment-15884347 ] Edward Capriolo commented on HIVE-16029: I do not thin you should change this now. This will change the behavior of many applications. I am -1 in the current form. I suggest you do this. CollectSet(a) <- original behaivor CollectSet(a,true) <- allow nulls. This way you get your feature and the result set you want and existing applications are not effected. > COLLECT_SET and COLLECT_LIST does not return NULL in the result > --- > > Key: HIVE-16029 > URL: https://issues.apache.org/jira/browse/HIVE-16029 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Eric Lin >Assignee: Eric Lin >Priority: Minor > Attachments: HIVE-16029.patch > > > See the test case below: > {code} > 0: jdbc:hive2://localhost:1/default> select * from collect_set_test; > +-+ > | collect_set_test.a | > +-+ > | 1 | > | 2 | > | NULL| > | 4 | > | NULL| > +-+ > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,4] | > +---+ > {code} > The correct result should be: > {code} > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,null,4] | > +---+ > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16029) COLLECT_SET and COLLECT_LIST does not return NULL in the result
[ https://issues.apache.org/jira/browse/HIVE-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882997#comment-15882997 ] Aihua Xu commented on HIVE-16029: - [~csun] You worked on the original feature. Not sure if it makes sense to include NULL in the result. Can you take a look? The set in java actually will also remove null. > COLLECT_SET and COLLECT_LIST does not return NULL in the result > --- > > Key: HIVE-16029 > URL: https://issues.apache.org/jira/browse/HIVE-16029 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Eric Lin >Assignee: Eric Lin >Priority: Minor > Attachments: HIVE-16029.patch > > > See the test case below: > {code} > 0: jdbc:hive2://localhost:1/default> select * from collect_set_test; > +-+ > | collect_set_test.a | > +-+ > | 1 | > | 2 | > | NULL| > | 4 | > | NULL| > +-+ > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,4] | > +---+ > {code} > The correct result should be: > {code} > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,null,4] | > +---+ > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16029) COLLECT_SET and COLLECT_LIST does not return NULL in the result
[ https://issues.apache.org/jira/browse/HIVE-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882467#comment-15882467 ] Hive QA commented on HIVE-16029: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12854312/HIVE-16029.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10258 tests executed *Failed tests:* {noformat} TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3749/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3749/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3749/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12854312 - PreCommit-HIVE-Build > COLLECT_SET and COLLECT_LIST does not return NULL in the result > --- > > Key: HIVE-16029 > URL: https://issues.apache.org/jira/browse/HIVE-16029 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Eric Lin >Assignee: Eric Lin >Priority: Minor > Attachments: HIVE-16029.patch > > > See the test case below: > {code} > 0: jdbc:hive2://localhost:1/default> select * from collect_set_test; > +-+ > | collect_set_test.a | > +-+ > | 1 | > | 2 | > | NULL| > | 4 | > | NULL| > +-+ > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,4] | > +---+ > {code} > The correct result should be: > {code} > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,null,4] | > +---+ > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16029) COLLECT_SET and COLLECT_LIST does not return NULL in the result
[ https://issues.apache.org/jira/browse/HIVE-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881671#comment-15881671 ] Eric Lin commented on HIVE-16029: - Review request sent: https://reviews.apache.org/r/57009/ > COLLECT_SET and COLLECT_LIST does not return NULL in the result > --- > > Key: HIVE-16029 > URL: https://issues.apache.org/jira/browse/HIVE-16029 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Eric Lin >Assignee: Eric Lin >Priority: Minor > Attachments: HIVE-16029.patch > > > See the test case below: > {code} > 0: jdbc:hive2://localhost:1/default> select * from collect_set_test; > +-+ > | collect_set_test.a | > +-+ > | 1 | > | 2 | > | NULL| > | 4 | > | NULL| > +-+ > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,4] | > +---+ > {code} > The correct result should be: > {code} > 0: jdbc:hive2://localhost:1/default> select collect_set(a) from > collect_set_test; > +---+ > | _c0 | > +---+ > | [1,2,null,4] | > +---+ > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)