[jira] [Commented] (HIVE-4989) Consolidate and simplify vectorization code and test generation
[ https://issues.apache.org/jira/browse/HIVE-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13759259#comment-13759259 ] Jitendra Nath Pandey commented on HIVE-4989: The tests fail even without the patch. I have filed HIVE-5226 to address this issue. The issue is unrelated to this patch. > Consolidate and simplify vectorization code and test generation > --- > > Key: HIVE-4989 > URL: https://issues.apache.org/jira/browse/HIVE-4989 > Project: Hive > Issue Type: Sub-task >Affects Versions: vectorization-branch >Reporter: Tony Murphy >Assignee: Tony Murphy > Fix For: vectorization-branch > > Attachments: HIVE-4989.1-vectorization.patch, > HIVE-4989.2-vectorization.patch, HIVE-4989.revert.patch, > HIVE-4989-vectorization.patch > > > The current code generation is unwieldy to use and prone to errors. This > change consolidates all the code and test generation into a single location, > and removes the need to manually place files which can lead to missing or > incomplete code or tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4989) Consolidate and simplify vectorization code and test generation
[
https://issues.apache.org/jira/browse/HIVE-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13759176#comment-13759176
]
Ashutosh Chauhan commented on HIVE-4989:
I randomly ran few tests from the above list (namely
orc_dictionary_threshold.q,orc_create.q,orc_empty_strings.q,orc_ends_with_nulls.q).
All of them failed with follow trace:
{code}
[junit] Failed query: orc_dictionary_threshold.q
[junit] java.lang.AssertionError
[junit] at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:260)
[junit] at
org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:248)
[junit] at
org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:242)
[junit] at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:170)
[junit] at
org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:439)
[junit] at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:519)
[junit] at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:495)
[junit] at
org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
[junit] at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1487)
[junit] at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
[junit] at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
[junit] at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
[junit] at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
[junit] at
org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:815)
[junit] at
org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:135)
[junit] at
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_dictionary_threshold(TestCliDriver.java:111)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit] at java.lang.reflect.Method.invoke(Method.java:597)
[junit] at junit.framework.TestCase.runTest(TestCase.java:154)
[junit] at junit.framework.TestCase.runBare(TestCase.java:127)
[junit] at junit.framework.TestResult$1.protect(TestResult.java:106)
[junit] at junit.framework.TestResult.runProtected(TestResult.java:124)
[junit] at junit.framework.TestResult.run(TestResult.java:109)
[junit] at junit.framework.TestCase.run(TestCase.java:118)
[junit] at junit.framework.TestSuite.runTest(TestSuite.java:208)
[junit] at junit.framework.TestSuite.run(TestSuite.java:203)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
{code}
I didn't run them on branch without patch. If they are failing even on branch,
we can take those up in separate jira.
> Consolidate and simplify vectorization code and test generation
> ---
>
> Key: HIVE-4989
> URL: https://issues.apache.org/jira/browse/HIVE-4989
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: vectorization-branch
>Reporter: Tony Murphy
>Assignee: Tony Murphy
> Fix For: vectorization-branch
>
> Attachments: HIVE-4989.1-vectorization.patch,
> HIVE-4989.2-vectorization.patch, HIVE-4989.revert.patch,
> HIVE-4989-vectorization.patch
>
>
> The current code generation is unwieldy to use and prone to errors. This
> change consolidates all the code and test generation into a single location,
> and removes the need to manually place files which can lead to missing or
> incomplete code or tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4989) Consolidate and simplify vectorization code and test generation
[
https://issues.apache.org/jira/browse/HIVE-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13758649#comment-13758649
]
Hive QA commented on HIVE-4989:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12601466/HIVE-4989.2-vectorization.patch
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 3716 tests
executed
*Failed tests:*
{noformat}
org.apache.hcatalog.pig.TestE2EScenarios.testReadOrcAndRCFromPig
org.apache.hcatalog.pig.TestOrcHCatLoaderComplexSchema.testTupleInBagInTupleInBag
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testEmptyFile
org.apache.hcatalog.pig.TestOrcHCatStorer.testStoreBasicTable
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_date_serde
org.apache.hcatalog.pig.TestOrcHCatStorer.testStorePartitionedTable
org.apache.hcatalog.pig.TestOrcHCatLoader.testReadPartitionedBasic
org.apache.hcatalog.pig.TestOrcHCatLoader.testReadDataBasic
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testDefaultTypes
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_ends_with_nulls
org.apache.hcatalog.pig.TestOrcHCatLoaderComplexSchema.testMapWithComplexData
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_empty_strings
org.apache.hcatalog.fileformats.TestOrcDynamicPartitioned.testHCatDynamicPartitionedTable
org.apache.hcatalog.fileformats.TestOrcDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testMROutput
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_create
org.apache.hcatalog.pig.TestOrcHCatStorer.testStoreTableMulti
org.apache.hcatalog.pig.TestOrcHCatLoader.testProjectionsBasic
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input4
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testInOutFormat
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_dictionary_threshold
org.apache.hcatalog.pig.TestOrcHCatLoaderComplexSchema.testSyntheticComplexSchema
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_plan_json
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/617/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/617/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
> Consolidate and simplify vectorization code and test generation
> ---
>
> Key: HIVE-4989
> URL: https://issues.apache.org/jira/browse/HIVE-4989
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: vectorization-branch
>Reporter: Tony Murphy
>Assignee: Tony Murphy
> Fix For: vectorization-branch
>
> Attachments: HIVE-4989.1-vectorization.patch,
> HIVE-4989.2-vectorization.patch, HIVE-4989.revert.patch,
> HIVE-4989-vectorization.patch
>
>
> The current code generation is unwieldy to use and prone to errors. This
> change consolidates all the code and test generation into a single location,
> and removes the need to manually place files which can lead to missing or
> incomplete code or tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4989) Consolidate and simplify vectorization code and test generation
[ https://issues.apache.org/jira/browse/HIVE-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13742525#comment-13742525 ] Tony Murphy commented on HIVE-4989: --- Not sure what happened with the last patch, but the formatting looks bad. i've regenerated the patch, manually inspected it, and successfully applied it after the revert patch. > Consolidate and simplify vectorization code and test generation > --- > > Key: HIVE-4989 > URL: https://issues.apache.org/jira/browse/HIVE-4989 > Project: Hive > Issue Type: Sub-task >Affects Versions: vectorization-branch >Reporter: Tony Murphy >Assignee: Tony Murphy > Fix For: vectorization-branch > > Attachments: HIVE-4989.1-vectorization.patch, HIVE-4989.revert.patch, > HIVE-4989-vectorization.patch > > > The current code generation is unwieldy to use and prone to errors. This > change consolidates all the code and test generation into a single location, > and removes the need to manually place files which can lead to missing or > incomplete code or tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4989) Consolidate and simplify vectorization code and test generation
[ https://issues.apache.org/jira/browse/HIVE-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739403#comment-13739403 ] Teddy Choi commented on HIVE-4989: -- [~ashutoshc], I could not compile the latest code on vectorization branch. I have double checked it. It seems like there was an error in applying the patch. Please check it again. :) * ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/templates/CodeGen.java : the old location. * ql/src/gen/vectorization/org/apache/hadoop/hive/ql/exec/vector/gen/CodeGen.java : expected location on https://reviews.apache.org/r/13274/diff/ * ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/templates/CodeGen.java : actual location in the vectorization branch on https://github.com/apache/hive/commit/e6f59f5d0711c52badc89868e4178a1b2ef54e53 > Consolidate and simplify vectorization code and test generation > --- > > Key: HIVE-4989 > URL: https://issues.apache.org/jira/browse/HIVE-4989 > Project: Hive > Issue Type: Sub-task >Affects Versions: vectorization-branch >Reporter: Tony Murphy >Assignee: Tony Murphy > Fix For: vectorization-branch > > Attachments: HIVE-4989-vectorization.patch > > > The current code generation is unwieldy to use and prone to errors. This > change consolidates all the code and test generation into a single location, > and removes the need to manually place files which can lead to missing or > incomplete code or tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4989) Consolidate and simplify vectorization code and test generation
[ https://issues.apache.org/jira/browse/HIVE-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13729754#comment-13729754 ] Tony Murphy commented on HIVE-4989: --- https://reviews.apache.org/r/13274/ > Consolidate and simplify vectorization code and test generation > --- > > Key: HIVE-4989 > URL: https://issues.apache.org/jira/browse/HIVE-4989 > Project: Hive > Issue Type: Sub-task >Affects Versions: vectorization-branch >Reporter: Tony Murphy >Assignee: Tony Murphy > Fix For: vectorization-branch > > Attachments: HIVE-4989-vectorization.patch > > > The current code generation is unwieldy to use and prone to errors. This > change consolidates all the code and test generation into a single location, > and removes the need to manually place files which can lead to missing or > incomplete code or tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4989) Consolidate and simplify vectorization code and test generation
[ https://issues.apache.org/jira/browse/HIVE-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13729745#comment-13729745 ] Tony Murphy commented on HIVE-4989: --- This patch should be good to go. HIVE-4971 covers the testVectorUDFUnixTimeStampLong failure. > Consolidate and simplify vectorization code and test generation > --- > > Key: HIVE-4989 > URL: https://issues.apache.org/jira/browse/HIVE-4989 > Project: Hive > Issue Type: Sub-task >Affects Versions: vectorization-branch >Reporter: Tony Murphy >Assignee: Tony Murphy > Fix For: vectorization-branch > > Attachments: HIVE-4989-vectorization.patch > > > The current code generation is unwieldy to use and prone to errors. This > change consolidates all the code and test generation into a single location, > and removes the need to manually place files which can lead to missing or > incomplete code or tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4989) Consolidate and simplify vectorization code and test generation
[
https://issues.apache.org/jira/browse/HIVE-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13728380#comment-13728380
]
Hive QA commented on HIVE-4989:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12595673/HIVE-4989-vectorization.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 3591 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorTimestampExpressions.testVectorUDFUnixTimeStampLong
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/291/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/291/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
> Consolidate and simplify vectorization code and test generation
> ---
>
> Key: HIVE-4989
> URL: https://issues.apache.org/jira/browse/HIVE-4989
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: vectorization-branch
>Reporter: Tony Murphy
>Assignee: Tony Murphy
> Fix For: vectorization-branch
>
> Attachments: HIVE-4989-vectorization.patch
>
>
> The current code generation is unwieldy to use and prone to errors. This
> change consolidates all the code and test generation into a single location,
> and removes the need to manually place files which can lead to missing or
> incomplete code or tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
