[jira] [Commented] (HIVE-4669) Make username available to semantic analyzer hooks
[
https://issues.apache.org/jira/browse/HIVE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13784854#comment-13784854
]
Hive QA commented on HIVE-4669:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12606432/HIVE-4669.4.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4044 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1005/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1005/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
Make username available to semantic analyzer hooks
--
Key: HIVE-4669
URL: https://issues.apache.org/jira/browse/HIVE-4669
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.10.0, 0.11.0
Reporter: Shreepadma Venugopalan
Assignee: Shreepadma Venugopalan
Attachments: HIVE-4669.1.patch, HIVE-4669.2.patch, HIVE-4669.3.patch,
HIVE-4669.4.patch
Make username available to the semantic analyzer hooks.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kanwaljit Singh updated HIVE-4605: -- Summary: Hive job fails while closing reducer output - Unable to rename (was: hive job fails when insert overwrite ORC table) Hive job fails while closing reducer output - Unable to rename -- Key: HIVE-4605 URL: https://issues.apache.org/jira/browse/HIVE-4605 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux Hadoop 1.1.2 Reporter: Link Qian Assignee: Brock Noland 1, create a table with ORC storage model create table iparea_analysis_orc (network int, ip string, ) stored as ORC; 2, insert table iparea_analysis_orc select network, ip, , the script success, but failed after add *OVERWRITE* keyword. the main error log list as here. ava.lang.RuntimeException: Hive Runtime Error while closing operators: Unable to rename output from: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 to: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 to: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197) at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309) ... 7 more -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13784874#comment-13784874 ] Kanwaljit Singh commented on HIVE-4605: --- Observed for text files also. There were as many as 8 reducers running concurrently. Hive job fails while closing reducer output - Unable to rename -- Key: HIVE-4605 URL: https://issues.apache.org/jira/browse/HIVE-4605 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux Hadoop 1.1.2 Reporter: Link Qian Assignee: Brock Noland 1, create a table with ORC storage model create table iparea_analysis_orc (network int, ip string, ) stored as ORC; 2, insert table iparea_analysis_orc select network, ip, , the script success, but failed after add *OVERWRITE* keyword. the main error log list as here. ava.lang.RuntimeException: Hive Runtime Error while closing operators: Unable to rename output from: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 to: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 to: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197) at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309) ... 7 more -- This message was sent by Atlassian JIRA (v6.1#6144)
hive 0.12 release update
Short summary: I hope to have a release candidate out this weekend or by Monday. I have setup unit tests runs for hive 0.12 branch - https://builds.apache.org/job/Hive-branch-0.12-hadoop1/ https://builds.apache.org/job/Hive-branch-0.12-hadoop2/ There were some unit test failures, caused by test issues. The fixes for those should be in by tomorrow ( HIVE-5419, HIVE-5426). There are other blocker issues for which fix should be committed by tomorrow - HIVE-5394, HIVE-5401, HIVE-5364. There is a jira marked as blocker that does not have a fix yet - HIVE-5235 . But that is not a regression, so I think we can have an RC without that if it does not get fixed in next day or two. There is also another jira HIVE-5087 that might get resolved only by Friday. Thanks, Thejas -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Commented] (HIVE-5426) TestThriftBinaryCLIService tests fail on branch 0.12
[
https://issues.apache.org/jira/browse/HIVE-5426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13784909#comment-13784909
]
Hive QA commented on HIVE-5426:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12606436/HIVE-5426.1.patch
{color:green}SUCCESS:{color} +1 4044 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1006/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1006/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
TestThriftBinaryCLIService tests fail on branch 0.12
Key: HIVE-5426
URL: https://issues.apache.org/jira/browse/HIVE-5426
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0
Reporter: Thejas M Nair
Priority: Blocker
Attachments: HIVE-5426.1.patch
Two tests of TestThriftBinaryCLIService are failing in branch 0.12.
See
https://builds.apache.org/job/Hive-branch-0.12-hadoop1/lastCompletedBuild/testReport/
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HIVE-5432) self join for a table with serde definition fails with classNotFoundException, single queries work fine
Nitin Pawar created HIVE-5432: - Summary: self join for a table with serde definition fails with classNotFoundException, single queries work fine Key: HIVE-5432 URL: https://issues.apache.org/jira/browse/HIVE-5432 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.11.0 Environment: rhel6.4 Reporter: Nitin Pawar Steps to reproduce hive add jar /home/hive/udfs/hive-serdes-1.0-SNAPSHOT.jar; Added /home/hive/udfs/hive-serdes-1.0-SNAPSHOT.jar to class path Added resource: /home/hive/udfs/hive-serdes-1.0-SNAPSHOT.jar hive create table if not exists test(a string,b string) ROW FORMAT SERDE 'com.cloudera.hive.serde.JSONSerDe'; OK Time taken: 0.159 seconds hive load data local inpath '/tmp/1' overwrite into table test; Copying data from file:/tmp/1 Copying file: file:/tmp/1 Loading data to table default.test Table default.test stats: [num_partitions: 0, num_files: 1, num_rows: 0, total_size: 51, raw_data_size: 0] OK Time taken: 0.659 seconds hive select a from test; Total MapReduce jobs = 1 Launching Job 1 out of 1 ... ... hive select * from (select b from test where a=test)x join (select b from test where a=test1)y on (x.b = y.b); Total MapReduce jobs = 1 setting HADOOP_USER_NAMEhive Execution log at: /tmp/hive/.log java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDe Continuing ... 2013-10-03 05:13:00 Starting to launch local task to process map join; maximum memory = 1065484288 org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception nulljava.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.FetchOperator.getRowInspectorFromTable(FetchOperator.java:230) at org.apache.hadoop.hive.ql.exec.FetchOperator.getOutputObjectInspector(FetchOperator.java:595) at org.apache.hadoop.hive.ql.exec.MapredLocalTask.initializeOperators(MapredLocalTask.java:406) at org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:290) at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:682) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:160) at org.apache.hadoop.hive.ql.exec.FetchOperator.getOutputObjectInspector(FetchOperator.java:631) at org.apache.hadoop.hive.ql.exec.MapredLocalTask.initializeOperators(MapredLocalTask.java:406) at org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:290) at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:682) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:160) Execution failed with exit status: 2 Obtaining error information Task failed! Task ID: -- This message was sent by Atlassian JIRA (v6.1#6144)
Self join issue
Hi, I just raised a ticket for a table with self join query. Table is created with json serde provided by cloudera. When I run a single query on the table like select col from table where col='xyz', this works perfectly fine with a mapreduce job. but when I try to run the query of self join on the table it says serde not found on query parsing. i have mentioned the steps in detail on JIRA HIVE-5432https://issues.apache.org/jira/browse/HIVE-5432 . Can somebody tell what's special when the query is parsed for join and stand alone query? Due to this issue, I have to create temporary tables and make sure I clean them up myself after the jobs are over. Thanks, Nitin Pawar
[jira] [Commented] (HIVE-5429) HiveVarcharWritable length not reset when value is changed
[
https://issues.apache.org/jira/browse/HIVE-5429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13784949#comment-13784949
]
Hive QA commented on HIVE-5429:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12606447/HIVE-5429.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4048 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1007/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1007/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
HiveVarcharWritable length not reset when value is changed
--
Key: HIVE-5429
URL: https://issues.apache.org/jira/browse/HIVE-5429
Project: Hive
Issue Type: Bug
Components: Types
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-5429.1.patch
HiveVarcharWritable has a field to save the character length when it is
determined the first time. There are a couple of setter methods where this
length field is not reset when the string value is changed.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5400) Allow admins to disable compile and other commands
[
https://issues.apache.org/jira/browse/HIVE-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785019#comment-13785019
]
Hive QA commented on HIVE-5400:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12606456/HIVE-5400.patch
{color:green}SUCCESS:{color} +1 4046 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1008/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1008/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
Allow admins to disable compile and other commands
--
Key: HIVE-5400
URL: https://issues.apache.org/jira/browse/HIVE-5400
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Assignee: Edward Capriolo
Attachments: HIVE-5400.patch, HIVE-5400.patch, HIVE-5400.patch
From here:
https://issues.apache.org/jira/browse/HIVE-5253?focusedCommentId=13782220page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13782220
I think we should afford admins who want to disable this functionality the
ability to do so. Since such admins might want to disable other commands such
as add or dfs, it wouldn't be much trouble to allow them to do this as well.
For example we could have a configuration option hive.available.commands
(or similar) which specified add,set,delete,reset, etc by default. Then check
this value in CommandProcessorFactory. It would probably make sense to add
this property to the restrict list.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5410) Hive command line option --auxpath still does not work post HIVE-5363
[
https://issues.apache.org/jira/browse/HIVE-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785032#comment-13785032
]
Hudson commented on HIVE-5410:
--
ABORTED: Integrated in Hive-branch-0.12-hadoop2 #2 (See
[https://builds.apache.org/job/Hive-branch-0.12-hadoop2/2/])
HIVE-5410 : Hive command line option --auxpath still does not work post
HIVE-5363 (Brock Noland via Thejas Nair) (thejas:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528635)
* /hive/branches/branch-0.12/bin/hive
Hive command line option --auxpath still does not work post HIVE-5363
-
Key: HIVE-5410
URL: https://issues.apache.org/jira/browse/HIVE-5410
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Blocker
Fix For: 0.12.0
Attachments: HIVE-5410.patch
In short, AUX_PARAM is set to:
{noformat}
$ echo file:///etc/passwd | sed 's/:/,file:\/\//g'
file,file:/etc/passwd
{noformat}
which is invalid because file is not a real file.
NO PRECOMMIT TESTS (since this is not tested)
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5363) HIVE-3978 broke the command line option --auxpath
[
https://issues.apache.org/jira/browse/HIVE-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785033#comment-13785033
]
Hudson commented on HIVE-5363:
--
ABORTED: Integrated in Hive-branch-0.12-hadoop2 #2 (See
[https://builds.apache.org/job/Hive-branch-0.12-hadoop2/2/])
HIVE-5410 : Hive command line option --auxpath still does not work post
HIVE-5363 (Brock Noland via Thejas Nair) (thejas:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528635)
* /hive/branches/branch-0.12/bin/hive
HIVE-3978 broke the command line option --auxpath
-
Key: HIVE-5363
URL: https://issues.apache.org/jira/browse/HIVE-5363
Project: Hive
Issue Type: Improvement
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Blocker
Fix For: 0.12.0
Attachments: HIVE-5363.patch
HIVE-3978 changed the seperator for HIVE_AUX_JARS_PATH to : from ,. However,
it's expected that it's , later on in the script, specifically here:
{noformat}
AUX_CLASSPATH=${HIVE_AUX_JARS_PATH}
AUX_PARAM=file://${HIVE_AUX_JARS_PATH}
AUX_PARAM=`echo $AUX_PARAM | sed 's/,/,file:\/\//g'
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5391) make ORC predicate pushdown work with vectorization
[
https://issues.apache.org/jira/browse/HIVE-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785067#comment-13785067
]
Hive QA commented on HIVE-5391:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12606460/HIVE-5391.01.patch
{color:red}ERROR:{color} -1 due to 35 failed/errored test(s), 4044 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_date_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_create
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_dictionary_threshold
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_diff_part_cols
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_empty_files
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_empty_strings
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_ends_with_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_varchar_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testDefaultTypes
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testInOutFormat
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testMROutput
org.apache.hcatalog.fileformats.TestOrcDynamicPartitioned.testHCatDynamicPartitionedTable
org.apache.hcatalog.fileformats.TestOrcDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask
org.apache.hcatalog.pig.TestOrcHCatLoader.testProjectionsBasic
org.apache.hcatalog.pig.TestOrcHCatLoader.testReadDataBasic
org.apache.hcatalog.pig.TestOrcHCatLoader.testReadPartitionedBasic
org.apache.hcatalog.pig.TestOrcHCatStorer.testStoreBasicTable
org.apache.hcatalog.pig.TestOrcHCatStorer.testStorePartitionedTable
org.apache.hcatalog.pig.TestOrcHCatStorer.testStoreTableMulti
org.apache.hive.hcatalog.fileformats.TestOrcDynamicPartitioned.testHCatDynamicPartitionedTable
org.apache.hive.hcatalog.fileformats.TestOrcDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask
org.apache.hive.hcatalog.hbase.TestHBaseInputFormat
org.apache.hive.hcatalog.pig.TestE2EScenarios.testReadOrcAndRCFromPig
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testProjectionsBasic
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataBasic
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadPartitionedBasic
org.apache.hive.hcatalog.pig.TestOrcHCatLoaderComplexSchema.testMapWithComplexData
org.apache.hive.hcatalog.pig.TestOrcHCatLoaderComplexSchema.testSyntheticComplexSchema
org.apache.hive.hcatalog.pig.TestOrcHCatLoaderComplexSchema.testTupleInBagInTupleInBag
org.apache.hive.hcatalog.pig.TestOrcHCatStorer.testStoreBasicTable
org.apache.hive.hcatalog.pig.TestOrcHCatStorer.testStorePartitionedTable
org.apache.hive.hcatalog.pig.TestOrcHCatStorer.testStoreTableMulti
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1009/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1009/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 35 tests failed
{noformat}
This message is automatically generated.
make ORC predicate pushdown work with vectorization
---
Key: HIVE-5391
URL: https://issues.apache.org/jira/browse/HIVE-5391
Project: Hive
Issue Type: Improvement
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HIVE-5391.01.patch, HIVE-5391.01-vectorization.patch,
HIVE-5391.patch, HIVE-5391-vectorization.patch
Vectorized execution doesn't utilize ORC predicate pushdown. It should.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-4850) Implement vectorized JOIN operators
[ https://issues.apache.org/jira/browse/HIVE-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated HIVE-4850: --- Attachment: HIVE-4850.2.patch This is a working implementation based on current trunk. It is simpler than the .1 patch in as it delegates the JOIN entirely to the row-mode MapJoinOperator. The vectorized operator is literally calling the row-mode implementaiton for each row in the input batch and collects the row-mode forward into the output batch. This is not as bad as it seems because the JOIN operators has to resort to row-mode operations anyway, due to the small tables (hashtables) being row-mode (objects and object-inspectors). By delegating the entire join logic to the row mode we piggyback on the correctness of exiting implementation. I do plan to come up with a full-vectorized mode implementation but that would require changes to the hash table creation-serialization. Note that the filtering and key evaluation of the big table *does* use vectorized operators. the row mode applies only to the key HT lookup and to the JOIN logic. Implement vectorized JOIN operators --- Key: HIVE-4850 URL: https://issues.apache.org/jira/browse/HIVE-4850 Project: Hive Issue Type: Sub-task Reporter: Remus Rusanu Assignee: Remus Rusanu Attachments: HIVE-4850.1.patch, HIVE-4850.2.patch Easysauce -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 13059: HIVE-4850 Implement vector mode map join
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13059/ --- (Updated Oct. 3, 2013, 2:17 p.m.) Review request for hive, Eric Hanson and Jitendra Pandey. Bugs: HIVE-4850 https://issues.apache.org/jira/browse/HIVE-4850 Repository: hive-git Description --- This is not the final iteration, but I thought is easier to discuss it with a review. This implementation works, handles multiple aliases and multiple values per key. The implementation uses the exiting hash tables saved by the local task for the map join, which are row mode hash tables (have row mode keys and store row mode writable object values). Going forward we should avoid the size-of-big-table conversions of big table keys to row-mode and conversion of small table values to vector data. This would require either converting on-the-fly the hash tables to vector friendly ones (when loaded) or changing the local task tahstable sink to create a vectorization friendly hash. First approach may have memory consumption problems (potentially two hash tables end up in memory, would have to stream the transformation or transform as reading from serialized format... nasty). Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java d320b47 ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java 86db044 ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 153b8ea ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 8ab5395 ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java cde1a59 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ColumnVector.java 8b4c615 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorColumnAssign.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorColumnAssignFactory.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorHashKeyWrapperBatch.java 9955d09 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorReduceSinkOperator.java 6df3551 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 02ebe14 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatch.java ff13f89 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpressionWriterFactory.java 9e189c9 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java df1c5a6 ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java a72ec8b Diff: https://reviews.apache.org/r/13059/diff/ Testing --- Manually run some join queries on alltypes_orc table. Thanks, Remus Rusanu
Re: Review Request 13059: HIVE-4850 Implement vector mode map join
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13059/ --- (Updated Oct. 3, 2013, 2:20 p.m.) Review request for hive, Eric Hanson and Jitendra Pandey. Bugs: HIVE-4850 https://issues.apache.org/jira/browse/HIVE-4850 Repository: hive-git Description (updated) --- This is a working implementation based on current trunk. It is simpler than the .1 patch in as it delegates the JOIN entirely to the row-mode MapJoinOperator. The vectorized operator is literally calling the row-mode implementaiton for each row in the input batch and collects the row-mode forward into the output batch. This is not as bad as it seems because the JOIN operators has to resort to row-mode operations anyway, due to the small tables (hashtables) being row-mode (objects and object-inspectors). By delegating the entire join logic to the row mode we piggyback on the correctness of exiting implementation. I do plan to come up with a full-vectorized mode implementation but that would require changes to the hash table creation-serialization. Note that the filtering and key evaluation of the big table does use vectorized operators. the row mode applies only to the key HT lookup and to the JOIN logic Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java d320b47 ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java 86db044 ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 153b8ea ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 8ab5395 ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java cde1a59 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ColumnVector.java 8b4c615 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorColumnAssign.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorColumnAssignFactory.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorHashKeyWrapperBatch.java 9955d09 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorReduceSinkOperator.java 6df3551 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 02ebe14 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatch.java ff13f89 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpressionWriterFactory.java 9e189c9 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java df1c5a6 ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java a72ec8b Diff: https://reviews.apache.org/r/13059/diff/ Testing --- Manually run some join queries on alltypes_orc table. Thanks, Remus Rusanu
[jira] [Commented] (HIVE-4850) Implement vectorized JOIN operators
[ https://issues.apache.org/jira/browse/HIVE-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785228#comment-13785228 ] Remus Rusanu commented on HIVE-4850: I have updated the reviewboard diff https://reviews.apache.org/r/13059/ Implement vectorized JOIN operators --- Key: HIVE-4850 URL: https://issues.apache.org/jira/browse/HIVE-4850 Project: Hive Issue Type: Sub-task Reporter: Remus Rusanu Assignee: Remus Rusanu Attachments: HIVE-4850.1.patch, HIVE-4850.2.patch Easysauce -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-3065) New lines in columns can cause problems even when using sequence files
[ https://issues.apache.org/jira/browse/HIVE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785229#comment-13785229 ] Andrew Ahmad commented on HIVE-3065: This problem still exists in 0.10.0. I'm using RCFile and ran into this issue today. Not sure about 0.11.0 as I'm limited to the packages available in the CDH distribution. New lines in columns can cause problems even when using sequence files -- Key: HIVE-3065 URL: https://issues.apache.org/jira/browse/HIVE-3065 Project: Hive Issue Type: Bug Affects Versions: 0.7.1, 0.8.1 Reporter: Joey Echeverria When using sequence files as the container format, I'd expect to be able to embed new lines in a column. However, this causes problems when the data is output if the newlines aren't manually stripped or escaped. This tends to show up as each row of output generating two (or more) rows with nulls after the column with a new line and nulls for the empty columns on the second row. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[
https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785249#comment-13785249
]
Hive QA commented on HIVE-5394:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12606459/HIVE-5394.3.patch
{color:green}SUCCESS:{color} +1 4018 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1010/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1010/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ObjectInspectorConverters.getConvertedOI() does not return the correct object
inspector for primitive type.
---
Key: HIVE-5394
URL: https://issues.apache.org/jira/browse/HIVE-5394
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Priority: Blocker
Attachments: HIVE-5394.12.branch.txt, HIVE-5394.1.patch,
HIVE-5394.2.patch.txt, HIVE-5394.3.patch
The code currently returns settable type of the input primitive object
inspector where as it should return settable type of output object inspector
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-4669) Make username available to semantic analyzer hooks
[ https://issues.apache.org/jira/browse/HIVE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785254#comment-13785254 ] Brock Noland commented on HIVE-4669: +1 Make username available to semantic analyzer hooks -- Key: HIVE-4669 URL: https://issues.apache.org/jira/browse/HIVE-4669 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0, 0.11.0 Reporter: Shreepadma Venugopalan Assignee: Shreepadma Venugopalan Attachments: HIVE-4669.1.patch, HIVE-4669.2.patch, HIVE-4669.3.patch, HIVE-4669.4.patch Make username available to the semantic analyzer hooks. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5087) Rename npath UDF to matchpath
[ https://issues.apache.org/jira/browse/HIVE-5087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785256#comment-13785256 ] Brock Noland commented on HIVE-5087: I think we should just commit this. Picking a name of the air is better than using one we know has has legal issues. Rename npath UDF to matchpath - Key: HIVE-5087 URL: https://issues.apache.org/jira/browse/HIVE-5087 Project: Hive Issue Type: Bug Reporter: Edward Capriolo Assignee: Edward Capriolo Priority: Blocker Fix For: 0.12.0 Attachments: HIVE-5087.1.patch.txt, HIVE-5087.99.patch.txt, HIVE-5087-matchpath.1.patch.txt, HIVE-5087.patch.txt, HIVE-5087.patch.txt, regex_path.diff -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[ https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5394: --- Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Hari! ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. --- Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Blocker Fix For: 0.13.0 Attachments: HIVE-5394.12.branch.txt, HIVE-5394.1.patch, HIVE-5394.2.patch.txt, HIVE-5394.3.patch The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5087) Rename npath UDF to matchpath
[ https://issues.apache.org/jira/browse/HIVE-5087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785282#comment-13785282 ] Edward Capriolo commented on HIVE-5087: --- I am back under the opinion we should just remove this UDF. You could make a sequel to 'office space' based on the story behind this UDF 'yea... im going to need you to come in on Saturday and rename this udf' 'yea...im going to need you to come in on sunday because its saturday and I dont know the name yet' 'yea...im going to need you to come in next saturday because we are not sure if we should rename it yet' It would be a block buster for sure. Rename npath UDF to matchpath - Key: HIVE-5087 URL: https://issues.apache.org/jira/browse/HIVE-5087 Project: Hive Issue Type: Bug Reporter: Edward Capriolo Assignee: Edward Capriolo Priority: Blocker Fix For: 0.12.0 Attachments: HIVE-5087.1.patch.txt, HIVE-5087.99.patch.txt, HIVE-5087-matchpath.1.patch.txt, HIVE-5087.patch.txt, HIVE-5087.patch.txt, regex_path.diff -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5087) Rename npath UDF to matchpath
[ https://issues.apache.org/jira/browse/HIVE-5087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785285#comment-13785285 ] Brock Noland commented on HIVE-5087: I'd be +1 for just removing it. Rename npath UDF to matchpath - Key: HIVE-5087 URL: https://issues.apache.org/jira/browse/HIVE-5087 Project: Hive Issue Type: Bug Reporter: Edward Capriolo Assignee: Edward Capriolo Priority: Blocker Fix For: 0.12.0 Attachments: HIVE-5087.1.patch.txt, HIVE-5087.99.patch.txt, HIVE-5087-matchpath.1.patch.txt, HIVE-5087.patch.txt, HIVE-5087.patch.txt, regex_path.diff -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-4888) listPartitionsByFilter doesn't support lt/gt/lte/gte
[ https://issues.apache.org/jira/browse/HIVE-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785297#comment-13785297 ] Ashutosh Chauhan commented on HIVE-4888: Patch looks good. Just for documentation purposes, we can now push: * Filters on integral types to underlying db via directSQL for eq/lt/gt/gte/lte * Filters on string to underlying db via directSQL for eq * Filters on string to underlying db via DN for lt/gt/gte/lte [~sershe] .q tests test correctness, but don't test optimization done here. Will writing a junit test be better here? listPartitionsByFilter doesn't support lt/gt/lte/gte Key: HIVE-4888 URL: https://issues.apache.org/jira/browse/HIVE-4888 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: D13101.1.patch, D13101.2.patch, D13101.3.patch, D13101.4.patch, HIVE-4888.00.patch, HIVE-4888.01.patch, HIVE-4888.04.patch, HIVE-4888.on-top-of-4914.patch Filter pushdown could be improved. Based on my experiments there's no reasonable way to do it with DN 2.0, due to DN bug in substring and Collection.get(int) not being implemented. With version as low as 2.1 we can use values.get on partition to extract values to compare to. Type compatibility is an issue, but is easy for strings and integral values. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-4642) Implement vectorized RLIKE and REGEXP filter expressions
[ https://issues.apache.org/jira/browse/HIVE-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4642: --- Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Teddy! Implement vectorized RLIKE and REGEXP filter expressions Key: HIVE-4642 URL: https://issues.apache.org/jira/browse/HIVE-4642 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Teddy Choi Fix For: 0.13.0 Attachments: HIVE-4642-1.patch, HIVE-4642.2.patch, HIVE-4642.3.patch.txt, HIVE-4642.4.patch.txt, HIVE-4642.5.patch.txt, HIVE-4642.6.patch.txt, HIVE-4642.7.patch.txt, HIVE-4642.8.patch.txt, HIVE-4642.8-vectorization.patch, Hive-Vectorized-Query-Execution-Design-rev10.docx See title. I will add more details next week. The goal is (a) make this work correctly and (b) optimize it as well as possible, at least for the common cases. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5365) Boolean constants in the query are not handled correctly.
[
https://issues.apache.org/jira/browse/HIVE-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785325#comment-13785325
]
Hive QA commented on HIVE-5365:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12606504/HIVE-5365.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4372 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.vector.TestVectorizationContext.testBooleanColumnCompareBooleanScalar
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1011/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1011/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
Boolean constants in the query are not handled correctly.
-
Key: HIVE-5365
URL: https://issues.apache.org/jira/browse/HIVE-5365
Project: Hive
Issue Type: Sub-task
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey
Attachments: HIVE-5365.1.patch
Boolean constants in the query are not handled correctly.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Resolved] (HIVE-5333) Milestone 2: Generate tests under maven
[ https://issues.apache.org/jira/browse/HIVE-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland resolved HIVE-5333. Resolution: Fixed Edward, thank you very much for the review! I have committed this to branch. Milestone 2: Generate tests under maven --- Key: HIVE-5333 URL: https://issues.apache.org/jira/browse/HIVE-5333 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Brock Noland Attachments: 1.patch.txt, HIVE-5333.patch, HIVE-5333.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5422) Upgrade Kyro to 2.22 now that it is released
[ https://issues.apache.org/jira/browse/HIVE-5422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785333#comment-13785333 ] Ashutosh Chauhan commented on HIVE-5422: Also, kryo jar appears in build/dist/lib which implies we will end up having it in hive distribution, though strictly speaking its not required since all kryo classes are in hive-exec jar anyways. Upgrade Kyro to 2.22 now that it is released Key: HIVE-5422 URL: https://issues.apache.org/jira/browse/HIVE-5422 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-5422.patch As noted [here|https://issues.apache.org/jira/browse/HIVE-1511?focusedCommentId=13783990page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13783990] v2.22 has been released. We should upgrade to that version as opposed to using the snapshot. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-4945) Make RLIKE/REGEXP run end-to-end by updating VectorizationContext
[ https://issues.apache.org/jira/browse/HIVE-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-4945: --- Status: Patch Available (was: Open) Make RLIKE/REGEXP run end-to-end by updating VectorizationContext - Key: HIVE-4945 URL: https://issues.apache.org/jira/browse/HIVE-4945 Project: Hive Issue Type: Sub-task Affects Versions: vectorization-branch Reporter: Eric Hanson Assignee: Teddy Choi Attachments: HIVE-4945.1.patch.txt, HIVE-4945.2.patch.txt, HIVE-4945.3.patch.txt -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-4945) Make RLIKE/REGEXP run end-to-end by updating VectorizationContext
[ https://issues.apache.org/jira/browse/HIVE-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-4945: --- Status: Open (was: Patch Available) Make RLIKE/REGEXP run end-to-end by updating VectorizationContext - Key: HIVE-4945 URL: https://issues.apache.org/jira/browse/HIVE-4945 Project: Hive Issue Type: Sub-task Affects Versions: vectorization-branch Reporter: Eric Hanson Assignee: Teddy Choi Attachments: HIVE-4945.1.patch.txt, HIVE-4945.2.patch.txt, HIVE-4945.3.patch.txt -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-4945) Make RLIKE/REGEXP run end-to-end by updating VectorizationContext
[ https://issues.apache.org/jira/browse/HIVE-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785344#comment-13785344 ] Jitendra Nath Pandey commented on HIVE-4945: HIVE-4642 is committed, so this patch is unblocked. Please add a unit test to the patch. Make RLIKE/REGEXP run end-to-end by updating VectorizationContext - Key: HIVE-4945 URL: https://issues.apache.org/jira/browse/HIVE-4945 Project: Hive Issue Type: Sub-task Affects Versions: vectorization-branch Reporter: Eric Hanson Assignee: Teddy Choi Attachments: HIVE-4945.1.patch.txt, HIVE-4945.2.patch.txt, HIVE-4945.3.patch.txt -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-4945) Make RLIKE/REGEXP run end-to-end by updating VectorizationContext
[ https://issues.apache.org/jira/browse/HIVE-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785346#comment-13785346 ] Jitendra Nath Pandey commented on HIVE-4945: I noticed that the first patch had couple of tests, please add them to the latest patch. Make RLIKE/REGEXP run end-to-end by updating VectorizationContext - Key: HIVE-4945 URL: https://issues.apache.org/jira/browse/HIVE-4945 Project: Hive Issue Type: Sub-task Affects Versions: vectorization-branch Reporter: Eric Hanson Assignee: Teddy Choi Attachments: HIVE-4945.1.patch.txt, HIVE-4945.2.patch.txt, HIVE-4945.3.patch.txt -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5429) HiveVarcharWritable length not reset when value is changed
[ https://issues.apache.org/jira/browse/HIVE-5429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785359#comment-13785359 ] Jason Dere commented on HIVE-5429: -- TestMinimrCliDriver.testCliDriver_bucket_num_reducers does not fail for me locally. HiveVarcharWritable length not reset when value is changed -- Key: HIVE-5429 URL: https://issues.apache.org/jira/browse/HIVE-5429 Project: Hive Issue Type: Bug Components: Types Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-5429.1.patch HiveVarcharWritable has a field to save the character length when it is determined the first time. There are a couple of setter methods where this length field is not reset when the string value is changed. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5422) Upgrade Kyro to 2.22 now that it is released
[ https://issues.apache.org/jira/browse/HIVE-5422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-5422: --- Attachment: HIVE-5422.patch That test failure looks transient. This latest patch removes the jar from dist/lib. In regards to removing it from the pom, I don't see a way to exclude it from the pom via ivy: http://ant.apache.org/ivy/history/2.3.0/use/makepom.html unless someone knows how to resolve this, I'd say we commit this change since it moves us off a snapshot and including kryo in the pom isn't a regression nor should hurt anyone even though it's ugly. Upgrade Kyro to 2.22 now that it is released Key: HIVE-5422 URL: https://issues.apache.org/jira/browse/HIVE-5422 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-5422.patch, HIVE-5422.patch As noted [here|https://issues.apache.org/jira/browse/HIVE-1511?focusedCommentId=13783990page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13783990] v2.22 has been released. We should upgrade to that version as opposed to using the snapshot. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5433) Fix varchar unit tests to work with hadoop-2.1.1
Jason Dere created HIVE-5433: Summary: Fix varchar unit tests to work with hadoop-2.1.1 Key: HIVE-5433 URL: https://issues.apache.org/jira/browse/HIVE-5433 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere A few of the varchar tests fail when testing against hadoop-2.1.1. It looks like some of the input/output rows used in the tests need to be sorted so that the results look consistent across hadoop versions. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5433) Fix varchar unit tests to work with hadoop-2.1.1
[ https://issues.apache.org/jira/browse/HIVE-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-5433: - Attachment: HIVE-5433.1.patch Attaching patch v1 Fix varchar unit tests to work with hadoop-2.1.1 Key: HIVE-5433 URL: https://issues.apache.org/jira/browse/HIVE-5433 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-5433.1.patch A few of the varchar tests fail when testing against hadoop-2.1.1. It looks like some of the input/output rows used in the tests need to be sorted so that the results look consistent across hadoop versions. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5433) Fix varchar unit tests to work with hadoop-2.1.1
[ https://issues.apache.org/jira/browse/HIVE-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-5433: - Status: Patch Available (was: Open) Fix varchar unit tests to work with hadoop-2.1.1 Key: HIVE-5433 URL: https://issues.apache.org/jira/browse/HIVE-5433 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-5433.1.patch A few of the varchar tests fail when testing against hadoop-2.1.1. It looks like some of the input/output rows used in the tests need to be sorted so that the results look consistent across hadoop versions. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5429) HiveVarcharWritable length not reset when value is changed
[ https://issues.apache.org/jira/browse/HIVE-5429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785402#comment-13785402 ] Ashutosh Chauhan commented on HIVE-5429: +1 HiveVarcharWritable length not reset when value is changed -- Key: HIVE-5429 URL: https://issues.apache.org/jira/browse/HIVE-5429 Project: Hive Issue Type: Bug Components: Types Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-5429.1.patch HiveVarcharWritable has a field to save the character length when it is determined the first time. There are a couple of setter methods where this length field is not reset when the string value is changed. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5423) Speed up testing of scalar UDFS
[
https://issues.apache.org/jira/browse/HIVE-5423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785413#comment-13785413
]
Hive QA commented on HIVE-5423:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12606511/HIVE-5423.1.patch.txt
{color:green}SUCCESS:{color} +1 4046 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1012/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1012/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
Speed up testing of scalar UDFS
---
Key: HIVE-5423
URL: https://issues.apache.org/jira/browse/HIVE-5423
Project: Hive
Issue Type: Improvement
Reporter: Edward Capriolo
Assignee: Edward Capriolo
Attachments: HIVE-5423.1.patch.txt, HIVE-5423.patch.txt
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HIVE-5434) Creating a new HiveConnection does not handle the case when properties are supplied separately from connection string
Vaibhav Gumashta created HIVE-5434: -- Summary: Creating a new HiveConnection does not handle the case when properties are supplied separately from connection string Key: HIVE-5434 URL: https://issues.apache.org/jira/browse/HIVE-5434 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 0.13.0 Reporter: Vaibhav Gumashta Fix For: 0.13.0 While creating a new connection: HiveConnection(String uri, Properties info), the Properties hash can also be used to specify connection properties. When username / password is specified in Properties, the current logic uses that, w/o checking the values in connection string. However, a comment in the code says it should not be like that: // extract user/password from JDBC connection properties if its not supplied in the connection URL Also, it is possible for Properties to contain other keys and they should be handled properly. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5391) make ORC predicate pushdown work with vectorization
[ https://issues.apache.org/jira/browse/HIVE-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-5391: --- Attachment: HIVE-5391.02.patch missing null check make ORC predicate pushdown work with vectorization --- Key: HIVE-5391 URL: https://issues.apache.org/jira/browse/HIVE-5391 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-5391.01.patch, HIVE-5391.01-vectorization.patch, HIVE-5391.02.patch, HIVE-5391.patch, HIVE-5391-vectorization.patch Vectorized execution doesn't utilize ORC predicate pushdown. It should. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5365) Boolean constants in the query are not handled correctly.
[ https://issues.apache.org/jira/browse/HIVE-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-5365: --- Status: Open (was: Patch Available) Boolean constants in the query are not handled correctly. - Key: HIVE-5365 URL: https://issues.apache.org/jira/browse/HIVE-5365 Project: Hive Issue Type: Sub-task Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5365.1.patch, HIVE-5365.2.patch Boolean constants in the query are not handled correctly. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5365) Boolean constants in the query are not handled correctly.
[ https://issues.apache.org/jira/browse/HIVE-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-5365: --- Status: Patch Available (was: Open) Boolean constants in the query are not handled correctly. - Key: HIVE-5365 URL: https://issues.apache.org/jira/browse/HIVE-5365 Project: Hive Issue Type: Sub-task Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5365.1.patch, HIVE-5365.2.patch Boolean constants in the query are not handled correctly. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5365) Boolean constants in the query are not handled correctly.
[ https://issues.apache.org/jira/browse/HIVE-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-5365: --- Attachment: HIVE-5365.2.patch Updated patch addressing the test failure. Boolean constants in the query are not handled correctly. - Key: HIVE-5365 URL: https://issues.apache.org/jira/browse/HIVE-5365 Project: Hive Issue Type: Sub-task Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HIVE-5365.1.patch, HIVE-5365.2.patch Boolean constants in the query are not handled correctly. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5423) Speed up testing of scalar UDFS
[ https://issues.apache.org/jira/browse/HIVE-5423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785434#comment-13785434 ] Mark Wagner commented on HIVE-5423: --- That looks much more friendly to new devs :). I have a couple comments on your patch: * Your patch has the old SimpleConcatTest as well as the ExtendedConcatTest. Was that intentional? * I think ExtendedConcatTest would be better in a package other than o.a.h.h.ql.testutil. These new tests should go in o.a.h.h.ql.udf(.generic) like other udf tests or their own new package. Maybe just change testutil to something like o.a.h.h.ql.udf.scalar? Speed up testing of scalar UDFS --- Key: HIVE-5423 URL: https://issues.apache.org/jira/browse/HIVE-5423 Project: Hive Issue Type: Improvement Reporter: Edward Capriolo Assignee: Edward Capriolo Attachments: HIVE-5423.1.patch.txt, HIVE-5423.patch.txt -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5334) Milestone 3: Some tests pass under maven
[ https://issues.apache.org/jira/browse/HIVE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-5334: --- Assignee: Brock Noland Status: Patch Available (was: Open) Milestone 3: Some tests pass under maven Key: HIVE-5334 URL: https://issues.apache.org/jira/browse/HIVE-5334 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-5334.patch This milestone is that some tests pass and therefore we have the basic unit test environment setup. We'll hunt down the rest of the failing tests in future jiras. NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5334) Milestone 3: Some tests pass under maven
[ https://issues.apache.org/jira/browse/HIVE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-5334: --- Attachment: HIVE-5334.patch With this patch: 1) I had to change the version number to 0.99 since trunk builds kept overwriting my maven repo 2) All tests on common and serde tests pass and all TestCliDriver udf* tests pass. 3) As opposed to referencing a property to find test resources I changed to get them out of the classpath. 4) I had to move the shims (via the roll back/forward scripts) to make eclipse work correctly. But it does! I think this completes this milestone, that is the basic unit tests environment seems to be setup. Milestone 3: Some tests pass under maven Key: HIVE-5334 URL: https://issues.apache.org/jira/browse/HIVE-5334 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Attachments: HIVE-5334.patch This milestone is that some tests pass and therefore we have the basic unit test environment setup. We'll hunt down the rest of the failing tests in future jiras. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5334) Milestone 3: Some tests pass under maven
[ https://issues.apache.org/jira/browse/HIVE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-5334: --- Description: This milestone is that some tests pass and therefore we have the basic unit test environment setup. We'll hunt down the rest of the failing tests in future jiras. NO PRECOMMIT TESTS was:This milestone is that some tests pass and therefore we have the basic unit test environment setup. We'll hunt down the rest of the failing tests in future jiras. Milestone 3: Some tests pass under maven Key: HIVE-5334 URL: https://issues.apache.org/jira/browse/HIVE-5334 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Attachments: HIVE-5334.patch This milestone is that some tests pass and therefore we have the basic unit test environment setup. We'll hunt down the rest of the failing tests in future jiras. NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5423) Speed up testing of scalar UDFS
[ https://issues.apache.org/jira/browse/HIVE-5423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785454#comment-13785454 ] Brock Noland commented on HIVE-5423: One note, PTest2 requires tests to be prefixed with Test as opposed to Suffixed. Speed up testing of scalar UDFS --- Key: HIVE-5423 URL: https://issues.apache.org/jira/browse/HIVE-5423 Project: Hive Issue Type: Improvement Reporter: Edward Capriolo Assignee: Edward Capriolo Attachments: HIVE-5423.1.patch.txt, HIVE-5423.patch.txt -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5087) Rename npath UDF to matchpath
[ https://issues.apache.org/jira/browse/HIVE-5087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785459#comment-13785459 ] Carl Steinbach commented on HIVE-5087: -- We are waiting until EOD Friday to commit this. If you are a Hive committer or PMC member and want more information about what's going on, then please send an email to the Hive PMC list. Thanks. Rename npath UDF to matchpath - Key: HIVE-5087 URL: https://issues.apache.org/jira/browse/HIVE-5087 Project: Hive Issue Type: Bug Reporter: Edward Capriolo Assignee: Edward Capriolo Priority: Blocker Fix For: 0.12.0 Attachments: HIVE-5087.1.patch.txt, HIVE-5087.99.patch.txt, HIVE-5087-matchpath.1.patch.txt, HIVE-5087.patch.txt, HIVE-5087.patch.txt, regex_path.diff -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5428) Direct SQL check fails during tests
[
https://issues.apache.org/jira/browse/HIVE-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-5428:
---
Attachment: HIVE-5428.01.patch
Direct SQL check fails during tests
---
Key: HIVE-5428
URL: https://issues.apache.org/jira/browse/HIVE-5428
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Attachments: D13245.1.patch, HIVE-5428.01.patch
Noticed this while work on mavenization. If you run the following command
{noformat}
ant test -Dtestcase=TestCliDriver -Dqfile=udf_case.q -Dtest.silent=false
{noformat}
and look at the top of the logs you see the exception below. It looks like
something needs to be changed in the initialization order.
{noformat}
2013-10-02 13:42:21,596 INFO metastore.ObjectStore
(ObjectStore.java:initialize(243)) - ObjectStore, initialize called
2013-10-02 13:42:22,048 DEBUG bonecp.BoneCPDataSource
(BoneCPDataSource.java:maybeInit(148)) - JDBC URL =
jdbc:derby:;databaseName=../build/test/junit_metastore_db;create=true,
Username = APP, partitions = 1, max (per partition) = 0, min (per partition)
= 0, helper threads = 3, idle max age = 60 min, idle test period = 240 min
2013-10-02 13:42:22,051 WARN bonecp.BoneCPConfig
(BoneCPConfig.java:sanitize(1537)) - Max Connections 1. Setting to 20
2013-10-02 13:42:30,218 INFO metastore.ObjectStore
(ObjectStore.java:getPMF(312)) - Setting MetaStore object pin classes with
hive.metastore.cache.pinobjtypes=Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order
2013-10-02 13:42:30,253 DEBUG bonecp.BoneCPDataSource
(BoneCPDataSource.java:maybeInit(148)) - JDBC URL =
jdbc:derby:;databaseName=../build/test/junit_metastore_db;create=true,
Username = APP, partitions = 1, max (per partition) = 0, min (per partition)
= 0, helper threads = 3, idle max age = 60 min, idle test period = 240 min
2013-10-02 13:42:30,253 WARN bonecp.BoneCPConfig
(BoneCPConfig.java:sanitize(1537)) - Max Connections 1. Setting to 20
2013-10-02 13:42:30,262 INFO metastore.MetaStoreDirectSql
(MetaStoreDirectSql.java:init(99)) - MySQL check failed, assuming we are
not on mysql: Lexical error at line 1, column 5. Encountered: @ (64),
after : .
2013-10-02 13:42:30,298 ERROR metastore.MetaStoreDirectSql
(MetaStoreDirectSql.java:init(112)) - Self-test query [select DB_ID from
DBS] failed; direct SQL is disabled
javax.jdo.JDODataStoreException: Error executing SQL query select DB_ID
from DBS.
at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:230)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.init(MetaStoreDirectSql.java:108)
at
org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:249)
at
org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:220)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at
org.apache.hadoop.hive.metastore.RetryingRawStore.init(RetryingRawStore.java:62)
at
org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:418)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:405)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:444)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:329)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:289)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.init(RetryingHMSHandler.java:54)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
at
org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4084)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.init(HiveMetaStoreClient.java:126)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1211)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:62)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
at
[jira] [Updated] (HIVE-5428) Direct SQL check fails during tests
[
https://issues.apache.org/jira/browse/HIVE-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-5428:
---
Assignee: Sergey Shelukhin
Status: Patch Available (was: Open)
Direct SQL check fails during tests
---
Key: HIVE-5428
URL: https://issues.apache.org/jira/browse/HIVE-5428
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Assignee: Sergey Shelukhin
Attachments: D13245.1.patch, HIVE-5428.01.patch
Noticed this while work on mavenization. If you run the following command
{noformat}
ant test -Dtestcase=TestCliDriver -Dqfile=udf_case.q -Dtest.silent=false
{noformat}
and look at the top of the logs you see the exception below. It looks like
something needs to be changed in the initialization order.
{noformat}
2013-10-02 13:42:21,596 INFO metastore.ObjectStore
(ObjectStore.java:initialize(243)) - ObjectStore, initialize called
2013-10-02 13:42:22,048 DEBUG bonecp.BoneCPDataSource
(BoneCPDataSource.java:maybeInit(148)) - JDBC URL =
jdbc:derby:;databaseName=../build/test/junit_metastore_db;create=true,
Username = APP, partitions = 1, max (per partition) = 0, min (per partition)
= 0, helper threads = 3, idle max age = 60 min, idle test period = 240 min
2013-10-02 13:42:22,051 WARN bonecp.BoneCPConfig
(BoneCPConfig.java:sanitize(1537)) - Max Connections 1. Setting to 20
2013-10-02 13:42:30,218 INFO metastore.ObjectStore
(ObjectStore.java:getPMF(312)) - Setting MetaStore object pin classes with
hive.metastore.cache.pinobjtypes=Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order
2013-10-02 13:42:30,253 DEBUG bonecp.BoneCPDataSource
(BoneCPDataSource.java:maybeInit(148)) - JDBC URL =
jdbc:derby:;databaseName=../build/test/junit_metastore_db;create=true,
Username = APP, partitions = 1, max (per partition) = 0, min (per partition)
= 0, helper threads = 3, idle max age = 60 min, idle test period = 240 min
2013-10-02 13:42:30,253 WARN bonecp.BoneCPConfig
(BoneCPConfig.java:sanitize(1537)) - Max Connections 1. Setting to 20
2013-10-02 13:42:30,262 INFO metastore.MetaStoreDirectSql
(MetaStoreDirectSql.java:init(99)) - MySQL check failed, assuming we are
not on mysql: Lexical error at line 1, column 5. Encountered: @ (64),
after : .
2013-10-02 13:42:30,298 ERROR metastore.MetaStoreDirectSql
(MetaStoreDirectSql.java:init(112)) - Self-test query [select DB_ID from
DBS] failed; direct SQL is disabled
javax.jdo.JDODataStoreException: Error executing SQL query select DB_ID
from DBS.
at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:230)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.init(MetaStoreDirectSql.java:108)
at
org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:249)
at
org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:220)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at
org.apache.hadoop.hive.metastore.RetryingRawStore.init(RetryingRawStore.java:62)
at
org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:418)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:405)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:444)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:329)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:289)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.init(RetryingHMSHandler.java:54)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
at
org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4084)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.init(HiveMetaStoreClient.java:126)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1211)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:62)
at
[jira] [Updated] (HIVE-5428) Direct SQL check fails during tests
[
https://issues.apache.org/jira/browse/HIVE-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-5428:
--
Attachment: D13245.2.patch
sershe updated the revision HIVE-5428 [jira] Direct SQL check fails during
tests.
Reviewers: JIRA
REVISION DETAIL
https://reviews.facebook.net/D13245
CHANGE SINCE LAST DIFF
https://reviews.facebook.net/D13245?vs=40863id=40899#toc
AFFECTED FILES
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
To: JIRA, sershe
Direct SQL check fails during tests
---
Key: HIVE-5428
URL: https://issues.apache.org/jira/browse/HIVE-5428
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Assignee: Sergey Shelukhin
Attachments: D13245.1.patch, D13245.2.patch, HIVE-5428.01.patch
Noticed this while work on mavenization. If you run the following command
{noformat}
ant test -Dtestcase=TestCliDriver -Dqfile=udf_case.q -Dtest.silent=false
{noformat}
and look at the top of the logs you see the exception below. It looks like
something needs to be changed in the initialization order.
{noformat}
2013-10-02 13:42:21,596 INFO metastore.ObjectStore
(ObjectStore.java:initialize(243)) - ObjectStore, initialize called
2013-10-02 13:42:22,048 DEBUG bonecp.BoneCPDataSource
(BoneCPDataSource.java:maybeInit(148)) - JDBC URL =
jdbc:derby:;databaseName=../build/test/junit_metastore_db;create=true,
Username = APP, partitions = 1, max (per partition) = 0, min (per partition)
= 0, helper threads = 3, idle max age = 60 min, idle test period = 240 min
2013-10-02 13:42:22,051 WARN bonecp.BoneCPConfig
(BoneCPConfig.java:sanitize(1537)) - Max Connections 1. Setting to 20
2013-10-02 13:42:30,218 INFO metastore.ObjectStore
(ObjectStore.java:getPMF(312)) - Setting MetaStore object pin classes with
hive.metastore.cache.pinobjtypes=Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order
2013-10-02 13:42:30,253 DEBUG bonecp.BoneCPDataSource
(BoneCPDataSource.java:maybeInit(148)) - JDBC URL =
jdbc:derby:;databaseName=../build/test/junit_metastore_db;create=true,
Username = APP, partitions = 1, max (per partition) = 0, min (per partition)
= 0, helper threads = 3, idle max age = 60 min, idle test period = 240 min
2013-10-02 13:42:30,253 WARN bonecp.BoneCPConfig
(BoneCPConfig.java:sanitize(1537)) - Max Connections 1. Setting to 20
2013-10-02 13:42:30,262 INFO metastore.MetaStoreDirectSql
(MetaStoreDirectSql.java:init(99)) - MySQL check failed, assuming we are
not on mysql: Lexical error at line 1, column 5. Encountered: @ (64),
after : .
2013-10-02 13:42:30,298 ERROR metastore.MetaStoreDirectSql
(MetaStoreDirectSql.java:init(112)) - Self-test query [select DB_ID from
DBS] failed; direct SQL is disabled
javax.jdo.JDODataStoreException: Error executing SQL query select DB_ID
from DBS.
at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:230)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.init(MetaStoreDirectSql.java:108)
at
org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:249)
at
org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:220)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at
org.apache.hadoop.hive.metastore.RetryingRawStore.init(RetryingRawStore.java:62)
at
org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:418)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:405)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:444)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:329)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:289)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.init(RetryingHMSHandler.java:54)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
at
org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4084)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.init(HiveMetaStoreClient.java:126)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
[jira] [Commented] (HIVE-4734) Use custom ObjectInspectors for AvroSerde
[
https://issues.apache.org/jira/browse/HIVE-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785482#comment-13785482
]
Hive QA commented on HIVE-4734:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12606510/HIVE-4734.4.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 4047 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_serde
org.apache.hadoop.hive.serde2.avro.TestAvroDeserializer.verifyCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1013/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1013/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
Use custom ObjectInspectors for AvroSerde
-
Key: HIVE-4734
URL: https://issues.apache.org/jira/browse/HIVE-4734
Project: Hive
Issue Type: Improvement
Components: Serializers/Deserializers
Reporter: Mark Wagner
Assignee: Mark Wagner
Fix For: 0.12.0
Attachments: HIVE-4734.1.patch, HIVE-4734.2.patch, HIVE-4734.3.patch,
HIVE-4734.4.patch
Currently, the AvroSerde recursively copies all fields of a record from the
GenericRecord to a List row object and provides the standard
ObjectInspectors. Performance can be improved by providing ObjectInspectors
to the Avro record itself.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-5372) Refactor TypeInfo and PrimitiveTypeEntry class hierachy to eliminate info repetition
[ https://issues.apache.org/jira/browse/HIVE-5372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-5372: -- Attachment: HIVE-5372.patch Refactor TypeInfo and PrimitiveTypeEntry class hierachy to eliminate info repetition Key: HIVE-5372 URL: https://issues.apache.org/jira/browse/HIVE-5372 Project: Hive Issue Type: Improvement Components: Types Affects Versions: 0.12.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Fix For: 0.13.0 Attachments: HIVE-5372.patch TypeInfo with its sub-classes and PrimititiveTypeEntry class seem having repetitive information, such as type names and type params. It will be good if we can streamline the information organization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5429) HiveVarcharWritable length not reset when value is changed
[ https://issues.apache.org/jira/browse/HIVE-5429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785503#comment-13785503 ] Thejas M Nair commented on HIVE-5429: - Yes, I think we can get this into 0.12. HiveVarcharWritable length not reset when value is changed -- Key: HIVE-5429 URL: https://issues.apache.org/jira/browse/HIVE-5429 Project: Hive Issue Type: Bug Components: Types Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-5429.1.patch HiveVarcharWritable has a field to save the character length when it is determined the first time. There are a couple of setter methods where this length field is not reset when the string value is changed. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5372) Refactor TypeInfo and PrimitiveTypeEntry class hierachy to eliminate info repetition
[ https://issues.apache.org/jira/browse/HIVE-5372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785506#comment-13785506 ] Xuefu Zhang commented on HIVE-5372: --- The refactoring work turned out to be much bigger than thought, but it's almost done and I'm running tests against it. The following are the highlights. 1. Removed type params from PrimitiveTypeEntry class, as it belongs to TypeInfo. 2. PrimitiveTypeInfo holds a reference to a PrimitiveTypeEntry instance, as a primitive type shares the same info in class PrimitiveTypeEntry regardless its possible parameters. 3. Wherever type information is needed, TypeInfo instance is referenced/passed. For instance, an object inspector should have a type info instance in order to correctly interpret the data and enforce limit. This includes UDFs as well. 4. Several factory classes, such as TypeInfoFactor, LazyObjectInspector factory, and so on are cleaned and consolidated. 5. Generic type parameter classes are replaced with type info class hierarchy, employing a more OO approach. As a result of the refactoring, I think the code is cleaner, simplified. Type information is consolidated and streamlined. Down the road, it should be easier for people to understand the code and maintain it. Refactor TypeInfo and PrimitiveTypeEntry class hierachy to eliminate info repetition Key: HIVE-5372 URL: https://issues.apache.org/jira/browse/HIVE-5372 Project: Hive Issue Type: Improvement Components: Types Affects Versions: 0.12.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Fix For: 0.13.0 Attachments: HIVE-5372.patch TypeInfo with its sub-classes and PrimititiveTypeEntry class seem having repetitive information, such as type names and type params. It will be good if we can streamline the information organization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5433) Fix varchar unit tests to work with hadoop-2.1.1
[ https://issues.apache.org/jira/browse/HIVE-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785507#comment-13785507 ] Thejas M Nair commented on HIVE-5433: - +1 Fix varchar unit tests to work with hadoop-2.1.1 Key: HIVE-5433 URL: https://issues.apache.org/jira/browse/HIVE-5433 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-5433.1.patch A few of the varchar tests fail when testing against hadoop-2.1.1. It looks like some of the input/output rows used in the tests need to be sorted so that the results look consistent across hadoop versions. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-4734) Use custom ObjectInspectors for AvroSerde
[ https://issues.apache.org/jira/browse/HIVE-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Wagner updated HIVE-4734: -- Attachment: HIVE-4734.5.patch I missed a couple things in the rebase. I've resolved those and the failing tests now pass. Use custom ObjectInspectors for AvroSerde - Key: HIVE-4734 URL: https://issues.apache.org/jira/browse/HIVE-4734 Project: Hive Issue Type: Improvement Components: Serializers/Deserializers Reporter: Mark Wagner Assignee: Mark Wagner Fix For: 0.12.0 Attachments: HIVE-4734.1.patch, HIVE-4734.2.patch, HIVE-4734.3.patch, HIVE-4734.4.patch, HIVE-4734.5.patch Currently, the AvroSerde recursively copies all fields of a record from the GenericRecord to a List row object and provides the standard ObjectInspectors. Performance can be improved by providing ObjectInspectors to the Avro record itself. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5372) Refactor TypeInfo and PrimitiveTypeEntry class hierachy to eliminate info repetition
[ https://issues.apache.org/jira/browse/HIVE-5372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785515#comment-13785515 ] Jason Dere commented on HIVE-5372: -- Hey Xuefu, can you post the patch to rb? Refactor TypeInfo and PrimitiveTypeEntry class hierachy to eliminate info repetition Key: HIVE-5372 URL: https://issues.apache.org/jira/browse/HIVE-5372 Project: Hive Issue Type: Improvement Components: Types Affects Versions: 0.12.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Fix For: 0.13.0 Attachments: HIVE-5372.patch TypeInfo with its sub-classes and PrimititiveTypeEntry class seem having repetitive information, such as type names and type params. It will be good if we can streamline the information organization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5334) Milestone 3: Some tests pass under maven
[ https://issues.apache.org/jira/browse/HIVE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-5334: --- Attachment: HIVE-5334.patch Updated patch gets service and jdbc tests passing as well. Milestone 3: Some tests pass under maven Key: HIVE-5334 URL: https://issues.apache.org/jira/browse/HIVE-5334 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-5334.patch, HIVE-5334.patch This milestone is that some tests pass and therefore we have the basic unit test environment setup. We'll hunt down the rest of the failing tests in future jiras. NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5423) Speed up testing of scalar UDFS
[
https://issues.apache.org/jira/browse/HIVE-5423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785535#comment-13785535
]
Hive QA commented on HIVE-5423:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12606511/HIVE-5423.1.patch.txt
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1014/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1014/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Tests failed with: ExecutionException: java.util.concurrent.ExecutionException:
org.apache.hive.ptest.execution.ssh.SSHExecutionException: RSyncResult
[localFile=/data/hive-ptest/logs/PreCommit-HIVE-Build-1014/succeeded/TestCliDriver-join23.q-auto_join4.q-input_dynamicserde.q-and-12-more,
remoteFile=/home/hiveptest/ip-10-74-22-60-hiveptest-1/logs/, getExitCode()=12,
getException()=null, getUser()=hiveptest, getHost()=ip-10-74-22-60,
getInstance()=1]: 'ssh_exchange_identification: read: Connection reset by peer
rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(600)
[receiver=3.0.6]
ssh: connect to host ip-10-74-22-60 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(600)
[receiver=3.0.6]
ssh: connect to host ip-10-74-22-60 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(600)
[receiver=3.0.6]
ssh: connect to host ip-10-74-22-60 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(600)
[receiver=3.0.6]
ssh: connect to host ip-10-74-22-60 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(600)
[receiver=3.0.6]
'
{noformat}
This message is automatically generated.
Speed up testing of scalar UDFS
---
Key: HIVE-5423
URL: https://issues.apache.org/jira/browse/HIVE-5423
Project: Hive
Issue Type: Improvement
Reporter: Edward Capriolo
Assignee: Edward Capriolo
Attachments: HIVE-5423.1.patch.txt, HIVE-5423.patch.txt
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5372) Refactor TypeInfo and PrimitiveTypeEntry class hierachy to eliminate info repetition
[ https://issues.apache.org/jira/browse/HIVE-5372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785536#comment-13785536 ] Xuefu Zhang commented on HIVE-5372: --- Hey, Jason. I'm still waiting for the tests to run. Once that passed, I'll submit the patch and post review request. Refactor TypeInfo and PrimitiveTypeEntry class hierachy to eliminate info repetition Key: HIVE-5372 URL: https://issues.apache.org/jira/browse/HIVE-5372 Project: Hive Issue Type: Improvement Components: Types Affects Versions: 0.12.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Fix For: 0.13.0 Attachments: HIVE-5372.patch TypeInfo with its sub-classes and PrimititiveTypeEntry class seem having repetitive information, such as type names and type params. It will be good if we can streamline the information organization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5385) StringUtils is not in commons codec 1.3
[ https://issues.apache.org/jira/browse/HIVE-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785539#comment-13785539 ] Kousuke Saruta commented on HIVE-5385: -- [~yhuai] Sorry, I had a mistake. I found why commons-codec 1.3 is downloaded is because hadoop-core(0.20.2) depends on commons-codec 1.3. So, if we use commons-codec 1.4, we should use newer hadoop-core ( maybe 0.20.205 or 1.x) for hadoop-core. StringUtils is not in commons codec 1.3 --- Key: HIVE-5385 URL: https://issues.apache.org/jira/browse/HIVE-5385 Project: Hive Issue Type: Bug Reporter: Yin Huai Priority: Trivial Attachments: HIVE-5385.1.patch In ThriftHttpServlet introduced by HIVE-4763, StringUtils is imported which was introduced by commons codec 1.4. But, our 0.20 shims depends on commons codec 1.3. Our eclipse classpath template is also using libs of 0.20 shims. So, we will get two errors in eclipse. Compiling hive will not have a problem because we are loading codec 1.4 for project service (1.4 is also used when -Dhadoop.version=0.20.2 -Dhadoop.mr.rev=20). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[ https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5394: Attachment: (was: HIVE-5394.12.branch.txt) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. --- Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Blocker Fix For: 0.13.0 Attachments: HIVE-5394.1.patch, HIVE-5394.2.patch.txt, HIVE-5394.3.patch The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[ https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5394: Attachment: HIVE-5394.12.branch.txt Thanks [~ashutoshc]. Reattaching patch for .12 branch ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. --- Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Blocker Fix For: 0.13.0 Attachments: HIVE-5394.12.branch.txt, HIVE-5394.1.patch, HIVE-5394.2.patch.txt, HIVE-5394.3.patch The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5335) Milestone 4: Most tests pass under maven
[ https://issues.apache.org/jira/browse/HIVE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-5335: --- Summary: Milestone 4: Most tests pass under maven (was: Milestone 4: PTest2 maven support) Milestone 4: Most tests pass under maven Key: HIVE-5335 URL: https://issues.apache.org/jira/browse/HIVE-5335 Project: Hive Issue Type: Sub-task Reporter: Brock Noland -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5435) Milestone 5: PTest2 maven support
Brock Noland created HIVE-5435: -- Summary: Milestone 5: PTest2 maven support Key: HIVE-5435 URL: https://issues.apache.org/jira/browse/HIVE-5435 Project: Hive Issue Type: Sub-task Reporter: Brock Noland -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5382) Allow strings represented as exponential notation to be typecasted to int/smallint/bigint/tinyint
[ https://issues.apache.org/jira/browse/HIVE-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785562#comment-13785562 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-5382: - Hi Gunther, Thanks for the review for addressing those issues. It seems like e notation explicitly means 10^value. I need to look more into how java language handles similar scenarios . One question is why cant we directly use 'Double.valueOf(string)' for this conversion as defined in http://developer.classpath.org/doc/java/lang/Double-source.html Theoretically, it should tally with the results that we obtain if we cast one number type to another number type in Hive. The bottom line is that the issues that Gunther addressed in RB need to be clearly understood and addressed before we can merge in the change. This will be fixed by 0.13. Thanks Hari Allow strings represented as exponential notation to be typecasted to int/smallint/bigint/tinyint - Key: HIVE-5382 URL: https://issues.apache.org/jira/browse/HIVE-5382 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5382.1.patch Follow up jira for HIVE-5352 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-4898) make vectorized math functions work end-to-end (update VectorizationContext.java)
[ https://issues.apache.org/jira/browse/HIVE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson updated HIVE-4898: -- Status: Patch Available (was: In Progress) This patch is based off trunk (since vectorization has merged to trunk). Enabled math functions to work end to end in SQL queries in vectorized mode. See the list of math functions affected in HIVE-4822. One function (unhex()) was removed because it has been changed to return BINARY, and binary is not yet supported with vectorization. Ad hoc testing was performed for all the math functions to verify they work from SQL queries. make vectorized math functions work end-to-end (update VectorizationContext.java) - Key: HIVE-4898 URL: https://issues.apache.org/jira/browse/HIVE-4898 Project: Hive Issue Type: Sub-task Affects Versions: vectorization-branch Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-4898.3.patch The vectorized math function VectorExpression classes were added in HIVE-4822. This JIRA is to allow those to actually be used in a SQL query end-to-end. This requires updating VectorizationContext to use the new classes in vectorized expression creation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-4898) make vectorized math functions work end-to-end (update VectorizationContext.java)
[ https://issues.apache.org/jira/browse/HIVE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson updated HIVE-4898: -- Attachment: HIVE-4898.3.patch make vectorized math functions work end-to-end (update VectorizationContext.java) - Key: HIVE-4898 URL: https://issues.apache.org/jira/browse/HIVE-4898 Project: Hive Issue Type: Sub-task Affects Versions: vectorization-branch Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-4898.3.patch The vectorized math function VectorExpression classes were added in HIVE-4822. This JIRA is to allow those to actually be used in a SQL query end-to-end. This requires updating VectorizationContext to use the new classes in vectorized expression creation. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-4821) Implement vectorized type casting for all types
[
https://issues.apache.org/jira/browse/HIVE-4821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Hanson updated HIVE-4821:
--
Assignee: Eric Hanson (was: Sarvesh Sakalanaga)
Implement vectorized type casting for all types
---
Key: HIVE-4821
URL: https://issues.apache.org/jira/browse/HIVE-4821
Project: Hive
Issue Type: Sub-task
Affects Versions: vectorization-branch
Reporter: Eric Hanson
Assignee: Eric Hanson
Implement vectorized support for casting from any type to any type.
From the documentation:
cast(expr as type): Converts the results of the expression expr to type
e.g. cast('1' as BIGINT) will convert the string '1' to it integral
representation. A null is returned if the conversion does not succeed
The current supported internal types are:
LONG
DOUBLE
STRING
TIMESTAMP
Before implementation, determine what are the semantics of explicit casting
to types less general than the internal types. E.g. what if you cast DOUBLE
to TINYINT? Can we just cast internally to LONG and let the output process
cast to TINYINT?
This JIRA includes all work to make casting operate end-to-end in a SQL query
in vectorized mode, including updates to VectorizationContext.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-5348) If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters
[ https://issues.apache.org/jira/browse/HIVE-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5348: Attachment: HIVE-5348.2.patch Made changes to resolve the failures hopefully If both table and partition have same serdes, then we don't even need to check for OI conversions in ObjectInspectorConverters -- Key: HIVE-5348 URL: https://issues.apache.org/jira/browse/HIVE-5348 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5348.1.patch.txt, HIVE-5348.2.patch Follow-up JIRA for HIVE-5202 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5336) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user
[ https://issues.apache.org/jira/browse/HIVE-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-5336: Attachment: HIVE-5336.3.patch same patch as 1 and 2. HCatSchema.remove(HCatFieldSchema hcatFieldSchema) should renumber the fieldPositionMap and the fieldPositionMap should not be cached by the end user -- Key: HIVE-5336 URL: https://issues.apache.org/jira/browse/HIVE-5336 Project: Hive Issue Type: Bug Components: HCatalog Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-5336.1.patch.txt, HIVE-5336.2.patch, HIVE-5336.3.patch HCatSchema.remove currently does not renumber the fieldPositionMap which can be a problem when there are interleaving append() and remove() calls. 1. We should document that fieldPositionMap should not be cached by the end-user 2. We should make sure that the fieldPositionMap gets renumbered after remove() because HcatSchema.get will otherwise return wrong FieldSchemas. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5413) StorageDelegationAuthorizationProvider uses non-existent org.apache.hive.hcatalog.hbase.HBaseHCatStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785652#comment-13785652
]
Hudson commented on HIVE-5413:
--
FAILURE: Integrated in Hive-trunk-hadoop2-ptest #123 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/123/])
HIVE-5413 : StorageDelegationAuthorizationProvider uses non-existent
org.apache.hive.hcatalog.hbase.HBaseHCatStorageHandler (Eugene Koifman via
Sushanth Sowmyan) (khorgath:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528633)
* /hive/trunk/hcatalog/conf/proto-hive-site.xml
*
/hive/trunk/hcatalog/core/src/main/java/org/apache/hcatalog/security/HdfsAuthorizationProvider.java
*
/hive/trunk/hcatalog/core/src/main/java/org/apache/hcatalog/security/StorageDelegationAuthorizationProvider.java
*
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/security/HdfsAuthorizationProvider.java
*
/hive/trunk/hcatalog/core/src/main/java/org/apache/hive/hcatalog/security/StorageDelegationAuthorizationProvider.java
*
/hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/security/TestHdfsAuthorizationProvider.java
*
/hive/trunk/hcatalog/core/src/test/java/org/apache/hive/hcatalog/security/TestHdfsAuthorizationProvider.java
* /hive/trunk/hcatalog/src/test/e2e/hcatalog/tests/hadoop.conf
* /hive/trunk/hcatalog/src/test/e2e/hcatalog/tests/pig.conf
*
/hive/trunk/hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hcatalog/utils/HBaseReadWrite.java
*
/hive/trunk/hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/HBaseReadWrite.java
*
/hive/trunk/hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/WriteTextPartitioned.java
* /hive/trunk/hcatalog/src/test/e2e/templeton/README.txt
*
/hive/trunk/hcatalog/storage-handlers/hbase/src/java/org/apache/hcatalog/hbase/HBaseAuthorizationProvider.java
StorageDelegationAuthorizationProvider uses non-existent
org.apache.hive.hcatalog.hbase.HBaseHCatStorageHandler
---
Key: HIVE-5413
URL: https://issues.apache.org/jira/browse/HIVE-5413
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.12.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Fix For: 0.12.0
Attachments: HIVE-5413.patch
org.apache.hive.hcatalog.security.StorageDelegationAuthorizationProvider
has a block like this:
static {
registerAuthProvider(org.apache.hadoop.hive.hbase.HBaseStorageHandler,
org.apache.hive.hcatalog.hbase.HBaseAuthorizationProvider);
registerAuthProvider(org.apache.hive.hcatalog.hbase.HBaseHCatStorageHandler,
org.apache.hive.hcatalog.hbase.HBaseAuthorizationProvider);
}
In reality, HBaseHCatStorageHandler and HBaseAuthorizationProvider only exist
in org.apache.hcatalog
This should be fixed by removing
org.apache.hive.hcatalog.security.StorageDelegationAuthorizationProvider. We
have org.apache.hcatalog.security.StorageDelegationAuthorizationProvider
which is only used by now deprecated hcatalog version of hbase storage
handler.
Also,
hcatalog/src/test/e2e/hcatalog/tests/pig.conf hadoop.conf have the same
problem.
In addition, the tests affected in pig.conf/hadoop.conf should use
org.apache.hcatalog.pig.HCatLoader/HCatStorer.
Finally, hadoop.conf#Hadoop_HBase is using
org.apache.hive.hcatalog.utils.HBaseReadWrite which internally refers to
org.apache.hive.hcatalog.* classes. The later should only use
org.apache.hcatalog.* since it's using HBaseHCatStoreageHandler. Also,
should move HBaseReadWrite to org.apache.hcatalog for clarity.
(see the last paragraph of the Description of HIVE-5274)
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[ https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785653#comment-13785653 ] Hudson commented on HIVE-5394: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #123 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/123/]) HIVE-5394 : ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. (Hari Sankar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528912) * /hive/trunk/ql/src/test/queries/clientpositive/alter_varchar2.q * /hive/trunk/ql/src/test/results/clientpositive/alter_varchar2.q.out * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. --- Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Blocker Fix For: 0.13.0 Attachments: HIVE-5394.12.branch.txt, HIVE-5394.1.patch, HIVE-5394.2.patch.txt, HIVE-5394.3.patch The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-4642) Implement vectorized RLIKE and REGEXP filter expressions
[ https://issues.apache.org/jira/browse/HIVE-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785659#comment-13785659 ] Hudson commented on HIVE-4642: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #123 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/123/]) HIVE-4642 : Implement vectorized RLIKE and REGEXP filter expressions (Teddy Choi via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528917) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/AbstractFilterStringColLikeStringScalar.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColRegExpStringScalar.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorStringExpressions.java Implement vectorized RLIKE and REGEXP filter expressions Key: HIVE-4642 URL: https://issues.apache.org/jira/browse/HIVE-4642 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Teddy Choi Fix For: 0.13.0 Attachments: HIVE-4642-1.patch, HIVE-4642.2.patch, HIVE-4642.3.patch.txt, HIVE-4642.4.patch.txt, HIVE-4642.5.patch.txt, HIVE-4642.6.patch.txt, HIVE-4642.7.patch.txt, HIVE-4642.8.patch.txt, HIVE-4642.8-vectorization.patch, Hive-Vectorized-Query-Execution-Design-rev10.docx See title. I will add more details next week. The goal is (a) make this work correctly and (b) optimize it as well as possible, at least for the common cases. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5363) HIVE-3978 broke the command line option --auxpath
[
https://issues.apache.org/jira/browse/HIVE-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785662#comment-13785662
]
Hudson commented on HIVE-5363:
--
FAILURE: Integrated in Hive-trunk-hadoop2-ptest #123 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/123/])
HIVE-5410 : Hive command line option --auxpath still does not work post
HIVE-5363 (Brock Noland via Thejas Nair) (thejas:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528634)
* /hive/trunk/bin/hive
HIVE-3978 broke the command line option --auxpath
-
Key: HIVE-5363
URL: https://issues.apache.org/jira/browse/HIVE-5363
Project: Hive
Issue Type: Improvement
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Blocker
Fix For: 0.12.0
Attachments: HIVE-5363.patch
HIVE-3978 changed the seperator for HIVE_AUX_JARS_PATH to : from ,. However,
it's expected that it's , later on in the script, specifically here:
{noformat}
AUX_CLASSPATH=${HIVE_AUX_JARS_PATH}
AUX_PARAM=file://${HIVE_AUX_JARS_PATH}
AUX_PARAM=`echo $AUX_PARAM | sed 's/,/,file:\/\//g'
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[ https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785645#comment-13785645 ] Hudson commented on HIVE-5394: -- ABORTED: Integrated in Hive-trunk-hadoop2 #473 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/473/]) HIVE-5394 : ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. (Hari Sankar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528912) * /hive/trunk/ql/src/test/queries/clientpositive/alter_varchar2.q * /hive/trunk/ql/src/test/results/clientpositive/alter_varchar2.q.out * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. --- Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Blocker Fix For: 0.13.0 Attachments: HIVE-5394.12.branch.txt, HIVE-5394.1.patch, HIVE-5394.2.patch.txt, HIVE-5394.3.patch The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-4642) Implement vectorized RLIKE and REGEXP filter expressions
[ https://issues.apache.org/jira/browse/HIVE-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785648#comment-13785648 ] Hudson commented on HIVE-4642: -- ABORTED: Integrated in Hive-trunk-hadoop2 #473 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/473/]) HIVE-4642 : Implement vectorized RLIKE and REGEXP filter expressions (Teddy Choi via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528917) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/AbstractFilterStringColLikeStringScalar.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColRegExpStringScalar.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorStringExpressions.java Implement vectorized RLIKE and REGEXP filter expressions Key: HIVE-4642 URL: https://issues.apache.org/jira/browse/HIVE-4642 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Teddy Choi Fix For: 0.13.0 Attachments: HIVE-4642-1.patch, HIVE-4642.2.patch, HIVE-4642.3.patch.txt, HIVE-4642.4.patch.txt, HIVE-4642.5.patch.txt, HIVE-4642.6.patch.txt, HIVE-4642.7.patch.txt, HIVE-4642.8.patch.txt, HIVE-4642.8-vectorization.patch, Hive-Vectorized-Query-Execution-Design-rev10.docx See title. I will add more details next week. The goal is (a) make this work correctly and (b) optimize it as well as possible, at least for the common cases. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-4642) Implement vectorized RLIKE and REGEXP filter expressions
[ https://issues.apache.org/jira/browse/HIVE-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785647#comment-13785647 ] Hudson commented on HIVE-4642: -- ABORTED: Integrated in Hive-trunk-h0.21 #2377 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2377/]) HIVE-4642 : Implement vectorized RLIKE and REGEXP filter expressions (Teddy Choi via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528917) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/AbstractFilterStringColLikeStringScalar.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColRegExpStringScalar.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorStringExpressions.java Implement vectorized RLIKE and REGEXP filter expressions Key: HIVE-4642 URL: https://issues.apache.org/jira/browse/HIVE-4642 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Teddy Choi Fix For: 0.13.0 Attachments: HIVE-4642-1.patch, HIVE-4642.2.patch, HIVE-4642.3.patch.txt, HIVE-4642.4.patch.txt, HIVE-4642.5.patch.txt, HIVE-4642.6.patch.txt, HIVE-4642.7.patch.txt, HIVE-4642.8.patch.txt, HIVE-4642.8-vectorization.patch, Hive-Vectorized-Query-Execution-Design-rev10.docx See title. I will add more details next week. The goal is (a) make this work correctly and (b) optimize it as well as possible, at least for the common cases. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5405) Need to implement PersistenceDelegate for org.antlr.runtime.CommonToken
[ https://issues.apache.org/jira/browse/HIVE-5405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785656#comment-13785656 ] Hudson commented on HIVE-5405: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #123 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/123/]) HIVE-5405 - Need to implement PersistenceDelegate for org.antlr.runtime.CommonToken (shanyu zhao via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528531) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java Need to implement PersistenceDelegate for org.antlr.runtime.CommonToken --- Key: HIVE-5405 URL: https://issues.apache.org/jira/browse/HIVE-5405 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: shanyu zhao Assignee: shanyu zhao Fix For: 0.12.0 Attachments: HIVE-5405-branch-0.12.patch, HIVE-5405.patch Prior to HIVE-1511, running hive join operation results in the following exception: java.lang.RuntimeException: Cannot serialize object at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities. java:639) at java.beans.XMLEncoder.writeStatement(XMLEncoder.java:426) ... Caused by: java.lang.InstantiationException: org.antlr.runtime.CommonToken at java.lang.Class.newInstance0(Class.java:357) at java.lang.Class.newInstance(Class.java:325) at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) HIVE-1511 introduced a new (and set to default) hive plan serialization format Kryo, which fixed this problem by implementing the Kryo serializer for CommonToken. However, if we set the following in configuration file: property namehive.plan.serialization.format/name valuejavaXML/value /property We'll see the same failure as before. We need to implement a PersistenceDelegate for the situation when javaXML is set to serialization format. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5394) ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type.
[ https://issues.apache.org/jira/browse/HIVE-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785646#comment-13785646 ] Hudson commented on HIVE-5394: -- ABORTED: Integrated in Hive-trunk-h0.21 #2377 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2377/]) HIVE-5394 : ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. (Hari Sankar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528912) * /hive/trunk/ql/src/test/queries/clientpositive/alter_varchar2.q * /hive/trunk/ql/src/test/results/clientpositive/alter_varchar2.q.out * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java ObjectInspectorConverters.getConvertedOI() does not return the correct object inspector for primitive type. --- Key: HIVE-5394 URL: https://issues.apache.org/jira/browse/HIVE-5394 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Priority: Blocker Fix For: 0.13.0 Attachments: HIVE-5394.12.branch.txt, HIVE-5394.1.patch, HIVE-5394.2.patch.txt, HIVE-5394.3.patch The code currently returns settable type of the input primitive object inspector where as it should return settable type of output object inspector -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-3807) Hive authorization should use short username when Kerberos authentication
[ https://issues.apache.org/jira/browse/HIVE-3807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785657#comment-13785657 ] Hudson commented on HIVE-3807: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #123 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/123/]) HIVE-3807 : Hive authorization should use short username when Kerberos authentication (Kai Zheng via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528523) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/security/HadoopDefaultAuthenticator.java Hive authorization should use short username when Kerberos authentication - Key: HIVE-3807 URL: https://issues.apache.org/jira/browse/HIVE-3807 Project: Hive Issue Type: Improvement Components: Authorization Affects Versions: 0.9.0, 0.10.0 Reporter: Kai Zheng Assignee: Kai Zheng Fix For: 0.12.0 Attachments: HIVE-3807.patch Currently when authentication method is Kerberos,Hive authorization uses user full name as privilege principal, for example, it uses j...@example.com instead of john. It should use the short name instead. The benefits: 1. Be consistent. Hadoop, HBase and etc they all use short name in related ACLs or authorizations. For Hive authorization works well with them, this should be. 2. Be convenient. It's very inconvenient to use the lengthy Kerberos principal name when grant or revoke privileges via Hive CLI. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5410) Hive command line option --auxpath still does not work post HIVE-5363
[
https://issues.apache.org/jira/browse/HIVE-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785649#comment-13785649
]
Hudson commented on HIVE-5410:
--
FAILURE: Integrated in Hive-trunk-hadoop2-ptest #123 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/123/])
HIVE-5410 : Hive command line option --auxpath still does not work post
HIVE-5363 (Brock Noland via Thejas Nair) (thejas:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528634)
* /hive/trunk/bin/hive
Hive command line option --auxpath still does not work post HIVE-5363
-
Key: HIVE-5410
URL: https://issues.apache.org/jira/browse/HIVE-5410
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Blocker
Fix For: 0.12.0
Attachments: HIVE-5410.patch
In short, AUX_PARAM is set to:
{noformat}
$ echo file:///etc/passwd | sed 's/:/,file:\/\//g'
file,file:/etc/passwd
{noformat}
which is invalid because file is not a real file.
NO PRECOMMIT TESTS (since this is not tested)
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5419) Fix schema tool issues with Oracle metastore
[ https://issues.apache.org/jira/browse/HIVE-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785654#comment-13785654 ] Hudson commented on HIVE-5419: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #123 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/123/]) HIVE-5419 : Fix schema tool issues with Oracle metastore (Prasad Mujumdar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528655) * /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java * /hive/trunk/beeline/src/test/org/apache/hive/beeline/src/test/TestSchemaTool.java * /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.13.0.derby.sql * /hive/trunk/metastore/scripts/upgrade/oracle/014-HIVE-3764.oracle.sql * /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.12.0.oracle.sql * /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.13.0.oracle.sql * /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.11.0-to-0.12.0.oracle.sql * /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.12.0-to-0.13.0.oracle.sql * /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.12.0.postgres.sql * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java Fix schema tool issues with Oracle metastore - Key: HIVE-5419 URL: https://issues.apache.org/jira/browse/HIVE-5419 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.12.0, 0.13.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Priority: Blocker Fix For: 0.13.0 Attachments: HIVE-5419.1.patch Address oracle schema upgrade script issue in 0.12 and trunk (0.13) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5196) ThriftCLIService.java uses stderr to print the stack trace, it should use the logger instead.
[ https://issues.apache.org/jira/browse/HIVE-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785661#comment-13785661 ] Hudson commented on HIVE-5196: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #123 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/123/]) HIVE-5196 : ThriftCLIService.java uses stderr to print the stack trace, it should use the logger instead. (Vaibhav Gumashta via Thejas Nair) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528675) * /hive/trunk/service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java ThriftCLIService.java uses stderr to print the stack trace, it should use the logger instead. - Key: HIVE-5196 URL: https://issues.apache.org/jira/browse/HIVE-5196 Project: Hive Issue Type: Bug Affects Versions: 0.12.0 Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta Fix For: 0.12.0 Attachments: D13107.1.patch, HIVE-5196.3.patch, HIVE-5196.D13107.1.patch, HIVE-5196.D13107.2.patch ThriftCLIService.java uses stderr to print the stack trace, it should use the logger instead. Using e.printStackTrace is not suitable for production. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5114) add a target to run tests without rebuilding them
[ https://issues.apache.org/jira/browse/HIVE-5114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785650#comment-13785650 ] Hudson commented on HIVE-5114: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #123 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/123/]) HIVE-5114 : add a target to run tests without rebuilding them (Sergey Shelukhin Brock Noland via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528230) * /hive/trunk/ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java * /hive/trunk/build-common.xml * /hive/trunk/build.xml * /hive/trunk/common/build.xml * /hive/trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm * /hive/trunk/hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm * /hive/trunk/hcatalog/build-support/ant/build-common.xml * /hive/trunk/hcatalog/build.xml * /hive/trunk/hcatalog/storage-handlers/hbase/build.xml * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java * /hive/trunk/ql/src/test/templates/TestBeeLineDriver.vm * /hive/trunk/ql/src/test/templates/TestCliDriver.vm * /hive/trunk/ql/src/test/templates/TestNegativeCliDriver.vm * /hive/trunk/ql/src/test/templates/TestParse.vm * /hive/trunk/ql/src/test/templates/TestParseNegative.vm add a target to run tests without rebuilding them - Key: HIVE-5114 URL: https://issues.apache.org/jira/browse/HIVE-5114 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: HIVE-5114.2.patch, HIVE-5114.D12399.1.patch it is sometimes annoying that each ant test ... cleans and rebuilds the tests. It is should be relatively easy to add a testonly target that would just run the test(s) on the existing build -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5296) Memory leak: OOM Error after multiple open/closed JDBC connections.
[
https://issues.apache.org/jira/browse/HIVE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785651#comment-13785651
]
Hudson commented on HIVE-5296:
--
FAILURE: Integrated in Hive-trunk-hadoop2-ptest #123 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/123/])
HIVE-5296: Memory leak: OOM Error after multiple open/closed JDBC connections.
(Kousuke Saruta via Thejas Nair) (thejas:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528678)
*
/hive/trunk/service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java
Memory leak: OOM Error after multiple open/closed JDBC connections.
Key: HIVE-5296
URL: https://issues.apache.org/jira/browse/HIVE-5296
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.12.0, 0.13.0
Environment: Hive 0.12.0, Hadoop 1.1.2, Debian.
Reporter: Douglas
Assignee: Kousuke Saruta
Labels: hiveserver
Fix For: 0.12.0, 0.13.0
Attachments: HIVE-5296.1.patch, HIVE-5296.2.patch, HIVE-5296.patch,
HIVE-5296.patch, HIVE-5296.patch
Original Estimate: 168h
Remaining Estimate: 168h
Multiple connections to Hiveserver2, all of which are closed and disposed of
properly show the Java heap size to grow extremely quickly.
This issue can be recreated using the following code
{code}
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import org.apache.hive.service.cli.HiveSQLException;
import org.apache.log4j.Logger;
/*
* Class which encapsulates the lifecycle of a query or statement.
* Provides functionality which allows you to create a connection
*/
public class HiveClient {
Connection con;
Logger logger;
private static String driverName = org.apache.hive.jdbc.HiveDriver;
private String db;
public HiveClient(String db)
{
logger = Logger.getLogger(HiveClient.class);
this.db=db;
try{
Class.forName(driverName);
}catch(ClassNotFoundException e){
logger.info(Can't find Hive driver);
}
String hiveHost = GlimmerServer.config.getString(hive/host);
String hivePort = GlimmerServer.config.getString(hive/port);
String connectionString = jdbc:hive2://+hiveHost+:+hivePort
+/default;
logger.info(String.format(Attempting to connect to
%s,connectionString));
try{
con =
DriverManager.getConnection(connectionString,,);
}catch(Exception e){
logger.error(Problem instantiating the
connection+e.getMessage());
}
}
public int update(String query)
{
Integer res = 0;
Statement stmt = null;
try{
stmt = con.createStatement();
String switchdb = USE +db;
logger.info(switchdb);
stmt.executeUpdate(switchdb);
logger.info(query);
res = stmt.executeUpdate(query);
logger.info(Query passed to server);
stmt.close();
}catch(HiveSQLException e){
logger.info(String.format(HiveSQLException thrown,
this can be valid, +
but check the error: %s from the query
%s,query,e.toString()));
}catch(SQLException e){
logger.error(String.format(Unable to execute query
SQLException %s. Error: %s,query,e));
}catch(Exception e){
logger.error(String.format(Unable to execute query %s.
Error: %s,query,e));
}
if(stmt!=null)
try{
stmt.close();
}catch(SQLException e){
logger.error(Cannot close the statment,
potentially memory leak +e);
}
return res;
}
public void close()
{
if(con!=null){
try {
con.close();
} catch (SQLException e) {
[jira] [Commented] (HIVE-4446) [HCatalog] Documentation for HIVE-4442, HIVE-4443, HIVE-4444
[ https://issues.apache.org/jira/browse/HIVE-4446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785664#comment-13785664 ] Lefty Leverenz commented on HIVE-4446: -- Oh, I hadn't realized that CMS migration includes moving wikidocs into the code. Hooray and arrgh!! (Hooray for getting versioned docs at long last, but arrgh for all the HCat WebHCat docs I converted to wikidocs.) However, the CMS move hasn't happened yet so do you want these changes on the wiki for now? I could take care of that after I wrap up a few more cross-references for the HCat WebHCat docs. Committing puts the changes into xml source files, not changing the HCat 0.5.0 docs on the Hive site (http://hive.apache.org/docs/). But will we be able to use new html docs generated from xml sources? That process is broken (HIVE-3896 and HIVE-4065). Besides, basing the CMS migration on xml docs risks losing editing tweaks and a few error corrections in the wikidocs. I suggest you assign this jira to me for inclusion in the wiki, instead of committing the patch. Committing won't do any harm, though. [HCatalog] Documentation for HIVE-4442, HIVE-4443, HIVE- Key: HIVE-4446 URL: https://issues.apache.org/jira/browse/HIVE-4446 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.12.0 Attachments: HIVE-4446-1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5235) Infinite loop with ORC file and Hive 0.11
[
https://issues.apache.org/jira/browse/HIVE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785663#comment-13785663
]
Prasanth J commented on HIVE-5235:
--
Hi Pere
Are you using generated dataset? If so can you please upload the data
generation script?
If any query that scans the whole table fails then it may be because ORC file
itself is corrupted. It will be much easier to have the data instead of ORC
file to reproduce this issue.
Infinite loop with ORC file and Hive 0.11
-
Key: HIVE-5235
URL: https://issues.apache.org/jira/browse/HIVE-5235
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0
Environment: Gentoo linux with Hortonworks Hadoop
hadoop-1.1.2.23.tar.gz and Apache Hive 0.11d
Reporter: Iván de Prado
Priority: Blocker
We are using Hive 0.11 with ORC file format and we get some tasks blocked in
some kind of infinite loop. They keep working indefinitely when we set a huge
task expiry timeout. If we the expiry time to 600 second, the taks fail
because of not reporting progress, and finally, the Job fails.
That is not consistent, and some times between jobs executions the behavior
changes. It happen for different queries.
We are using Hive 0.11 with Hadoop hadoop-1.1.2.23 from Hortonworks. The taks
that is blocked keeps consuming 100% of CPU usage, and the stack trace is
always the same consistently. Everything points to some kind of infinite
loop. My guessing is that it has some relation to the ORC file. Maybe some
pointer is not right when writing generating some kind of infinite loop when
reading. Or maybe there is a bug in the reading stage.
More information below. The stack trace:
{noformat}
main prio=10 tid=0x7f20a000a800 nid=0x1ed2 runnable [0x7f20a8136000]
java.lang.Thread.State: RUNNABLE
at java.util.zip.Inflater.inflateBytes(Native Method)
at java.util.zip.Inflater.inflate(Inflater.java:256)
- locked 0xf42a6ca0 (a java.util.zip.ZStreamRef)
at
org.apache.hadoop.hive.ql.io.orc.ZlibCodec.decompress(ZlibCodec.java:64)
at
org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.readHeader(InStream.java:128)
at
org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.read(InStream.java:143)
at
org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readVulong(SerializationUtils.java:54)
at
org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readVslong(SerializationUtils.java:65)
at
org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReader.readValues(RunLengthIntegerReader.java:66)
at
org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReader.next(RunLengthIntegerReader.java:81)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$IntTreeReader.next(RecordReaderImpl.java:332)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.next(RecordReaderImpl.java:802)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1214)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:71)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:46)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:274)
at
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:101)
at
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:41)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:108)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:300)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:218)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
- eliminated 0xe1459700 (a
org.apache.hadoop.mapred.MapTask$TrackedRecordReader)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)
- locked 0xe1459700 (a
org.apache.hadoop.mapred.MapTask$TrackedRecordReader)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1178)
at
[jira] [Updated] (HIVE-5295) HiveConnection#configureConnection tries to execute statement even after it is closed
[ https://issues.apache.org/jira/browse/HIVE-5295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-5295: Component/s: HiveServer2 HiveConnection#configureConnection tries to execute statement even after it is closed - Key: HIVE-5295 URL: https://issues.apache.org/jira/browse/HIVE-5295 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta Fix For: 0.12.0 Attachments: D12957.1.patch, D12957.2.patch, D12957.3.patch, HIVE-5295.D12957.3.patch, HIVE-5295.D12957.3.patch, HIVE-5295.D12957.4.patch HiveConnection#configureConnection tries to execute statement even after it is closed. For remote JDBC client, it tries to set the conf var using 'set foo=bar' by calling HiveStatement.execute for each conf var pair, but closes the statement after the 1st iteration through the conf var pairs. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5418) Integer overflow bug in ConditionalResolverCommonJoin.AliasFileSizePair
[
https://issues.apache.org/jira/browse/HIVE-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steven Wong updated HIVE-5418:
--
Affects Version/s: 0.13.0
Integer overflow bug in ConditionalResolverCommonJoin.AliasFileSizePair
---
Key: HIVE-5418
URL: https://issues.apache.org/jira/browse/HIVE-5418
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0, 0.13.0
Reporter: Steven Wong
Assignee: Steven Wong
Attachments: HIVE-5418.0.patch, HIVE-5418.1.patch
Sometimes, auto map join conversion unexpectedly fails to choose map join
over a common join, even if the auto map join conversion's size criterion is
satisfied.
This is caused by an integer overflow bug in the method {{compareTo}} of the
class {{ConditionalResolverCommonJoin.AliasFileSizePair}}.
The bug is triggered only if the big table size exceeds the small table size
by at least 2**31 bytes.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HIVE-5436) Hive's casting behavior needs to be consistent
Hari Sankar Sivarama Subramaniyan created HIVE-5436:
---
Summary: Hive's casting behavior needs to be consistent
Key: HIVE-5436
URL: https://issues.apache.org/jira/browse/HIVE-5436
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Priority: Critical
Hive's casting behavior is inconsistent and the behavior of casting from one
type to another undocumented as of now when the casted value is out of range.
For example, casting out of range values from one type to another can result in
incorrect results.
Eg:
1. select cast('1000' as tinyint) from t1;
NULL
2. select 1000Y from t1;
FAILED: SemanticException [Error 10029]: Line 1:7 Invalid numerical constant
'1000Y'
3. select cast(1000 as tinyint) from t1;
-24
4.select cast(1.1e3-1000/0 as tinyint) from t1;
0
5. select cast(10/0 as tinyint) from pw18;
-1
The hive user can accidently try to typecast an out of range value. For example
in the e.g. 4/5 even though the final result is NaN, Hive can typecast to a
random result. Either we should document that the end user should take care of
overflow, underflow, division by 0, etc. by himself/herself or we should
return NULLs when the final result is out of range.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-4446) [HCatalog] Documentation for HIVE-4442, HIVE-4443, HIVE-4444
[ https://issues.apache.org/jira/browse/HIVE-4446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-4446: Assignee: Lefty Leverenz (was: Daniel Dai) [HCatalog] Documentation for HIVE-4442, HIVE-4443, HIVE- Key: HIVE-4446 URL: https://issues.apache.org/jira/browse/HIVE-4446 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Daniel Dai Assignee: Lefty Leverenz Fix For: 0.12.0 Attachments: HIVE-4446-1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-4446) [HCatalog] Documentation for HIVE-4442, HIVE-4443, HIVE-4444
[ https://issues.apache.org/jira/browse/HIVE-4446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785705#comment-13785705 ] Thejas M Nair commented on HIVE-4446: - [~le...@hortonworks.com] I have assigned it to you after speaking to Daniel offline. These changes are part of hive 0.12 release. I am hoping to have the release out next week. So it would be great if you can get to this by sometime next week. [HCatalog] Documentation for HIVE-4442, HIVE-4443, HIVE- Key: HIVE-4446 URL: https://issues.apache.org/jira/browse/HIVE-4446 Project: Hive Issue Type: Improvement Components: HCatalog Reporter: Daniel Dai Assignee: Lefty Leverenz Fix For: 0.12.0 Attachments: HIVE-4446-1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5437) Add map/reduce input map to MapWork/ReduceWork for multi input
[ https://issues.apache.org/jira/browse/HIVE-5437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-5437: - Description: When there are multiple incoming edges to the tezprocessor we'll need to be able to tell which alias or tag they belong to. In the map case: - Potentially multiple MRInputs. Need to map those to aliases/operators In the reduce case: - Each tag will be represented by a logical input. We need to know which table each input belongs to. NO PRECOMMIT TESTS (this is wip for the tez branch) was: When there are multiple incoming edges to the tezprocessor we'll need to be able to tell which alias or tag they belong to. In the map case: - Potentially multiple MRInputs. Need to map those to aliases/operators In the reduce case: - Each tag will be represented by a logical input. We need to know which table each input belongs to. Add map/reduce input map to MapWork/ReduceWork for multi input -- Key: HIVE-5437 URL: https://issues.apache.org/jira/browse/HIVE-5437 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch When there are multiple incoming edges to the tezprocessor we'll need to be able to tell which alias or tag they belong to. In the map case: - Potentially multiple MRInputs. Need to map those to aliases/operators In the reduce case: - Each tag will be represented by a logical input. We need to know which table each input belongs to. NO PRECOMMIT TESTS (this is wip for the tez branch) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5437) Add map/reduce input map to MapWork/ReduceWork for multi input
Gunther Hagleitner created HIVE-5437: Summary: Add map/reduce input map to MapWork/ReduceWork for multi input Key: HIVE-5437 URL: https://issues.apache.org/jira/browse/HIVE-5437 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: tez-branch When there are multiple incoming edges to the tezprocessor we'll need to be able to tell which alias or tag they belong to. In the map case: - Potentially multiple MRInputs. Need to map those to aliases/operators In the reduce case: - Each tag will be represented by a logical input. We need to know which table each input belongs to. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-4690) stats_partscan_1.q makes different result with different hadhoop.mr.rev
[ https://issues.apache.org/jira/browse/HIVE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785710#comment-13785710 ] Ashutosh Chauhan commented on HIVE-4690: Committed to trunk. Thanks, Navis Brock! stats_partscan_1.q makes different result with different hadhoop.mr.rev Key: HIVE-4690 URL: https://issues.apache.org/jira/browse/HIVE-4690 Project: Hive Issue Type: Sub-task Affects Versions: 0.11.0 Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-4690.2.patch, HIVE-4690.D11163.1.patch stats_partscan_1.q uses mapred.min/max.split.size and logs number of files, which can be different with different hadoop.mr.rev. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5422) Upgrade Kyro to 2.22 now that it is released
[ https://issues.apache.org/jira/browse/HIVE-5422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785721#comment-13785721 ] Ashutosh Chauhan commented on HIVE-5422: +1 Lets get this in. When we move to maven this will be non-issue any way, so no point in spending more time on it. Upgrade Kyro to 2.22 now that it is released Key: HIVE-5422 URL: https://issues.apache.org/jira/browse/HIVE-5422 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-5422.patch, HIVE-5422.patch As noted [here|https://issues.apache.org/jira/browse/HIVE-1511?focusedCommentId=13783990page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13783990] v2.22 has been released. We should upgrade to that version as opposed to using the snapshot. -- This message was sent by Atlassian JIRA (v6.1#6144)
