Re: Review Request 22804: HIVE-7263 - Missing fixes from review of parquet-timestamp
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22804/ --- (Updated June 20, 2014, 6:04 a.m.) Review request for hive and Brock Noland. Bugs: HIVE-7263 https://issues.apache.org/jira/browse/HIVE-7263 Repository: hive-git Description --- This is for HIVE-6394 (parquet timestamp). There had been a review comment for not relying on example parquet classes, which are just suggestion of how to implement timestamp. It is trivial, just to implement that sample class in Hive code base. I had addressed it in one of the patch, but the next patch did not carry that on as I made a mistake. Addressing again now. Diffs - ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java 73cf0f5 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTime.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/utils/NanoTimeUtils.java 06987ad ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriter.java 8bb9cb1 ql/src/test/org/apache/hadoop/hive/ql/io/parquet/serde/TestParquetTimestampUtils.java f56a643 Diff: https://reviews.apache.org/r/22804/diff/ Testing --- Ran affected parquet timestamp tests. Thanks, Szehon Ho
[jira] [Updated] (HIVE-7263) Missing fixes from review of parquet-timestamp
[ https://issues.apache.org/jira/browse/HIVE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-7263: Status: Patch Available (was: Open) Missing fixes from review of parquet-timestamp -- Key: HIVE-7263 URL: https://issues.apache.org/jira/browse/HIVE-7263 Project: Hive Issue Type: Bug Reporter: Szehon Ho Assignee: Szehon Ho Attachments: HIVE-7263.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7263) Missing fixes from review of parquet-timestamp
[ https://issues.apache.org/jira/browse/HIVE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-7263: Attachment: HIVE-7263.patch [~brocknoland] I'm sorry , when I merged with the commit, I saw I made a mistake. There had been a review comment in HIVE-6394 to not use example parquet class, which is a suggestion of how to implement timestamp. It was trivial non-functional change, just to write that in Hive code base. I had actually addressed it an intermed patch, but the final patch did not carry that on. Putting that back, sorry again about that. Missing fixes from review of parquet-timestamp -- Key: HIVE-7263 URL: https://issues.apache.org/jira/browse/HIVE-7263 Project: Hive Issue Type: Bug Reporter: Szehon Ho Assignee: Szehon Ho Attachments: HIVE-7263.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7218) java.io.IOException: error=7, Argument list too long
[
https://issues.apache.org/jira/browse/HIVE-7218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038551#comment-14038551
]
Hive QA commented on HIVE-7218:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12651328/HIVE-7218.1.patch.txt
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 5669 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.ql.exec.TestOperators.testScriptOperatorEnvVarsProcessing
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/525/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/525/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-525/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12651328
java.io.IOException: error=7, Argument list too long
Key: HIVE-7218
URL: https://issues.apache.org/jira/browse/HIVE-7218
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.7.0, 0.7.1, 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0,
0.12.0, 0.13.0, 0.13.1
Reporter: Ryan Harris
Assignee: Navis
Attachments: HIVE-7218.1.patch.txt
HIVE-2372 was originally created in response to this error message, however
that patch was merely a work-around to handle the condition where
mapred.input.dir is too long.
Any other environment variable that is too long for the host OS will still
cause a job failure.
In my case:
While creating a table with a large number of columns, a large hive variable
is temporarily created using SET, the variable contains the columns and
column descriptions.
A CREATE TABLE statement then successfully uses that large variable.
After successfully creating the table the hive script attempts to load data
into the table using a TRANSFORM script, triggering the error:
java.io.IOException: error=7, Argument list too long
Since the variable is no longer used after the table is created, the hive
script was updated to SET the large variable to empty.
After setting the variable empty the second statement in the hive script ran
fine.
Hive should more gracefully notify the user as to the cause of the problem
and offer a configurable approach for automatically handling the condition.
In this case, originally identifying the cause of the issue was somewhat
confusing since the portion of the hive script that referenced the long
variable ran successfully, and the portion of the script that failed didn't
even use/reference the variable that was causing that portion to fail.
Since HIVE-2372 has already been Fixed this JIRA re-opens the issue since
the original issue was worked around, not resolved...
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5155) Support secure proxy user access to HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-5155: - Labels: TODOC13 (was: ) Support secure proxy user access to HiveServer2 --- Key: HIVE-5155 URL: https://issues.apache.org/jira/browse/HIVE-5155 Project: Hive Issue Type: Improvement Components: Authentication, HiveServer2, JDBC Affects Versions: 0.12.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Labels: TODOC13 Fix For: 0.13.0 Attachments: HIVE-5155-1-nothrift.patch, HIVE-5155-noThrift.2.patch, HIVE-5155-noThrift.4.patch, HIVE-5155-noThrift.5.patch, HIVE-5155-noThrift.6.patch, HIVE-5155-noThrift.7.patch, HIVE-5155-noThrift.8.patch, HIVE-5155.1.patch, HIVE-5155.2.patch, HIVE-5155.3.patch, HIVE-5155.4.patch, HIVE-5155.5.patch, ProxyAuth.java, ProxyAuth.out, TestKERBEROS_Hive_JDBC.java The HiveServer2 can authenticate a client using via Kerberos and impersonate the connecting user with underlying secure hadoop. This becomes a gateway for a remote client to access secure hadoop cluster. Now this works fine for when the client obtains Kerberos ticket and directly connects to HiveServer2. There's another big use case for middleware tools where the end user wants to access Hive via another server. For example Oozie action or Hue submitting queries or a BI tool server accessing to HiveServer2. In these cases, the third party server doesn't have end user's Kerberos credentials and hence it can't submit queries to HiveServer2 on behalf of the end user. This ticket is for enabling proxy access to HiveServer2 for third party tools on behalf of end users. There are two parts of the solution proposed in this ticket: 1) Delegation token based connection for Oozie (OOZIE-1457) This is the common mechanism for Hadoop ecosystem components. Hive Remote Metastore and HCatalog already support this. This is suitable for tool like Oozie that submits the MR jobs as actions on behalf of its client. Oozie already uses similar mechanism for Metastore/HCatalog access. 2) Direct proxy access for privileged hadoop users The delegation token implementation can be a challenge for non-hadoop (especially non-java) components. This second part enables a privileged user to directly specify an alternate session user during the connection. If the connecting user has hadoop level privilege to impersonate the requested userid, then HiveServer2 will run the session as that requested user. For example, user Hue is allowed to impersonate user Bob (via core-site.xml proxy user configuration). Then user Hue can connect to HiveServer2 and specify Bob as session user via a session property. HiveServer2 will verify Hue's proxy user privilege and then impersonate user Bob instead of Hue. This will enable any third party tool to impersonate alternate userid without having to implement delegation token connection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7250) Adaptive compression buffer size for wide tables in ORC
[ https://issues.apache.org/jira/browse/HIVE-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038568#comment-14038568 ] Lefty Leverenz commented on HIVE-7250: -- No user doc? Adaptive compression buffer size for wide tables in ORC --- Key: HIVE-7250 URL: https://issues.apache.org/jira/browse/HIVE-7250 Project: Hive Issue Type: Improvement Components: File Formats Affects Versions: 0.14.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile Fix For: 0.14.0 Attachments: HIVE-7250.1.patch, HIVE-7250.2.patch, HIVE-7250.3.patch, HIVE-7250.4.patch, HIVE-7250.5.patch If the input table is wide (in the order of 1000s), ORC compression buffer size overhead becomes significant causing OOM issues. To overcome this issue, buffer size should be adaptively chosen based on the available memory and the number of columns. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7188) sum(if()) returns wrong results with vectorization
[ https://issues.apache.org/jira/browse/HIVE-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-7188: - Fix Version/s: 0.14.0 sum(if()) returns wrong results with vectorization -- Key: HIVE-7188 URL: https://issues.apache.org/jira/browse/HIVE-7188 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Fix For: 0.14.0 Attachments: HIVE-7188.1.patch, HIVE-7188.2.patch, hike-vector-sum-bug.tgz 1. The tgz file containing the setup is attached. 2. Run the following query select sum(if(is_returning=true and is_free=false,1,0)) as unpaid_returning from hike_error.ttr_day0; returns 0 rows with vectorization turned on whereas it return 131 rows with vectorization turned off. hive source insert.sql ; OK Time taken: 0.359 seconds OK Time taken: 0.015 seconds OK Time taken: 0.069 seconds OK Time taken: 0.176 seconds Loading data to table hike_error.ttr_day0 Table hike_error.ttr_day0 stats: [numFiles=1, numRows=0, totalSize=3581, rawDataSize=0] OK Time taken: 0.33 seconds hive select sum(if(is_returning=true and is_free=false,1,0)) as unpaid_returning from hike_error.ttr_day0; Query ID = hsubramaniyan_20140606134646_04790d3d-ca9a-427a-8cf9-3174536114ed Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=number In order to limit the maximum number of reducers: set hive.exec.reducers.max=number In order to set a constant number of reducers: set mapred.reduce.tasks=number Execution log at: /var/folders/r0/9x0wltgx2nv4m4b18m71z1y4gr/T//hsubramaniyan/hsubramaniyan_20140606134646_04790d3d-ca9a-427a-8cf9-3174536114ed.log Job running in-process (local Hadoop) Hadoop job information for null: number of mappers: 0; number of reducers: 0 2014-06-06 13:47:02,043 null map = 0%, reduce = 100% Ended Job = job_local773704964_0001 Execution completed successfully MapredLocal task succeeded OK 131 Time taken: 5.325 seconds, Fetched: 1 row(s) hive set hive.vectorized.execution.enabled=true; hive select sum(if(is_returning=true and is_free=false,1,0)) as unpaid_returning from hike_error.ttr_day0; Query ID = hsubramaniyan_20140606134747_1182c765-90ac-4a33-a8b1-760adca6bf38 Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=number In order to limit the maximum number of reducers: set hive.exec.reducers.max=number In order to set a constant number of reducers: set mapred.reduce.tasks=number Execution log at: /var/folders/r0/9x0wltgx2nv4m4b18m71z1y4gr/T//hsubramaniyan/hsubramaniyan_20140606134747_1182c765-90ac-4a33-a8b1-760adca6bf38.log Job running in-process (local Hadoop) Hadoop job information for null: number of mappers: 0; number of reducers: 0 2014-06-06 13:47:18,604 null map = 0%, reduce = 100% Ended Job = job_local701415676_0001 Execution completed successfully MapredLocal task succeeded OK 0 Time taken: 5.52 seconds, Fetched: 1 row(s) hive explain select sum(if(is_returning=true and is_free=false,1,0)) as unpaid_returning from hike_error.ttr_day0; OK STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: ttr_day0 Statistics: Num rows: 447 Data size: 3581 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: is_returning (type: boolean), is_free (type: boolean) outputColumnNames: is_returning, is_free Statistics: Num rows: 447 Data size: 3581 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: sum(if(((is_returning = true) and (is_free = false)), 1, 0)) mode: hash outputColumnNames: _col0 Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator sort order: Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE value expressions: _col0 (type: bigint) Execution mode: vectorized Reduce Operator Tree: Group By Operator aggregations: sum(VALUE._col0) mode: mergepartial
[jira] [Updated] (HIVE-6394) Implement Timestmap in ParquetSerde
[ https://issues.apache.org/jira/browse/HIVE-6394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-6394: - Labels: Parquet TODOC14 (was: Parquet) Implement Timestmap in ParquetSerde --- Key: HIVE-6394 URL: https://issues.apache.org/jira/browse/HIVE-6394 Project: Hive Issue Type: Sub-task Components: Serializers/Deserializers Reporter: Jarek Jarcec Cecho Assignee: Szehon Ho Labels: Parquet, TODOC14 Fix For: 0.14.0 Attachments: HIVE-6394.2.patch, HIVE-6394.3.patch, HIVE-6394.4.patch, HIVE-6394.5.patch, HIVE-6394.6.patch, HIVE-6394.6.patch, HIVE-6394.7.patch, HIVE-6394.patch This JIRA is to implement timestamp support in Parquet SerDe. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6394) Implement Timestmap in ParquetSerde
[ https://issues.apache.org/jira/browse/HIVE-6394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038572#comment-14038572 ] Lefty Leverenz commented on HIVE-6394: -- Document this for 0.14.0 here: * [Language Manual -- Parquet -- Limitations | https://cwiki.apache.org/confluence/display/Hive/Parquet#Parquet-Limitations] Implement Timestmap in ParquetSerde --- Key: HIVE-6394 URL: https://issues.apache.org/jira/browse/HIVE-6394 Project: Hive Issue Type: Sub-task Components: Serializers/Deserializers Reporter: Jarek Jarcec Cecho Assignee: Szehon Ho Labels: Parquet, TODOC14 Fix For: 0.14.0 Attachments: HIVE-6394.2.patch, HIVE-6394.3.patch, HIVE-6394.4.patch, HIVE-6394.5.patch, HIVE-6394.6.patch, HIVE-6394.6.patch, HIVE-6394.7.patch, HIVE-6394.patch This JIRA is to implement timestamp support in Parquet SerDe. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-4924) Support query timeout for jdbc2
[
https://issues.apache.org/jira/browse/HIVE-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038621#comment-14038621
]
Thorsten Meinl commented on HIVE-4924:
--
Any news on this? Currently the timeout for the Socket created by thrift is
taken from the login (!) timeout set by the java.sql.DriverManager. First of
all nobody expects the login timeout to affect the read timeout during a query
and secondly this timeout can only be set globally for the whole JVM. This
means if you have connections to other databases they will also be affected
when the value is changed.
Support query timeout for jdbc2
---
Key: HIVE-4924
URL: https://issues.apache.org/jira/browse/HIVE-4924
Project: Hive
Issue Type: Improvement
Components: HiveServer2, JDBC
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-4924.D11787.1.patch, HIVE-4924.D11787.2.patch
support
{code}
Statement.setQueryTimeout(int timeout)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6622) UDF translate doesn't take either CHAR or VARCHAR as any of its arguments
[
https://issues.apache.org/jira/browse/HIVE-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038670#comment-14038670
]
Hive QA commented on HIVE-6622:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12651332/HIVE-6622.1.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 5653 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/526/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/526/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-526/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12651332
UDF translate doesn't take either CHAR or VARCHAR as any of its arguments
-
Key: HIVE-6622
URL: https://issues.apache.org/jira/browse/HIVE-6622
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.12.0
Reporter: Xuefu Zhang
Assignee: Wilbur Yang
Attachments: HIVE-6622.1.patch
UDF translate(input, from, to) doesn't accept any CHAR or VARCHAR typed
argument.
{code}
hive desc vc;
c char(5) None
vcvarchar(7) None
s string None
hive select translate(my input, c, ll) from vc;
FAILED: SemanticException [Error 10016]: Line 1:29 Argument type mismatch
'c': A string argument was expected but an argument of type char(5) was given.
{code}
However, if the type is String, then the UDF works fine.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7230) Add Eclipse formatter file for Hive coding conventions
[
https://issues.apache.org/jira/browse/HIVE-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038808#comment-14038808
]
Swarnim Kulkarni commented on HIVE-7230:
Sorry for the late reply.
{quote}
in my patch, I am pointing the Maven Eclipse plugin to the formatter file that
sits in the root of the source tree and not the remote guide.
{quote}
Ah. My bad. That should work well. Looks good. I also pulled down the patch and
verified locally. Looks good.
+1.
Add Eclipse formatter file for Hive coding conventions
--
Key: HIVE-7230
URL: https://issues.apache.org/jira/browse/HIVE-7230
Project: Hive
Issue Type: Improvement
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7230.1.patch, HIVE-7230.2.patch, HIVE-7230.3.patch
Eclipse's formatter is a convenient way to clean up formatting for Java code.
Currently, there is no Eclipse formatter file checked into Hive's codebase.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7186) Unable to perform join on table
[
https://issues.apache.org/jira/browse/HIVE-7186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038830#comment-14038830
]
Alex Nastetsky commented on HIVE-7186:
--
I just saw a similar problem with with a different stacktrace. This time, the
join got to the very end of the job and failed as it finished:
{code}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.EOFException:
Premature EOF: no length prefix available
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:514)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:332)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:159)
at
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:548)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:599)
Caused by: java.io.EOFException: Premature EOF: no length prefix available
at
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1492)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:962)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:930)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1031)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:823)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:475)
{code}
Unable to perform join on table
---
Key: HIVE-7186
URL: https://issues.apache.org/jira/browse/HIVE-7186
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Environment: Hortonworks Data Platform 2.0.6.0
Reporter: Alex Nastetsky
Occasionally, a table will start exhibiting behavior that will prevent it
from being used in a JOIN.
When doing a map join, it will just stall at Starting to launch local task
to process map join; .
When doing a regular join, it will make progress but then error out with a
IndexOutOfBoundsException:
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.IndexOutOfBoundsException
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:365)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
... 9 more
Caused by: java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:532)
at
java.nio.ByteBufferAsIntBufferL.put(ByteBufferAsIntBufferL.java:131)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1153)
at
org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:586)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:372)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:334)
... 15 more
Doing simple selects against this table work fine and do not show any
apparent problems with the data.
Assume that the table in question is called tableA and was created by queryA.
Doing either of the following has helped resolve the issue in the past.
1) create table tableB as select * from tableA;
Then just use tableB instead in the JOIN.
2) regenerate tableA using queryA
Then use tableA in the JOIN again. It usually works the second time.
When doing a describe formatted on the tables, the totalSize will be
different between the original tableA and tableB, and sometimes (but not
always) between the original tableA and the regenerated tableA. The numRows
will be the same across all versions of the tables.
This problem can not be reproduced consistently, but the
[jira] [Commented] (HIVE-7264) TPCDS Query 78 throws NPE when Vectorization is turned on
[ https://issues.apache.org/jira/browse/HIVE-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038936#comment-14038936 ] Hitesh Shah commented on HIVE-7264: --- Moving this jira to the Hive project. TPCDS Query 78 throws NPE when Vectorization is turned on - Key: HIVE-7264 URL: https://issues.apache.org/jira/browse/HIVE-7264 Project: Hive Issue Type: Bug Affects Versions: 0.13.0, 0.13.1 Environment: HDP 2.1 Hive 0.13 / Tez Reporter: Douglas Moore I get the same identical stack traceback found in TEZ-975 running TPC-DS query78 with vectorization turned while running on HDP 2.1 / Hive 0.13/Tez. The HDP 2.1 release notes state that HIVE-6742 was fixed (and TEZ-975 was closed as fixed/duplicate) When I turn off vectorization and the query completes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Moved] (HIVE-7264) TPCDS Query 78 throws NPE when Vectorization is turned on
[ https://issues.apache.org/jira/browse/HIVE-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah moved TEZ-1215 to HIVE-7264: Affects Version/s: (was: 0.4.0) 0.13.0 0.13.1 Key: HIVE-7264 (was: TEZ-1215) Project: Hive (was: Apache Tez) TPCDS Query 78 throws NPE when Vectorization is turned on - Key: HIVE-7264 URL: https://issues.apache.org/jira/browse/HIVE-7264 Project: Hive Issue Type: Bug Affects Versions: 0.13.1, 0.13.0 Environment: HDP 2.1 Hive 0.13 / Tez Reporter: Douglas Moore I get the same identical stack traceback found in TEZ-975 running TPC-DS query78 with vectorization turned while running on HDP 2.1 / Hive 0.13/Tez. The HDP 2.1 release notes state that HIVE-6742 was fixed (and TEZ-975 was closed as fixed/duplicate) When I turn off vectorization and the query completes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7264) TPCDS Query 78 throws NPE when Vectorization is turned on
[ https://issues.apache.org/jira/browse/HIVE-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038937#comment-14038937 ] Hitesh Shah commented on HIVE-7264: --- [~hagleitn] [~vikram.dixit] Care to take a look? TPCDS Query 78 throws NPE when Vectorization is turned on - Key: HIVE-7264 URL: https://issues.apache.org/jira/browse/HIVE-7264 Project: Hive Issue Type: Bug Affects Versions: 0.13.0, 0.13.1 Environment: HDP 2.1 Hive 0.13 / Tez Reporter: Douglas Moore I get the same identical stack traceback found in TEZ-975 running TPC-DS query78 with vectorization turned while running on HDP 2.1 / Hive 0.13/Tez. The HDP 2.1 release notes state that HIVE-6742 was fixed (and TEZ-975 was closed as fixed/duplicate) When I turn off vectorization and the query completes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7202) DbTxnManager deadlocks in hcatalog.cli.TestSematicAnalysis.testAlterTblFFpart()
[
https://issues.apache.org/jira/browse/HIVE-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038949#comment-14038949
]
Ashutosh Chauhan commented on HIVE-7202:
+1
DbTxnManager deadlocks in
hcatalog.cli.TestSematicAnalysis.testAlterTblFFpart()
---
Key: HIVE-7202
URL: https://issues.apache.org/jira/browse/HIVE-7202
Project: Hive
Issue Type: Bug
Components: Locking
Affects Versions: 0.13.1
Reporter: Eugene Koifman
Assignee: Alan Gates
Fix For: 0.14.0
Attachments: HIVE-7202.patch
select * from HIVE_LOCKS produces
{noformat}
6 |1 |0 |default
|junit_sem_analysis
|NULL
|w|r|1402354627716
|NULL|unknown
|ekoifman.local
6 |2 |0 |default
|junit_sem_analysis
|b=2010-10-10
|w|e|1402354627716
|NULL|unknown
|ekoifman.local
2 rows selected
{noformat}
easiest way to repro this is to add
hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY, true);
hiveConf.setVar(HiveConf.ConfVars.HIVE_TXN_MANAGER,
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager);
in HCatBaseTest.setUpHiveConf()
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6967) Hive transaction manager fails when SQLServer is used as an RDBMS
[
https://issues.apache.org/jira/browse/HIVE-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038958#comment-14038958
]
Ashutosh Chauhan commented on HIVE-6967:
+1
Hive transaction manager fails when SQLServer is used as an RDBMS
-
Key: HIVE-6967
URL: https://issues.apache.org/jira/browse/HIVE-6967
Project: Hive
Issue Type: Bug
Components: Locking
Affects Versions: 0.13.0
Reporter: Alan Gates
Assignee: Alan Gates
Attachments: HIVE-6967.patch
When using SQLServer as an RDBMS for the metastore, any transaction or
DbLockMgr operations fail with:
{code}
MetaException(message:Unable to select from transaction database
com.microsoft.sqlserver.jdbc.SQLServerException: Line 1: FOR UPDATE clause
allowed only for DECLARE CURSOR.
{code}
The issue is that SQLServer does not support the FOR UPDATE clause in SELECT.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7263) Missing fixes from review of parquet-timestamp
[ https://issues.apache.org/jira/browse/HIVE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039028#comment-14039028 ] Brock Noland commented on HIVE-7263: No worries! +1 pending tests Missing fixes from review of parquet-timestamp -- Key: HIVE-7263 URL: https://issues.apache.org/jira/browse/HIVE-7263 Project: Hive Issue Type: Bug Reporter: Szehon Ho Assignee: Szehon Ho Attachments: HIVE-7263.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6967) Hive transaction manager fails when SQLServer is used as an RDBMS
[
https://issues.apache.org/jira/browse/HIVE-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-6967:
---
Resolution: Fixed
Fix Version/s: 0.14.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks, Alan!
Hive transaction manager fails when SQLServer is used as an RDBMS
-
Key: HIVE-6967
URL: https://issues.apache.org/jira/browse/HIVE-6967
Project: Hive
Issue Type: Bug
Components: Locking
Affects Versions: 0.13.0
Reporter: Alan Gates
Assignee: Alan Gates
Fix For: 0.14.0
Attachments: HIVE-6967.patch
When using SQLServer as an RDBMS for the metastore, any transaction or
DbLockMgr operations fail with:
{code}
MetaException(message:Unable to select from transaction database
com.microsoft.sqlserver.jdbc.SQLServerException: Line 1: FOR UPDATE clause
allowed only for DECLARE CURSOR.
{code}
The issue is that SQLServer does not support the FOR UPDATE clause in SELECT.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HIVE-7265) BINARY columns use BytesWritable::getBytes() without ::getLength()
Gopal V created HIVE-7265:
-
Summary: BINARY columns use BytesWritable::getBytes() without
::getLength()
Key: HIVE-7265
URL: https://issues.apache.org/jira/browse/HIVE-7265
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.14.0
Reporter: Gopal V
Priority: Minor
The Text conversion for BINARY columns does
{code}
case BINARY:
t.set(((BinaryObjectInspector)
inputOI).getPrimitiveWritableObject(input).getBytes());
return t;
{code}
This omission was noticed while investigating a different String related bug,
in a list of functions which call getBytes() without calling
getSize/getLength().
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7241) Wrong lock acquired for alter table rename partition
[ https://issues.apache.org/jira/browse/HIVE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-7241: - Attachment: HIVE-7241.patch Wrong lock acquired for alter table rename partition Key: HIVE-7241 URL: https://issues.apache.org/jira/browse/HIVE-7241 Project: Hive Issue Type: Bug Components: Locking Affects Versions: 0.13.0 Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-7241.patch, HIVE-7241.patch Doing an alter table foo partition (bar='x') rename to partition (bar='y') acquires a read lock on table foo. It should instead acquire an exclusive lock on partition bar=x. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7242) alter table drop partition is acquiring the wrong type of lock
[
https://issues.apache.org/jira/browse/HIVE-7242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-7242:
-
Status: Patch Available (was: Open)
Marking patch available now that HIVE-7202 is committed.
alter table drop partition is acquiring the wrong type of lock
--
Key: HIVE-7242
URL: https://issues.apache.org/jira/browse/HIVE-7242
Project: Hive
Issue Type: Bug
Components: Locking
Affects Versions: 0.13.0
Reporter: Alan Gates
Assignee: Alan Gates
Fix For: 0.14.0
Attachments: HIVE-7242.patch
Doing an alter table foo drop partition ('bar=x') acquired a shared-write
lock on partition bar=x. It should be acquiring an exclusive lock in that
case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7242) alter table drop partition is acquiring the wrong type of lock
[
https://issues.apache.org/jira/browse/HIVE-7242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-7242:
-
Attachment: (was: HIVE-7242.patch)
alter table drop partition is acquiring the wrong type of lock
--
Key: HIVE-7242
URL: https://issues.apache.org/jira/browse/HIVE-7242
Project: Hive
Issue Type: Bug
Components: Locking
Affects Versions: 0.13.0
Reporter: Alan Gates
Assignee: Alan Gates
Fix For: 0.14.0
Attachments: HIVE-7242.patch
Doing an alter table foo drop partition ('bar=x') acquired a shared-write
lock on partition bar=x. It should be acquiring an exclusive lock in that
case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HIVE-7266) Optimized HashTable with vectorized map-joins results in String columns extending
Gopal V created HIVE-7266:
-
Summary: Optimized HashTable with vectorized map-joins results in
String columns extending
Key: HIVE-7266
URL: https://issues.apache.org/jira/browse/HIVE-7266
Project: Hive
Issue Type: Bug
Components: Vectorization
Affects Versions: 0.14.0
Reporter: Gopal V
Assignee: Jitendra Nath Pandey
The following query returns different results when both vectorized mapjoin and
the new optimized hashtable are enabled.
{code}
hive set hive.vectorized.execution.enabled=false;
hive select s_suppkey, n_name from supplier, nation where s_nationkey =
n_nationkey limit 25;
...
316869 JAPAN
1636869 RUSSIA
1096869 IRAN
7236869 RUSSIA
2276869 INDIA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIA
2616869 FRANCE
{code}
But when vectorization is enabled, the results are
{code}
316869 JAPAN
1636869 RUSSIA
1096869 IRANIA
7236869 RUSSIA
2276869 INDIAA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIAQUE
2616869 FRANCEAQUE
{code}
When the new optimized map-join hashtable is disabled as well
{code}
hive set hive.vectorized.execution.enabled=true;
hive set hive.mapjoin.optimized.hashtable=false;
hive select s_suppkey, n_name from supplier, nation where s_nationkey =
n_nationkey limit 25;
316869 JAPAN
1636869 RUSSIA
1096869 IRAN
7236869 RUSSIA
2276869 INDIA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIA
2616869 FRANCE
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HIVE-7267) can not trigger unit tests by command `mvn test -Phadoop-2'
John created HIVE-7267: -- Summary: can not trigger unit tests by command `mvn test -Phadoop-2' Key: HIVE-7267 URL: https://issues.apache.org/jira/browse/HIVE-7267 Project: Hive Issue Type: Bug Affects Versions: 0.13.1 Reporter: John 1. download hive 0.13.1 2. decompress and unpack the tarball 3. change the directory to hive 0.13.1 4. run `mvn clean test -Phadoop-2' Could not found any unit tests -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7267) can not trigger unit tests by command `mvn clean test -Phadoop-2'
[ https://issues.apache.org/jira/browse/HIVE-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John updated HIVE-7267: --- Summary: can not trigger unit tests by command `mvn clean test -Phadoop-2' (was: can not trigger unit tests by command `mvn test -Phadoop-2') can not trigger unit tests by command `mvn clean test -Phadoop-2' - Key: HIVE-7267 URL: https://issues.apache.org/jira/browse/HIVE-7267 Project: Hive Issue Type: Bug Affects Versions: 0.13.1 Reporter: John 1. download hive 0.13.1 2. decompress and unpack the tarball 3. change the directory to hive 0.13.1 4. run `mvn clean test -Phadoop-2' Could not found any unit tests -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7267) can not trigger unit tests by command `mvn clean test -Phadoop-2'
[ https://issues.apache.org/jira/browse/HIVE-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John updated HIVE-7267: --- Description: 1. download hive 0.13.1 2. decompress and unpack the tarball 3. change the directory to hive 0.13.1 4. run `mvn clean test -Phadoop-2' Could not find any unit tests was: 1. download hive 0.13.1 2. decompress and unpack the tarball 3. change the directory to hive 0.13.1 4. run `mvn clean test -Phadoop-2' Could not found any unit tests can not trigger unit tests by command `mvn clean test -Phadoop-2' - Key: HIVE-7267 URL: https://issues.apache.org/jira/browse/HIVE-7267 Project: Hive Issue Type: Bug Affects Versions: 0.13.1 Reporter: John 1. download hive 0.13.1 2. decompress and unpack the tarball 3. change the directory to hive 0.13.1 4. run `mvn clean test -Phadoop-2' Could not find any unit tests -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7242) alter table drop partition is acquiring the wrong type of lock
[
https://issues.apache.org/jira/browse/HIVE-7242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-7242:
-
Attachment: HIVE-7242.patch
alter table drop partition is acquiring the wrong type of lock
--
Key: HIVE-7242
URL: https://issues.apache.org/jira/browse/HIVE-7242
Project: Hive
Issue Type: Bug
Components: Locking
Affects Versions: 0.13.0
Reporter: Alan Gates
Assignee: Alan Gates
Fix For: 0.14.0
Attachments: HIVE-7242.patch
Doing an alter table foo drop partition ('bar=x') acquired a shared-write
lock on partition bar=x. It should be acquiring an exclusive lock in that
case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7241) Wrong lock acquired for alter table rename partition
[ https://issues.apache.org/jira/browse/HIVE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-7241: - Status: Patch Available (was: Open) Marking patch available now that HIVE-7202 is committed. Wrong lock acquired for alter table rename partition Key: HIVE-7241 URL: https://issues.apache.org/jira/browse/HIVE-7241 Project: Hive Issue Type: Bug Components: Locking Affects Versions: 0.13.0 Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-7241.patch, HIVE-7241.patch Doing an alter table foo partition (bar='x') rename to partition (bar='y') acquires a read lock on table foo. It should instead acquire an exclusive lock on partition bar=x. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7266) Optimized HashTable with vectorized map-joins results in String columns extending
[
https://issues.apache.org/jira/browse/HIVE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-7266:
--
Attachment: hive-7266-small-test.tgz
Optimized HashTable with vectorized map-joins results in String columns
extending
-
Key: HIVE-7266
URL: https://issues.apache.org/jira/browse/HIVE-7266
Project: Hive
Issue Type: Bug
Components: Vectorization
Affects Versions: 0.14.0
Reporter: Gopal V
Assignee: Jitendra Nath Pandey
Attachments: hive-7266-small-test.tgz
The following query returns different results when both vectorized mapjoin
and the new optimized hashtable are enabled.
{code}
hive set hive.vectorized.execution.enabled=false;
hive select s_suppkey, n_name from supplier, nation where s_nationkey =
n_nationkey limit 25;
...
316869 JAPAN
1636869 RUSSIA
1096869 IRAN
7236869 RUSSIA
2276869 INDIA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIA
2616869 FRANCE
{code}
But when vectorization is enabled, the results are
{code}
316869 JAPAN
1636869 RUSSIA
1096869 IRANIA
7236869 RUSSIA
2276869 INDIAA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIAQUE
2616869 FRANCEAQUE
{code}
When the new optimized map-join hashtable is disabled as well
{code}
hive set hive.vectorized.execution.enabled=true;
hive set hive.mapjoin.optimized.hashtable=false;
hive select s_suppkey, n_name from supplier, nation where s_nationkey =
n_nationkey limit 25;
316869 JAPAN
1636869 RUSSIA
1096869 IRAN
7236869 RUSSIA
2276869 INDIA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIA
2616869 FRANCE
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7118) Oracle upgrade schema scripts do not map Java long datatype columns correctly for transaction related tables
[
https://issues.apache.org/jira/browse/HIVE-7118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039106#comment-14039106
]
Deepesh Khandelwal commented on HIVE-7118:
--
Its not clear to me as to where would the upgrade script
(019-HIVE-7118.oracle.sql) be invoked from. It may not be desirable to call
this from upgrade-0.12.0-to-0.13.0.oracle.sql script as people will miss it as
they are already on 0.13. What do you think?
Oracle upgrade schema scripts do not map Java long datatype columns correctly
for transaction related tables
Key: HIVE-7118
URL: https://issues.apache.org/jira/browse/HIVE-7118
Project: Hive
Issue Type: Bug
Components: Database/Schema
Affects Versions: 0.14.0
Environment: Oracle DB
Reporter: Deepesh Khandelwal
Assignee: Deepesh Khandelwal
Fix For: 0.14.0
Attachments: HIVE-7118.1.patch
In Transaction related tables, Java long column fields are mapped to
NUMBER(10) which results in failure to persist the transaction ids which are
incompatible. Following error is seen:
{noformat}
ORA-01438: value larger than specified precision allowed for this column
{noformat}
NO PRECOMMIT TESTS
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7118) Oracle upgrade schema scripts do not map Java long datatype columns correctly for transaction related tables
[
https://issues.apache.org/jira/browse/HIVE-7118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039119#comment-14039119
]
Alan Gates commented on HIVE-7118:
--
Good point. For the Oracle scripts we should fix the hive-schema-0.13 and
hive-txn-schema-0.13 scripts so they don't need to call the upgrade. If
they've already installed 13 and need to fix this bug they can just call the
019-HIVE-7118 script manually.
Oracle upgrade schema scripts do not map Java long datatype columns correctly
for transaction related tables
Key: HIVE-7118
URL: https://issues.apache.org/jira/browse/HIVE-7118
Project: Hive
Issue Type: Bug
Components: Database/Schema
Affects Versions: 0.14.0
Environment: Oracle DB
Reporter: Deepesh Khandelwal
Assignee: Deepesh Khandelwal
Fix For: 0.14.0
Attachments: HIVE-7118.1.patch
In Transaction related tables, Java long column fields are mapped to
NUMBER(10) which results in failure to persist the transaction ids which are
incompatible. Following error is seen:
{noformat}
ORA-01438: value larger than specified precision allowed for this column
{noformat}
NO PRECOMMIT TESTS
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7246) Hive transaction manager hardwires bonecp as the JDBC pooling implementation
[
https://issues.apache.org/jira/browse/HIVE-7246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039154#comment-14039154
]
Hive QA commented on HIVE-7246:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12651354/HIVE-7246.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 5668 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/530/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/530/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-530/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12651354
Hive transaction manager hardwires bonecp as the JDBC pooling implementation
Key: HIVE-7246
URL: https://issues.apache.org/jira/browse/HIVE-7246
Project: Hive
Issue Type: Bug
Components: Transactions
Affects Versions: 0.13.0
Reporter: Alan Gates
Assignee: Alan Gates
Attachments: HIVE-7246.patch
Currently TxnManager hardwires BoneCP as the JDBC connection pooling
implementation. Instead it should use the same connection pooling that the
metastore does.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7118) Oracle upgrade schema scripts do not map Java long datatype columns correctly for transaction related tables
[
https://issues.apache.org/jira/browse/HIVE-7118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepesh Khandelwal updated HIVE-7118:
-
Attachment: HIVE-7118-0.13.0.1.patch
I have attached a new patch which provides the following:
019-HIVE-7118.oracle.sql - Intended for users who are already on Hive 0.13.0.
Will need to run the script manually against their existing hive metastore
schema.
hive-txn-schema-0.13.0.oracle.sql hive-schema-0.13.0.oracle.sql - For fresh
installs.
Oracle upgrade schema scripts do not map Java long datatype columns correctly
for transaction related tables
Key: HIVE-7118
URL: https://issues.apache.org/jira/browse/HIVE-7118
Project: Hive
Issue Type: Bug
Components: Database/Schema
Affects Versions: 0.14.0
Environment: Oracle DB
Reporter: Deepesh Khandelwal
Assignee: Deepesh Khandelwal
Fix For: 0.14.0
Attachments: HIVE-7118-0.13.0.1.patch, HIVE-7118.1.patch
In Transaction related tables, Java long column fields are mapped to
NUMBER(10) which results in failure to persist the transaction ids which are
incompatible. Following error is seen:
{noformat}
ORA-01438: value larger than specified precision allowed for this column
{noformat}
NO PRECOMMIT TESTS
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7167) Hive Metastore fails to start with SQLServerException
[ https://issues.apache.org/jira/browse/HIVE-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HIVE-7167: Attachment: HIVE-7167.2.patch Hive Metastore fails to start with SQLServerException - Key: HIVE-7167 URL: https://issues.apache.org/jira/browse/HIVE-7167 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.13.0 Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou Labels: patch,, test Attachments: HIVE-7167.1.patch, HIVE-7167.2.patch In the case that hiveserver2 uses embedded metastore and hiveserver uses remote metastore, this exception comes up when hiveserver2 and hiveserver are started simultaneously. metastore service status is running but when I launch hive cli, I get following metastore connection error: C:\apps\dist\hive-0.13.0.2.1.2.0-1660\binhive.cmd 14/05/09 17:40:03 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no l onger has any effect. Use hive.hmshandler.retry.* instead Logging initialized using configuration in file:/C:/apps/dist/hive-0.13.0.2.1.2. 0-1660/conf/hive-log4j.properties Exception in thread main java.lang.RuntimeException: java.lang.RuntimeExceptio n: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:347) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.h ive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1413) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(Retry ingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret ryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja va:2444) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2456) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:341) ... 7 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct orAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC onstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1411) ... 12 more Caused by: MetaException(message:Could not connect to meta store using any of th e URIs provided. Most recent failure: org.apache.thrift.transport.TTransportExce ption: java.net.ConnectException: Connection refused: connect at org.apache.thrift.transport.TSocket.open(TSocket.java:185) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaSto reClient.java:336) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.init(HiveMetaS toreClient.java:214) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct orAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC onstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1411) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(Retry ingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret ryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja va:2444) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2456) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:341) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at
[jira] [Commented] (HIVE-7249) HiveTxnManager.closeTxnManger() throws if called after commitTxn()
[
https://issues.apache.org/jira/browse/HIVE-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039329#comment-14039329
]
Hive QA commented on HIVE-7249:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12651359/HIVE-7249.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 5669 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/531/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/531/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-531/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12651359
HiveTxnManager.closeTxnManger() throws if called after commitTxn()
--
Key: HIVE-7249
URL: https://issues.apache.org/jira/browse/HIVE-7249
Project: Hive
Issue Type: Bug
Components: Locking
Affects Versions: 0.13.1
Reporter: Eugene Koifman
Assignee: Alan Gates
Attachments: HIVE-7249.patch
I openTxn() and acquireLocks() for a query that looks like INSERT INTO T
PARTITION(p) SELECT * FROM T.
Then I call commitTxn(). Then I call closeTxnManger() I get an exception
saying lock not found (the only lock in this txn). So it seems TxnMgr
doesn't know that commit released the locks.
Here is the stack trace and some log output which maybe useful:
{noformat}
2014-06-17 15:54:40,771 DEBUG mapreduce.TransactionContext
(TransactionContext.java:onCommitJob(128)) -
onCommitJob(job_local557130041_0001). this=46719652
2014-06-17 15:54:40,771 DEBUG lockmgr.DbTxnManager
(DbTxnManager.java:commitTxn(205)) - Committing txn 1
2014-06-17 15:54:40,771 DEBUG txn.TxnHandler (TxnHandler.java:getDbTime(872))
- Going to execute query values current_timestamp
2014-06-17 15:54:40,772 DEBUG txn.TxnHandler
(TxnHandler.java:heartbeatTxn(1423)) - Going to execute query select
txn_state from TXNS where txn_id = 1 for\
update
2014-06-17 15:54:40,773 DEBUG txn.TxnHandler
(TxnHandler.java:heartbeatTxn(1438)) - Going to execute update update TXNS
set txn_last_heartbeat = 140304568\
0772 where txn_id = 1
2014-06-17 15:54:40,778 DEBUG txn.TxnHandler
(TxnHandler.java:heartbeatTxn(1440)) - Going to commit
2014-06-17 15:54:40,779 DEBUG txn.TxnHandler (TxnHandler.java:commitTxn(344))
- Going to execute insert insert into COMPLETED_TXN_COMPONENTS select tc_txn\
id, tc_database, tc_table, tc_partition from TXN_COMPONENTS where tc_txnid =
1
2014-06-17 15:54:40,784 DEBUG txn.TxnHandler (TxnHandler.java:commitTxn(352))
- Going to execute update delete from TXN_COMPONENTS where tc_txnid = 1
2014-06-17 15:54:40,788 DEBUG txn.TxnHandler (TxnHandler.java:commitTxn(356))
- Going to execute update delete from HIVE_LOCKS where hl_txnid = 1
2014-06-17 15:54:40,791 DEBUG txn.TxnHandler (TxnHandler.java:commitTxn(359))
- Going to execute update delete from TXNS where txn_id = 1
2014-06-17 15:54:40,794 DEBUG txn.TxnHandler (TxnHandler.java:commitTxn(361))
- Going to commit
2014-06-17 15:54:40,795 WARN mapreduce.TransactionContext
(TransactionContext.java:cleanup(317)) -
cleanupJob(JobID=job_local557130041_0001)this=46719652
2014-06-17 15:54:40,795 DEBUG lockmgr.DbLockManager
(DbLockManager.java:unlock(109)) - Unlocking id:1
2014-06-17 15:54:40,796 DEBUG txn.TxnHandler (TxnHandler.java:getDbTime(872))
- Going to execute query values current_timestamp
2014-06-17 15:54:40,796 DEBUG txn.TxnHandler
(TxnHandler.java:heartbeatLock(1402)) - Going to execute update update
HIVE_LOCKS set hl_last_heartbeat = 140\
3045680796 where hl_lock_ext_id = 1
2014-06-17 15:54:40,800 DEBUG txn.TxnHandler
(TxnHandler.java:heartbeatLock(1405)) - Going to rollback
2014-06-17 15:54:40,804 ERROR metastore.RetryingHMSHandler
(RetryingHMSHandler.java:invoke(143)) - NoSuchLockException(message:No such
lock: 1)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.heartbeatLock(TxnHandler.java:1407)
at
[jira] [Commented] (HIVE-7167) Hive Metastore fails to start with SQLServerException
[ https://issues.apache.org/jira/browse/HIVE-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039334#comment-14039334 ] Xiaobing Zhou commented on HIVE-7167: - Thanks [~sershe], 1) Here's the exception. 2) it's because these three methods are potential places that causes contention when two metastores both try to initialize DB in the same time. 3) I don't think so. If one tried and failed in the first time, JDOException is caught for one more time try, if failed again, simply ignored, means another metastore succeeds. 2014-05-15 21:06:01,659 WARN server.HiveServer2 (HiveServer2.java:startHiveServer2(112)) - Error starting HiveServer2 on attempt 1, will retry in 60 seconds java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:347) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:292) at org.apache.hive.service.cli.session.SessionManager.applyAuthorizationConfigPolicy(SessionManager.java:88) at org.apache.hive.service.cli.session.SessionManager.init(SessionManager.java:63) at org.apache.hive.service.CompositeService.init(CompositeService.java:59) at org.apache.hive.service.cli.CLIService.init(CLIService.java:110) at org.apache.hive.service.CompositeService.init(CompositeService.java:59) at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:68) at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:100) at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:149) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1413) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:341) ... 9 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1411) ... 14 more Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastore NestedThrowables: java.sql.BatchUpdateException: Violation of UNIQUE KEY constraint 'GLOBALPRIVILEGEINDEX'. Cannot insert duplicate key in object 'dbo.GLOBAL_PRIVS'. The duplicate key value is (admin, ROLE, All, admin, ROLE). at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) at org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:406) at org.apache.hadoop.hive.metastore.ObjectStore.grantPrivileges(ObjectStore.java:3877) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108) at $Proxy10.grantPrivileges(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultRoles(HiveMetaStore.java:567) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:398) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:356) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.init(RetryingHMSHandler.java:54) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59) at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.init(HiveMetaStoreClient.java:171) ... 19 more Caused by: java.sql.BatchUpdateException: Violation of
[jira] [Commented] (HIVE-7220) Empty dir in external table causes issue (root_dir_external_table.q failure)
[
https://issues.apache.org/jira/browse/HIVE-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039337#comment-14039337
]
Gunther Hagleitner commented on HIVE-7220:
--
You're right, the IsValidSplit only drops directory only splits. My bad.
Empty dir in external table causes issue (root_dir_external_table.q failure)
Key: HIVE-7220
URL: https://issues.apache.org/jira/browse/HIVE-7220
Project: Hive
Issue Type: Bug
Reporter: Szehon Ho
Assignee: Szehon Ho
Attachments: HIVE-7220.patch
While looking at root_dir_external_table.q failure, which is doing a query on
an external table located at root ('/'), I noticed that latest Hadoop2
CombineFileInputFormat returns split representing empty directories (like
'/Users'), which leads to failure in Hive's CombineFileRecordReader as it
tries to open the directory for processing.
Tried with an external table in a normal HDFS directory, and it also returns
the same error. Looks like a real bug.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6584) Add HiveHBaseTableSnapshotInputFormat
[
https://issues.apache.org/jira/browse/HIVE-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039340#comment-14039340
]
Hive QA commented on HIVE-6584:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12651366/HIVE-6584.5.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/532/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/532/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-532/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO] -
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[10,66]
package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not
exist
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[17,17]
cannot find symbol
symbol: class TableSnapshotRegionSplit
location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[24,29]
cannot find symbol
symbol: class TableSnapshotRegionSplit
location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[29,10]
cannot find symbol
symbol: class TableSnapshotRegionSplit
location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[28,41]
cannot find symbol
symbol: class TableSnapshotInputFormat
location: package org.apache.hadoop.hbase.mapreduce
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[31,66]
package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not
exist
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[33,47]
cannot find symbol
symbol: class ColumnMapping
location: class org.apache.hadoop.hive.hbase.HBaseSerDe
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,3]
cannot find symbol
symbol: class TableSnapshotInputFormat
location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[34,41]
cannot find symbol
symbol: class TableSnapshotInputFormatImpl
location: package org.apache.hadoop.hbase.mapreduce
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[37,47]
cannot find symbol
symbol: class ColumnMapping
location: class org.apache.hadoop.hive.hbase.HBaseSerDe
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[21,17]
cannot find symbol
symbol: class TableSnapshotRegionSplit
location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,43]
cannot find symbol
symbol: class TableSnapshotInputFormat
location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[87,10]
cannot find symbol
symbol: class ColumnMapping
location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[89,54]
incompatible types
required: java.util.ListColumnMapping
found:org.apache.hadoop.hive.hbase.ColumnMappings
[ERROR]
/data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[108,9]
cannot find symbol
symbol: variable HiveHBaseInputFormatUtil
location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
[ERROR]
Re: Review Request 22191: HIVE-7167: Hive Metastore fails to start with SQLServerException
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22191/ --- (Updated June 20, 2014, 8:55 p.m.) Review request for hive. Bugs: HIVE-7167 https://issues.apache.org/jira/browse/HIVE-7167 Repository: hive-git Description --- In the case that hiveserver2 uses embedded metastore and hiveserver uses remote metastore, this exception comes up when hiveserver2 and hiveserver are started simultaneously. See the bug link: https://issues.apache.org/jira/browse/HIVE-7167 The idea is to catch JDOException from concurrent defaultDB and roles initializations or privilege insertions, and try createDefaultDB, createDefaultRoles and addAdminUsers one more time, if still fails, simply throw up exceptions. Diffs - metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 487d292 Diff: https://reviews.apache.org/r/22191/diff/ Testing --- done Thanks, XIAOBING ZHOU
Re: Review Request 22191: HIVE-7167: Hive Metastore fails to start with SQLServerException
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22191/ --- (Updated June 20, 2014, 8:55 p.m.) Review request for hive. Changes --- new patch HIVE-7167.2.patch is made on trunk and attached. Bugs: HIVE-7167 https://issues.apache.org/jira/browse/HIVE-7167 Repository: hive-git Description --- In the case that hiveserver2 uses embedded metastore and hiveserver uses remote metastore, this exception comes up when hiveserver2 and hiveserver are started simultaneously. See the bug link: https://issues.apache.org/jira/browse/HIVE-7167 The idea is to catch JDOException from concurrent defaultDB and roles initializations or privilege insertions, and try createDefaultDB, createDefaultRoles and addAdminUsers one more time, if still fails, simply throw up exceptions. Diffs (updated) - metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 487d292 Diff: https://reviews.apache.org/r/22191/diff/ Testing --- done Thanks, XIAOBING ZHOU
[jira] [Updated] (HIVE-7167) Hive Metastore fails to start with SQLServerException
[ https://issues.apache.org/jira/browse/HIVE-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HIVE-7167: Resolution: Fixed Fix Version/s: 0.14.0 Release Note: Marked it as resolved due to HIVE-7167.2.patch Status: Resolved (was: Patch Available) Hive Metastore fails to start with SQLServerException - Key: HIVE-7167 URL: https://issues.apache.org/jira/browse/HIVE-7167 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.13.0 Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou Labels: patch,, test Fix For: 0.14.0 Attachments: HIVE-7167.1.patch, HIVE-7167.2.patch In the case that hiveserver2 uses embedded metastore and hiveserver uses remote metastore, this exception comes up when hiveserver2 and hiveserver are started simultaneously. metastore service status is running but when I launch hive cli, I get following metastore connection error: C:\apps\dist\hive-0.13.0.2.1.2.0-1660\binhive.cmd 14/05/09 17:40:03 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no l onger has any effect. Use hive.hmshandler.retry.* instead Logging initialized using configuration in file:/C:/apps/dist/hive-0.13.0.2.1.2. 0-1660/conf/hive-log4j.properties Exception in thread main java.lang.RuntimeException: java.lang.RuntimeExceptio n: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:347) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.h ive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1413) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(Retry ingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret ryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja va:2444) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2456) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:341) ... 7 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct orAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC onstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1411) ... 12 more Caused by: MetaException(message:Could not connect to meta store using any of th e URIs provided. Most recent failure: org.apache.thrift.transport.TTransportExce ption: java.net.ConnectException: Connection refused: connect at org.apache.thrift.transport.TSocket.open(TSocket.java:185) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaSto reClient.java:336) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.init(HiveMetaS toreClient.java:214) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct orAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC onstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1411) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(Retry ingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret ryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja va:2444) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2456) at
[jira] [Updated] (HIVE-7266) Optimized HashTable with vectorized map-joins results in String columns extending
[
https://issues.apache.org/jira/browse/HIVE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-7266:
--
Description:
The following query returns different results when both vectorized mapjoin and
the new optimized hashtable are enabled.
{code}
hive set hive.vectorized.execution.enabled=false;
hive select s_suppkey, n_name from supplier, nation where s_nationkey =
n_nationkey limit 25;
...
316869 JAPAN
1636869 RUSSIA
1096869 IRAN
7236869 RUSSIA
2276869 INDIA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIA
2616869 FRANCE
{code}
But when vectorization is enabled, the results are
{code}
316869 JAPAN
1636869 RUSSIA
1096869 IRANIA
7236869 RUSSIA
2276869 INDIAA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIAQUE
2616869 FRANCEAQUE
{code}
it works correctly with vectorization when the new optimized map-join hashtable
is disabled
{code}
hive set hive.vectorized.execution.enabled=true;
hive set hive.mapjoin.optimized.hashtable=false;
hive select s_suppkey, n_name from supplier, nation where s_nationkey =
n_nationkey limit 25;
316869 JAPAN
1636869 RUSSIA
1096869 IRAN
7236869 RUSSIA
2276869 INDIA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIA
2616869 FRANCE
{code}
was:
The following query returns different results when both vectorized mapjoin and
the new optimized hashtable are enabled.
{code}
hive set hive.vectorized.execution.enabled=false;
hive select s_suppkey, n_name from supplier, nation where s_nationkey =
n_nationkey limit 25;
...
316869 JAPAN
1636869 RUSSIA
1096869 IRAN
7236869 RUSSIA
2276869 INDIA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIA
2616869 FRANCE
{code}
But when vectorization is enabled, the results are
{code}
316869 JAPAN
1636869 RUSSIA
1096869 IRANIA
7236869 RUSSIA
2276869 INDIAA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIAQUE
2616869 FRANCEAQUE
{code}
When the new optimized map-join hashtable is disabled as well
{code}
hive set hive.vectorized.execution.enabled=true;
hive set hive.mapjoin.optimized.hashtable=false;
hive select s_suppkey, n_name from supplier, nation where s_nationkey =
n_nationkey limit 25;
316869 JAPAN
1636869 RUSSIA
1096869 IRAN
7236869 RUSSIA
2276869 INDIA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIA
2616869 FRANCE
{code}
Optimized HashTable with vectorized map-joins results in String columns
extending
-
Key: HIVE-7266
URL: https://issues.apache.org/jira/browse/HIVE-7266
Project: Hive
Issue Type: Bug
Components: Tez, Vectorization
Affects Versions: 0.14.0
Reporter: Gopal V
Assignee: Jitendra Nath Pandey
Attachments: hive-7266-small-test.tgz
The following query returns different results when both vectorized mapjoin
and the new optimized hashtable are enabled.
{code}
hive set hive.vectorized.execution.enabled=false;
hive select s_suppkey, n_name from supplier, nation where s_nationkey =
n_nationkey limit 25;
...
316869 JAPAN
1636869 RUSSIA
1096869 IRAN
7236869 RUSSIA
2276869 INDIA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIA
2616869 FRANCE
{code}
But when vectorization is enabled, the results are
{code}
316869 JAPAN
1636869 RUSSIA
1096869 IRANIA
7236869 RUSSIA
2276869 INDIAA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIAQUE
2616869 FRANCEAQUE
{code}
it works correctly with vectorization when the new optimized map-join
hashtable is disabled
{code}
hive set hive.vectorized.execution.enabled=true;
hive set hive.mapjoin.optimized.hashtable=false;
hive select s_suppkey, n_name from supplier, nation where s_nationkey =
n_nationkey limit 25;
316869 JAPAN
1636869 RUSSIA
1096869 IRAN
7236869 RUSSIA
2276869 INDIA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIA
2616869 FRANCE
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6622) UDF translate doesn't take either CHAR or VARCHAR as any of its arguments
[
https://issues.apache.org/jira/browse/HIVE-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-6622:
--
Resolution: Fixed
Fix Version/s: 0.14.0
Status: Resolved (was: Patch Available)
Patch committed to trunk. Thanks Wilbur for the contribution.
UDF translate doesn't take either CHAR or VARCHAR as any of its arguments
-
Key: HIVE-6622
URL: https://issues.apache.org/jira/browse/HIVE-6622
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.12.0
Reporter: Xuefu Zhang
Assignee: Wilbur Yang
Fix For: 0.14.0
Attachments: HIVE-6622.1.patch
UDF translate(input, from, to) doesn't accept any CHAR or VARCHAR typed
argument.
{code}
hive desc vc;
c char(5) None
vcvarchar(7) None
s string None
hive select translate(my input, c, ll) from vc;
FAILED: SemanticException [Error 10016]: Line 1:29 Argument type mismatch
'c': A string argument was expected but an argument of type char(5) was given.
{code}
However, if the type is String, then the UDF works fine.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Review Request 22772: HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input
On June 20, 2014, 1:04 a.m., Jason Dere wrote: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java, line 99 https://reviews.apache.org/r/22772/diff/2/?file=613098#file613098line99 Rather than having to compare the actual class/class name of the type, use call PrimitiveObjectInpsector.getPrimitiveCategory() which returns an enum corresponding to the type. Take a look at GenericUDFPrintf, where Xuefu made similar changes to the printf() function to support char/varchar. Jason, good suggestion. Made required changes. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22772/#review46246 --- On June 20, 2014, 9:15 p.m., Ashish Singh wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22772/ --- (Updated June 20, 2014, 9:15 p.m.) Review request for hive. Bugs: HIVE-6637 https://issues.apache.org/jira/browse/HIVE-6637 Repository: hive-git Description --- HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input Diffs - ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java ea52537d0b85191f0b633a29aa3f7ddb556c288d ql/src/test/queries/clientpositive/udf_in_file.q 9d9efe8e23d6e73429ee5cd2c8470359ba2b3498 ql/src/test/results/clientpositive/udf_in_file.q.out b63143760d80f3f6a8ba0a23c0d87e8bb86fce66 Diff: https://reviews.apache.org/r/22772/diff/ Testing --- Tested with qtest. Thanks, Ashish Singh
Re: Review Request 22772: HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22772/ --- (Updated June 20, 2014, 9:15 p.m.) Review request for hive. Changes --- Changes based on review. Bugs: HIVE-6637 https://issues.apache.org/jira/browse/HIVE-6637 Repository: hive-git Description --- HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java ea52537d0b85191f0b633a29aa3f7ddb556c288d ql/src/test/queries/clientpositive/udf_in_file.q 9d9efe8e23d6e73429ee5cd2c8470359ba2b3498 ql/src/test/results/clientpositive/udf_in_file.q.out b63143760d80f3f6a8ba0a23c0d87e8bb86fce66 Diff: https://reviews.apache.org/r/22772/diff/ Testing --- Tested with qtest. Thanks, Ashish Singh
Re: Review Request 22772: HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22772/ --- (Updated June 20, 2014, 9:28 p.m.) Review request for hive. Changes --- Remove redundant imports. Bugs: HIVE-6637 https://issues.apache.org/jira/browse/HIVE-6637 Repository: hive-git Description --- HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java ea52537d0b85191f0b633a29aa3f7ddb556c288d ql/src/test/queries/clientpositive/udf_in_file.q 9d9efe8e23d6e73429ee5cd2c8470359ba2b3498 ql/src/test/results/clientpositive/udf_in_file.q.out b63143760d80f3f6a8ba0a23c0d87e8bb86fce66 Diff: https://reviews.apache.org/r/22772/diff/ Testing --- Tested with qtest. Thanks, Ashish Singh
[jira] [Created] (HIVE-7268) On Windows Hive jobs in Webhcat always run on default MR mode
Deepesh Khandelwal created HIVE-7268: Summary: On Windows Hive jobs in Webhcat always run on default MR mode Key: HIVE-7268 URL: https://issues.apache.org/jira/browse/HIVE-7268 Project: Hive Issue Type: Bug Components: WebHCat Affects Versions: 0.13.0 Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal Fix For: 0.14.0 On Windows fix from HIVE-7065 doesn't work as the templeton.cmd script does not include the Hive configuration directory in the classpath. So when hive.execution.engine property is set to tez in HIVE_CONF_DIR/hive-site.xml, webhcat doesn't see it and defaults it to mr. This prevents Hive jobs running from WebHCat to use the tez execution engine. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7167) Hive Metastore fails to start with SQLServerException
[ https://issues.apache.org/jira/browse/HIVE-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HIVE-7167: Fix Version/s: (was: 0.14.0) Hive Metastore fails to start with SQLServerException - Key: HIVE-7167 URL: https://issues.apache.org/jira/browse/HIVE-7167 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.14.0 Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou Labels: patch,, test Attachments: HIVE-7167.1.patch, HIVE-7167.2.patch In the case that hiveserver2 uses embedded metastore and hiveserver uses remote metastore, this exception comes up when hiveserver2 and hiveserver are started simultaneously. metastore service status is running but when I launch hive cli, I get following metastore connection error: C:\apps\dist\hive-0.13.0.2.1.2.0-1660\binhive.cmd 14/05/09 17:40:03 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no l onger has any effect. Use hive.hmshandler.retry.* instead Logging initialized using configuration in file:/C:/apps/dist/hive-0.13.0.2.1.2. 0-1660/conf/hive-log4j.properties Exception in thread main java.lang.RuntimeException: java.lang.RuntimeExceptio n: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:347) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.h ive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1413) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(Retry ingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret ryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja va:2444) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2456) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:341) ... 7 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct orAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC onstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1411) ... 12 more Caused by: MetaException(message:Could not connect to meta store using any of th e URIs provided. Most recent failure: org.apache.thrift.transport.TTransportExce ption: java.net.ConnectException: Connection refused: connect at org.apache.thrift.transport.TSocket.open(TSocket.java:185) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaSto reClient.java:336) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.init(HiveMetaS toreClient.java:214) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct orAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC onstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1411) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(Retry ingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret ryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja va:2444) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2456) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:341) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at
[jira] [Updated] (HIVE-7167) Hive Metastore fails to start with SQLServerException
[ https://issues.apache.org/jira/browse/HIVE-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HIVE-7167: Affects Version/s: (was: 0.13.0) 0.14.0 Hive Metastore fails to start with SQLServerException - Key: HIVE-7167 URL: https://issues.apache.org/jira/browse/HIVE-7167 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.14.0 Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou Labels: patch,, test Attachments: HIVE-7167.1.patch, HIVE-7167.2.patch In the case that hiveserver2 uses embedded metastore and hiveserver uses remote metastore, this exception comes up when hiveserver2 and hiveserver are started simultaneously. metastore service status is running but when I launch hive cli, I get following metastore connection error: C:\apps\dist\hive-0.13.0.2.1.2.0-1660\binhive.cmd 14/05/09 17:40:03 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no l onger has any effect. Use hive.hmshandler.retry.* instead Logging initialized using configuration in file:/C:/apps/dist/hive-0.13.0.2.1.2. 0-1660/conf/hive-log4j.properties Exception in thread main java.lang.RuntimeException: java.lang.RuntimeExceptio n: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:347) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.h ive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1413) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(Retry ingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret ryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja va:2444) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2456) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:341) ... 7 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct orAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC onstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1411) ... 12 more Caused by: MetaException(message:Could not connect to meta store using any of th e URIs provided. Most recent failure: org.apache.thrift.transport.TTransportExce ption: java.net.ConnectException: Connection refused: connect at org.apache.thrift.transport.TSocket.open(TSocket.java:185) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaSto reClient.java:336) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.init(HiveMetaS toreClient.java:214) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct orAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC onstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1411) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(Retry ingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret ryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja va:2444) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2456) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:341) at
[jira] [Reopened] (HIVE-7167) Hive Metastore fails to start with SQLServerException
[ https://issues.apache.org/jira/browse/HIVE-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou reopened HIVE-7167: - closed by mistake, reopen it. Hive Metastore fails to start with SQLServerException - Key: HIVE-7167 URL: https://issues.apache.org/jira/browse/HIVE-7167 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.14.0 Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou Labels: patch,, test Attachments: HIVE-7167.1.patch, HIVE-7167.2.patch In the case that hiveserver2 uses embedded metastore and hiveserver uses remote metastore, this exception comes up when hiveserver2 and hiveserver are started simultaneously. metastore service status is running but when I launch hive cli, I get following metastore connection error: C:\apps\dist\hive-0.13.0.2.1.2.0-1660\binhive.cmd 14/05/09 17:40:03 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no l onger has any effect. Use hive.hmshandler.retry.* instead Logging initialized using configuration in file:/C:/apps/dist/hive-0.13.0.2.1.2. 0-1660/conf/hive-log4j.properties Exception in thread main java.lang.RuntimeException: java.lang.RuntimeExceptio n: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:347) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.h ive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1413) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(Retry ingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret ryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja va:2444) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2456) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:341) ... 7 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct orAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC onstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1411) ... 12 more Caused by: MetaException(message:Could not connect to meta store using any of th e URIs provided. Most recent failure: org.apache.thrift.transport.TTransportExce ption: java.net.ConnectException: Connection refused: connect at org.apache.thrift.transport.TSocket.open(TSocket.java:185) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaSto reClient.java:336) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.init(HiveMetaS toreClient.java:214) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct orAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC onstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1411) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(Retry ingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret ryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja va:2444) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2456) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:341) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at
[jira] [Updated] (HIVE-6637) UDF in_file() doesn't take CHAR or VARCHAR as input
[
https://issues.apache.org/jira/browse/HIVE-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Kumar Singh updated HIVE-6637:
-
Attachment: HIVE-6637.2.patch
Updated based on review.
UDF in_file() doesn't take CHAR or VARCHAR as input
---
Key: HIVE-6637
URL: https://issues.apache.org/jira/browse/HIVE-6637
Project: Hive
Issue Type: Bug
Components: Types, UDF
Affects Versions: 0.14.0
Reporter: Xuefu Zhang
Assignee: Ashish Kumar Singh
Attachments: HIVE-6637.1.patch, HIVE-6637.2.patch
{code}
hive desc alter_varchar_1;
key string None
value varchar(3) None
key2 int None
value2varchar(10) None
hive select in_file(value, value2) from alter_varchar_1;
FAILED: SemanticException [Error 10016]: Line 1:15 Argument type mismatch
'value': The 1st argument of function IN_FILE must be a string but
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableHiveVarcharObjectInspector@10f1f34a
was given.
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7167) Hive Metastore fails to start with SQLServerException
[ https://issues.apache.org/jira/browse/HIVE-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HIVE-7167: Release Note: (was: Marked it as resolved due to HIVE-7167.2.patch) Status: Patch Available (was: Reopened) HIVE-7167.2.patch is available. Hive Metastore fails to start with SQLServerException - Key: HIVE-7167 URL: https://issues.apache.org/jira/browse/HIVE-7167 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.14.0 Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou Labels: patch,, test Attachments: HIVE-7167.1.patch, HIVE-7167.2.patch In the case that hiveserver2 uses embedded metastore and hiveserver uses remote metastore, this exception comes up when hiveserver2 and hiveserver are started simultaneously. metastore service status is running but when I launch hive cli, I get following metastore connection error: C:\apps\dist\hive-0.13.0.2.1.2.0-1660\binhive.cmd 14/05/09 17:40:03 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no l onger has any effect. Use hive.hmshandler.retry.* instead Logging initialized using configuration in file:/C:/apps/dist/hive-0.13.0.2.1.2. 0-1660/conf/hive-log4j.properties Exception in thread main java.lang.RuntimeException: java.lang.RuntimeExceptio n: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:347) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.h ive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1413) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(Retry ingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret ryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja va:2444) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2456) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav a:341) ... 7 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct orAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC onstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1411) ... 12 more Caused by: MetaException(message:Could not connect to meta store using any of th e URIs provided. Most recent failure: org.apache.thrift.transport.TTransportExce ption: java.net.ConnectException: Connection refused: connect at org.apache.thrift.transport.TSocket.open(TSocket.java:185) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaSto reClient.java:336) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.init(HiveMetaS toreClient.java:214) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct orAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC onstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore Utils.java:1411) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(Retry ingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret ryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja va:2444) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2456) at
[jira] [Updated] (HIVE-7268) On Windows Hive jobs in Webhcat always run on default MR mode
[ https://issues.apache.org/jira/browse/HIVE-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepesh Khandelwal updated HIVE-7268: - Attachment: HIVE-7268.1.patch Attaching a patch which adds HIVE_HOME/conf to the webhcat classpath on Windows. On Windows Hive jobs in Webhcat always run on default MR mode - Key: HIVE-7268 URL: https://issues.apache.org/jira/browse/HIVE-7268 Project: Hive Issue Type: Bug Components: WebHCat Affects Versions: 0.13.0 Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal Fix For: 0.14.0 Attachments: HIVE-7268.1.patch On Windows fix from HIVE-7065 doesn't work as the templeton.cmd script does not include the Hive configuration directory in the classpath. So when hive.execution.engine property is set to tez in HIVE_CONF_DIR/hive-site.xml, webhcat doesn't see it and defaults it to mr. This prevents Hive jobs running from WebHCat to use the tez execution engine. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-7269) First query in ptf.q (Partition Table Function test) fails when input table is changed to ORC format
Matt McCline created HIVE-7269:
--
Summary: First query in ptf.q (Partition Table Function test)
fails when input table is changed to ORC format
Key: HIVE-7269
URL: https://issues.apache.org/jira/browse/HIVE-7269
Project: Hive
Issue Type: Bug
Reporter: Matt McCline
Assignee: Harish Butani
This fails:
{noformat}
CREATE TABLE partorc(
p_partkey INT,
p_name STRING,
p_mfgr STRING,
p_brand STRING,
p_type STRING,
p_size INT,
p_container STRING,
p_retailprice DOUBLE,
p_comment STRING
) STORED AS ORC;
LOAD DATA LOCAL INPATH '/Users/mmccline/hive_ptf/data/files/part_tiny.txt'
overwrite into table partorc;
select
p_mfgr,
p_name,
p_size,
rank()
over (partition by p_mfgr order by p_name) as r,
dense_rank()
over (partition by p_mfgr order by p_name) as dr,
sum(p_retailprice)
over (partition by p_mfgr order by p_name rows between unbounded preceding
and current row) as s1
from noop(on part
partition by p_mfgr
order by p_name
);
{noformat}
The same thing works when STORED AS ORC clause removed.
If you specify set hive.execution.engine=tez, you get these failure stack
traces for the ORC table.
{noformat}
14/06/20 15:05:33 [main]: ERROR tez.TezJobMonitor: Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1403230487252_0002_1_02,
diagnostics=[Task failed, taskId=task_1403230487252_0002_1_02_00,
diagnostics=[AttemptID:attempt_1403230487252_0002_1_02_00_0 Info:Error:
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException:
java.io.IOException: Malformed ORC file
hdfs://localhost:9000/user/hive/warehouse/partorc/part_tiny.txt. Invalid
postscript.
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:581)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:570)
Caused by: java.lang.RuntimeException: java.io.IOException:
java.io.IOException: Malformed ORC file
hdfs://localhost:9000/user/hive/warehouse/partorc/part_tiny.txt. Invalid
postscript.
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:174)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.init(TezGroupedSplitsInputFormat.java:113)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:79)
at
org.apache.tez.mapreduce.input.MRInput.setupOldRecordReader(MRInput.java:250)
at
org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:400)
at
org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:379)
at
org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:110)
at
org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:79)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142)
... 6 more
Caused by: java.io.IOException: java.io.IOException: Malformed ORC file
hdfs://localhost:9000/user/hive/warehouse/partorc/part_tiny.txt. Invalid
postscript.
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:243)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:171)
... 14 more
Caused by: java.io.IOException: Malformed ORC file
hdfs://localhost:9000/user/hive/warehouse/partorc/part_tiny.txt. Invalid
postscript.
at
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.ensureOrcFooter(ReaderImpl.java:226)
at
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:336)
at
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.init(ReaderImpl.java:292)
at
org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:201)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1010)
at
[jira] [Commented] (HIVE-7269) First query in ptf.q (Partition Table Function test) fails when input table is changed to ORC format
[
https://issues.apache.org/jira/browse/HIVE-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039462#comment-14039462
]
Matt McCline commented on HIVE-7269:
My problem (from a Google search)...
LOAD DATA just copies the files to hive datafiles. Hive does not do any
transformation while loading data into tables.
First query in ptf.q (Partition Table Function test) fails when input table
is changed to ORC format
Key: HIVE-7269
URL: https://issues.apache.org/jira/browse/HIVE-7269
Project: Hive
Issue Type: Bug
Reporter: Matt McCline
Assignee: Matt McCline
This fails:
{noformat}
CREATE TABLE partorc(
p_partkey INT,
p_name STRING,
p_mfgr STRING,
p_brand STRING,
p_type STRING,
p_size INT,
p_container STRING,
p_retailprice DOUBLE,
p_comment STRING
) STORED AS ORC;
LOAD DATA LOCAL INPATH '/Users/mmccline/hive_ptf/data/files/part_tiny.txt'
overwrite into table partorc;
select
p_mfgr,
p_name,
p_size,
rank()
over (partition by p_mfgr order by p_name) as r,
dense_rank()
over (partition by p_mfgr order by p_name) as dr,
sum(p_retailprice)
over (partition by p_mfgr order by p_name rows between unbounded
preceding and current row) as s1
from noop(on part
partition by p_mfgr
order by p_name
);
{noformat}
The same thing works when STORED AS ORC clause removed.
If you specify set hive.execution.engine=tez, you get these failure stack
traces for the ORC table.
{noformat}
14/06/20 15:05:33 [main]: ERROR tez.TezJobMonitor: Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1403230487252_0002_1_02,
diagnostics=[Task failed, taskId=task_1403230487252_0002_1_02_00,
diagnostics=[AttemptID:attempt_1403230487252_0002_1_02_00_0 Info:Error:
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException:
java.io.IOException: Malformed ORC file
hdfs://localhost:9000/user/hive/warehouse/partorc/part_tiny.txt. Invalid
postscript.
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:581)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:570)
Caused by: java.lang.RuntimeException: java.io.IOException:
java.io.IOException: Malformed ORC file
hdfs://localhost:9000/user/hive/warehouse/partorc/part_tiny.txt. Invalid
postscript.
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:174)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.init(TezGroupedSplitsInputFormat.java:113)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:79)
at
org.apache.tez.mapreduce.input.MRInput.setupOldRecordReader(MRInput.java:250)
at
org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:400)
at
org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:379)
at
org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:110)
at
org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:79)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142)
... 6 more
Caused by: java.io.IOException: java.io.IOException: Malformed ORC file
hdfs://localhost:9000/user/hive/warehouse/partorc/part_tiny.txt. Invalid
postscript.
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:243)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:171)
... 14 more
Caused by: java.io.IOException: Malformed ORC file
hdfs://localhost:9000/user/hive/warehouse/partorc/part_tiny.txt. Invalid
postscript.
at
[jira] [Resolved] (HIVE-7269) First query in ptf.q (Partition Table Function test) fails when input table is changed to ORC format
[
https://issues.apache.org/jira/browse/HIVE-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline resolved HIVE-7269.
Resolution: Not a Problem
First query in ptf.q (Partition Table Function test) fails when input table
is changed to ORC format
Key: HIVE-7269
URL: https://issues.apache.org/jira/browse/HIVE-7269
Project: Hive
Issue Type: Bug
Reporter: Matt McCline
Assignee: Matt McCline
This fails:
{noformat}
CREATE TABLE partorc(
p_partkey INT,
p_name STRING,
p_mfgr STRING,
p_brand STRING,
p_type STRING,
p_size INT,
p_container STRING,
p_retailprice DOUBLE,
p_comment STRING
) STORED AS ORC;
LOAD DATA LOCAL INPATH '/Users/mmccline/hive_ptf/data/files/part_tiny.txt'
overwrite into table partorc;
select
p_mfgr,
p_name,
p_size,
rank()
over (partition by p_mfgr order by p_name) as r,
dense_rank()
over (partition by p_mfgr order by p_name) as dr,
sum(p_retailprice)
over (partition by p_mfgr order by p_name rows between unbounded
preceding and current row) as s1
from noop(on part
partition by p_mfgr
order by p_name
);
{noformat}
The same thing works when STORED AS ORC clause removed.
If you specify set hive.execution.engine=tez, you get these failure stack
traces for the ORC table.
{noformat}
14/06/20 15:05:33 [main]: ERROR tez.TezJobMonitor: Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1403230487252_0002_1_02,
diagnostics=[Task failed, taskId=task_1403230487252_0002_1_02_00,
diagnostics=[AttemptID:attempt_1403230487252_0002_1_02_00_0 Info:Error:
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException:
java.io.IOException: Malformed ORC file
hdfs://localhost:9000/user/hive/warehouse/partorc/part_tiny.txt. Invalid
postscript.
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:581)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:570)
Caused by: java.lang.RuntimeException: java.io.IOException:
java.io.IOException: Malformed ORC file
hdfs://localhost:9000/user/hive/warehouse/partorc/part_tiny.txt. Invalid
postscript.
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:174)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.init(TezGroupedSplitsInputFormat.java:113)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:79)
at
org.apache.tez.mapreduce.input.MRInput.setupOldRecordReader(MRInput.java:250)
at
org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:400)
at
org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:379)
at
org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:110)
at
org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:79)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142)
... 6 more
Caused by: java.io.IOException: java.io.IOException: Malformed ORC file
hdfs://localhost:9000/user/hive/warehouse/partorc/part_tiny.txt. Invalid
postscript.
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:243)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:171)
... 14 more
Caused by: java.io.IOException: Malformed ORC file
hdfs://localhost:9000/user/hive/warehouse/partorc/part_tiny.txt. Invalid
postscript.
at
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.ensureOrcFooter(ReaderImpl.java:226)
at
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:336)
at
[jira] [Commented] (HIVE-6584) Add HiveHBaseTableSnapshotInputFormat
[
https://issues.apache.org/jira/browse/HIVE-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039467#comment-14039467
]
Nick Dimiduk commented on HIVE-6584:
Can you regenerate your patch, rooted in the trunk directory instead of above
it? That's the reason this patch fails the buildbot.
Add HiveHBaseTableSnapshotInputFormat
-
Key: HIVE-6584
URL: https://issues.apache.org/jira/browse/HIVE-6584
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Fix For: 0.14.0
Attachments: HIVE-6584.0.patch, HIVE-6584.1.patch, HIVE-6584.2.patch,
HIVE-6584.3.patch, HIVE-6584.4.patch, HIVE-6584.5.patch
HBASE-8369 provided mapreduce support for reading from HBase table snapsopts.
This allows a MR job to consume a stable, read-only view of an HBase table
directly off of HDFS. Bypassing the online region server API provides a nice
performance boost for the full scan. HBASE-10642 is backporting that feature
to 0.94/0.96 and also adding a {{mapred}} implementation. Once that's
available, we should add an input format. A follow-on patch could work out
how to integrate this functionality into the StorageHandler, similar to how
HIVE-6473 integrates the HFileOutputFormat into existing table definitions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7205) Wrong results when union all of grouping followed by group by with correlation optimization
[
https://issues.apache.org/jira/browse/HIVE-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039506#comment-14039506
]
Hive QA commented on HIVE-7205:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12651392/HIVE-7205.1.patch.txt
{color:red}ERROR:{color} -1 due to 43 failed/errored test(s), 5669 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_count
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_noskew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_noskew_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3_map
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3_map_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3_noskew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3_noskew_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_bigdata
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_cube1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_multi_single_reducer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_multi_single_reducer2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppr_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_rollup1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join18
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join18_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_lateral_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_merge
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_count
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_distinct_gby
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_count
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_groupby2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_groupby3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_groupby2
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/533/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/533/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-533/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 43 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12651392
Wrong results when union all of grouping followed by group by with
correlation optimization
---
Key: HIVE-7205
URL: https://issues.apache.org/jira/browse/HIVE-7205
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 0.13.0, 0.13.1
Reporter: dima machlin
Assignee: Navis
Priority: Critical
Attachments: HIVE-7205.1.patch.txt
use case :
table TBL (a string,b string) contains
[jira] [Commented] (HIVE-7266) Optimized HashTable with vectorized map-joins results in String columns extending
[
https://issues.apache.org/jira/browse/HIVE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039509#comment-14039509
]
Eric Hanson commented on HIVE-7266:
---
This looks like it might be related to using setRef() in BytesColumnVector whe
setVal() should be used. That is something to look into.
Optimized HashTable with vectorized map-joins results in String columns
extending
-
Key: HIVE-7266
URL: https://issues.apache.org/jira/browse/HIVE-7266
Project: Hive
Issue Type: Bug
Components: Tez, Vectorization
Affects Versions: 0.14.0
Reporter: Gopal V
Assignee: Jitendra Nath Pandey
Attachments: hive-7266-small-test.tgz
The following query returns different results when both vectorized mapjoin
and the new optimized hashtable are enabled.
{code}
hive set hive.vectorized.execution.enabled=false;
hive select s_suppkey, n_name from supplier, nation where s_nationkey =
n_nationkey limit 25;
...
316869 JAPAN
1636869 RUSSIA
1096869 IRAN
7236869 RUSSIA
2276869 INDIA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIA
2616869 FRANCE
{code}
But when vectorization is enabled, the results are
{code}
316869 JAPAN
1636869 RUSSIA
1096869 IRANIA
7236869 RUSSIA
2276869 INDIAA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIAQUE
2616869 FRANCEAQUE
{code}
it works correctly with vectorization when the new optimized map-join
hashtable is disabled
{code}
hive set hive.vectorized.execution.enabled=true;
hive set hive.mapjoin.optimized.hashtable=false;
hive select s_suppkey, n_name from supplier, nation where s_nationkey =
n_nationkey limit 25;
316869 JAPAN
1636869 RUSSIA
1096869 IRAN
7236869 RUSSIA
2276869 INDIA
8516869 ARGENTINA
2636869 MOZAMBIQUE
3836869 ROMANIA
2616869 FRANCE
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Review Request 22772: HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22772/#review46347 --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java https://reviews.apache.org/r/22772/#comment81679 Same thing here about using getPrimitiveCategory() rather than class/class name. Sorry I missed this on the previous review. - Jason Dere On June 20, 2014, 9:28 p.m., Ashish Singh wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22772/ --- (Updated June 20, 2014, 9:28 p.m.) Review request for hive. Bugs: HIVE-6637 https://issues.apache.org/jira/browse/HIVE-6637 Repository: hive-git Description --- HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input Diffs - ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java ea52537d0b85191f0b633a29aa3f7ddb556c288d ql/src/test/queries/clientpositive/udf_in_file.q 9d9efe8e23d6e73429ee5cd2c8470359ba2b3498 ql/src/test/results/clientpositive/udf_in_file.q.out b63143760d80f3f6a8ba0a23c0d87e8bb86fce66 Diff: https://reviews.apache.org/r/22772/diff/ Testing --- Tested with qtest. Thanks, Ashish Singh
[jira] [Commented] (HIVE-7236) Tez progress monitor should indicate running/failed tasks
[
https://issues.apache.org/jira/browse/HIVE-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039616#comment-14039616
]
Gopal V commented on HIVE-7236:
---
[~leftylev]: Not sure.
The (+n,-m) needs explaining, but I do not see any Tez specific documentation
for hive on the wiki to add this to.
Tez progress monitor should indicate running/failed tasks
-
Key: HIVE-7236
URL: https://issues.apache.org/jira/browse/HIVE-7236
Project: Hive
Issue Type: Bug
Components: Tez
Affects Versions: 0.14.0
Reporter: Gopal V
Assignee: Gopal V
Priority: Minor
Fix For: 0.14.0
Attachments: HIVE-7236.1.patch
Currently, the only logging in TezJobMonitor is for completed tasks.
This makes it hard to locate task stalls and task failures. Failure scenarios
are harder to debug, in particular when analyzing query runs on a cluster
with bad nodes.
Change the job monitor to log running failed tasks as follows.
{code}
Map 1: 0(+157,-1)/1755 Reducer 2: 0/1
Map 1: 0(+168,-1)/1755 Reducer 2: 0/1
Map 1: 0(+189,-1)/1755 Reducer 2: 0/1
Map 1: 0(+189,-1)/1755 Reducer 2: 0/1
{code}
That is 189 tasks running, 1 failure and 0 complete.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7159) For inner joins push a 'is not null predicate' to the join sources for every non nullSafe join condition
[
https://issues.apache.org/jira/browse/HIVE-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039621#comment-14039621
]
Hive QA commented on HIVE-7159:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12651427/HIVE-7159.9.patch
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 5668 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mergejoins_mixed
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subq_where_serialization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_views
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_exists
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/534/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/534/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-534/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12651427
For inner joins push a 'is not null predicate' to the join sources for every
non nullSafe join condition
Key: HIVE-7159
URL: https://issues.apache.org/jira/browse/HIVE-7159
Project: Hive
Issue Type: Bug
Reporter: Harish Butani
Assignee: Harish Butani
Attachments: HIVE-7159.1.patch, HIVE-7159.2.patch, HIVE-7159.3.patch,
HIVE-7159.4.patch, HIVE-7159.5.patch, HIVE-7159.6.patch, HIVE-7159.7.patch,
HIVE-7159.8.patch, HIVE-7159.9.patch
A join B on A.x = B.y
can be transformed to
(A where x is not null) join (B where y is not null) on A.x = B.y
Apart from avoiding shuffling null keyed rows it also avoids issues with
reduce-side skew when there are a lot of null values in the data.
Thanks to [~gopalv] for the analysis and coming up with the solution.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Review Request 22772: HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22772/ --- (Updated June 21, 2014, 1:26 a.m.) Review request for hive. Changes --- Missed out a change in last rb post. Bugs: HIVE-6637 https://issues.apache.org/jira/browse/HIVE-6637 Repository: hive-git Description (updated) --- HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input Add unit tests for Udf in_file Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java ea52537d0b85191f0b633a29aa3f7ddb556c288d ql/src/test/queries/clientpositive/udf_in_file.q 9d9efe8e23d6e73429ee5cd2c8470359ba2b3498 ql/src/test/results/clientpositive/udf_in_file.q.out b63143760d80f3f6a8ba0a23c0d87e8bb86fce66 Diff: https://reviews.apache.org/r/22772/diff/ Testing --- Tested with qtest. Thanks, Ashish Singh
Re: Review Request 22772: HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input
On June 20, 2014, 11:52 p.m., Jason Dere wrote: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java, line 84 https://reviews.apache.org/r/22772/diff/4/?file=614380#file614380line84 Same thing here about using getPrimitiveCategory() rather than class/class name. Sorry I missed this on the previous review. That was my bad. Thanks for catching it. - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22772/#review46347 --- On June 21, 2014, 1:26 a.m., Ashish Singh wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22772/ --- (Updated June 21, 2014, 1:26 a.m.) Review request for hive. Bugs: HIVE-6637 https://issues.apache.org/jira/browse/HIVE-6637 Repository: hive-git Description --- HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input Add unit tests for Udf in_file Diffs - ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java ea52537d0b85191f0b633a29aa3f7ddb556c288d ql/src/test/queries/clientpositive/udf_in_file.q 9d9efe8e23d6e73429ee5cd2c8470359ba2b3498 ql/src/test/results/clientpositive/udf_in_file.q.out b63143760d80f3f6a8ba0a23c0d87e8bb86fce66 Diff: https://reviews.apache.org/r/22772/diff/ Testing --- Tested with qtest. Thanks, Ashish Singh
Re: Review Request 22772: HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22772/ --- (Updated June 21, 2014, 1:27 a.m.) Review request for hive. Changes --- Remove redundant text from desc. Bugs: HIVE-6637 https://issues.apache.org/jira/browse/HIVE-6637 Repository: hive-git Description (updated) --- HIVE-6637: UDF in_file() doesn't take CHAR or VARCHAR as input Diffs - ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java ea52537d0b85191f0b633a29aa3f7ddb556c288d ql/src/test/queries/clientpositive/udf_in_file.q 9d9efe8e23d6e73429ee5cd2c8470359ba2b3498 ql/src/test/results/clientpositive/udf_in_file.q.out b63143760d80f3f6a8ba0a23c0d87e8bb86fce66 Diff: https://reviews.apache.org/r/22772/diff/ Testing --- Tested with qtest. Thanks, Ashish Singh
[jira] [Updated] (HIVE-6637) UDF in_file() doesn't take CHAR or VARCHAR as input
[
https://issues.apache.org/jira/browse/HIVE-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Kumar Singh updated HIVE-6637:
-
Attachment: (was: HIVE-6637.2.patch)
UDF in_file() doesn't take CHAR or VARCHAR as input
---
Key: HIVE-6637
URL: https://issues.apache.org/jira/browse/HIVE-6637
Project: Hive
Issue Type: Bug
Components: Types, UDF
Affects Versions: 0.14.0
Reporter: Xuefu Zhang
Assignee: Ashish Kumar Singh
Attachments: HIVE-6637.1.patch, HIVE-6637.2.patch
{code}
hive desc alter_varchar_1;
key string None
value varchar(3) None
key2 int None
value2varchar(10) None
hive select in_file(value, value2) from alter_varchar_1;
FAILED: SemanticException [Error 10016]: Line 1:15 Argument type mismatch
'value': The 1st argument of function IN_FILE must be a string but
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableHiveVarcharObjectInspector@10f1f34a
was given.
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7268) On Windows Hive jobs in Webhcat always run on default MR mode
[ https://issues.apache.org/jira/browse/HIVE-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039664#comment-14039664 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-7268: - non-binding +1. [~daijy] Can you please look at this. Thanks Hari On Windows Hive jobs in Webhcat always run on default MR mode - Key: HIVE-7268 URL: https://issues.apache.org/jira/browse/HIVE-7268 Project: Hive Issue Type: Bug Components: WebHCat Affects Versions: 0.13.0 Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal Fix For: 0.14.0 Attachments: HIVE-7268.1.patch On Windows fix from HIVE-7065 doesn't work as the templeton.cmd script does not include the Hive configuration directory in the classpath. So when hive.execution.engine property is set to tez in HIVE_CONF_DIR/hive-site.xml, webhcat doesn't see it and defaults it to mr. This prevents Hive jobs running from WebHCat to use the tez execution engine. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7144) GC pressure during ORC StringDictionary writes
[ https://issues.apache.org/jira/browse/HIVE-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-7144: -- Attachment: HIVE-7144.1.patch GC pressure during ORC StringDictionary writes --- Key: HIVE-7144 URL: https://issues.apache.org/jira/browse/HIVE-7144 Project: Hive Issue Type: Bug Components: File Formats Affects Versions: 0.14.0 Environment: ORC Table ~ 12 string columns Reporter: Gopal V Assignee: Gopal V Labels: ORC, Performance Attachments: HIVE-7144.1.patch, orc-string-write.png When ORC string dictionary writes data out, it suffers from bad GC performance due to a few allocations in-loop. !orc-string-write.png! The conversions are as follows StringTreeWriter::getStringValue() causes 2 conversions LazyString - Text (LazyString::getWritableObject) Text - String (LazyStringObjectInspector::getPrimitiveJavaObject) Then StringRedBlackTree::add() does one conversion String - Text This causes some GC pressure with un-necessary String and byte[] array allocations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7225) Unclosed Statement's in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HIVE-7225:
-
Status: Patch Available (was: Open)
Unclosed Statement's in TxnHandler
--
Key: HIVE-7225
URL: https://issues.apache.org/jira/browse/HIVE-7225
Project: Hive
Issue Type: Bug
Reporter: Ted Yu
Assignee: steve, Oh
Attachments: HIVE-7225.1.patch, hive-7225.3.patch
There're several methods in TxnHandler where Statement (local to the method)
is not closed upon return.
Here're a few examples:
In compact():
{code}
stmt.executeUpdate(s);
LOG.debug(Going to commit);
dbConn.commit();
{code}
In showCompact():
{code}
Statement stmt = dbConn.createStatement();
String s = select cq_database, cq_table, cq_partition, cq_state,
cq_type, cq_worker_id, +
cq_start, cq_run_as from COMPACTION_QUEUE;
LOG.debug(Going to execute query + s + );
ResultSet rs = stmt.executeQuery(s);
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7225) Unclosed Statement's in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HIVE-7225:
-
Attachment: hive-7225.3.patch
Patch v3 addresses Alan's comments above.
Unclosed Statement's in TxnHandler
--
Key: HIVE-7225
URL: https://issues.apache.org/jira/browse/HIVE-7225
Project: Hive
Issue Type: Bug
Reporter: Ted Yu
Assignee: steve, Oh
Attachments: HIVE-7225.1.patch, hive-7225.3.patch
There're several methods in TxnHandler where Statement (local to the method)
is not closed upon return.
Here're a few examples:
In compact():
{code}
stmt.executeUpdate(s);
LOG.debug(Going to commit);
dbConn.commit();
{code}
In showCompact():
{code}
Statement stmt = dbConn.createStatement();
String s = select cq_database, cq_table, cq_partition, cq_state,
cq_type, cq_worker_id, +
cq_start, cq_run_as from COMPACTION_QUEUE;
LOG.debug(Going to execute query + s + );
ResultSet rs = stmt.executeQuery(s);
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6637) UDF in_file() doesn't take CHAR or VARCHAR as input
[
https://issues.apache.org/jira/browse/HIVE-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039669#comment-14039669
]
Hive QA commented on HIVE-6637:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12651738/HIVE-6637.2.patch
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 5668 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_in_file
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_storage_queries
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/535/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/535/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-535/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12651738
UDF in_file() doesn't take CHAR or VARCHAR as input
---
Key: HIVE-6637
URL: https://issues.apache.org/jira/browse/HIVE-6637
Project: Hive
Issue Type: Bug
Components: Types, UDF
Affects Versions: 0.14.0
Reporter: Xuefu Zhang
Assignee: Ashish Kumar Singh
Attachments: HIVE-6637.1.patch, HIVE-6637.2.patch
{code}
hive desc alter_varchar_1;
key string None
value varchar(3) None
key2 int None
value2varchar(10) None
hive select in_file(value, value2) from alter_varchar_1;
FAILED: SemanticException [Error 10016]: Line 1:15 Argument type mismatch
'value': The 1st argument of function IN_FILE must be a string but
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableHiveVarcharObjectInspector@10f1f34a
was given.
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6637) UDF in_file() doesn't take CHAR or VARCHAR as input
[
https://issues.apache.org/jira/browse/HIVE-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039698#comment-14039698
]
Ashish Kumar Singh commented on HIVE-6637:
--
Test errors do not look related.
UDF in_file() doesn't take CHAR or VARCHAR as input
---
Key: HIVE-6637
URL: https://issues.apache.org/jira/browse/HIVE-6637
Project: Hive
Issue Type: Bug
Components: Types, UDF
Affects Versions: 0.14.0
Reporter: Xuefu Zhang
Assignee: Ashish Kumar Singh
Attachments: HIVE-6637.1.patch, HIVE-6637.2.patch
{code}
hive desc alter_varchar_1;
key string None
value varchar(3) None
key2 int None
value2varchar(10) None
hive select in_file(value, value2) from alter_varchar_1;
FAILED: SemanticException [Error 10016]: Line 1:15 Argument type mismatch
'value': The 1st argument of function IN_FILE must be a string but
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableHiveVarcharObjectInspector@10f1f34a
was given.
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
