[jira] [Updated] (HIVE-4246) Implement predicate pushdown for ORC
[ https://issues.apache.org/jira/browse/HIVE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4246: -- Attachment: HIVE-4246.D11415.1.patch omalley requested code review of HIVE-4246 [jira] Implement predicate pushdown for ORC. Reviewers: JIRA Record Reader changes By using the push down predicates from the table scan operator, ORC can skip over 10,000 rows at a time that won't satisfy the predicate. This will help a lot, especially if the file is sorted by the column that is used in the predicate. TEST PLAN unit tests and cluster test REVISION DETAIL https://reviews.facebook.net/D11415 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/BitFieldReader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthByteReader.java ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgument.java ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitFieldReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInStream.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRecordReaderImpl.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthByteReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthIntegerReader.java ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/27075/ To: JIRA, omalley Implement predicate pushdown for ORC Key: HIVE-4246 URL: https://issues.apache.org/jira/browse/HIVE-4246 Project: Hive Issue Type: New Feature Components: File Formats Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-4246.D11415.1.patch By using the push down predicates from the table scan operator, ORC can skip over 10,000 rows at a time that won't satisfy the predicate. This will help a lot, especially if the file is sorted by the column that is used in the predicate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4246) Implement predicate pushdown for ORC
[ https://issues.apache.org/jira/browse/HIVE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-4246: Status: Patch Available (was: Open) This patch: * Adds the column names for the required columns * Uses the SearchArgument interface added in HIVE-4579 * Updates the ORC reader to skip over sets of rows that aren't useful. * Extends InStream to read from multiple sets of byte buffers * Updates the ORC reader to skip over ignored rows after each next Implement predicate pushdown for ORC Key: HIVE-4246 URL: https://issues.apache.org/jira/browse/HIVE-4246 Project: Hive Issue Type: New Feature Components: File Formats Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-4246.D11415.1.patch By using the push down predicates from the table scan operator, ORC can skip over 10,000 rows at a time that won't satisfy the predicate. This will help a lot, especially if the file is sorted by the column that is used in the predicate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4745) java.lang.RuntimeException: Hive Runtime Error while closing operators: java.lang.ClassCastException: org.apache.hadoop.io.NullWritable cannot be cast to org.apache.hadoop
[
https://issues.apache.org/jira/browse/HIVE-4745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HIVE-4745:
---
Attachment: HIVE-4745.3.patch
I made a minor modification and changed assertSame back to assertEquals for
null checks
java.lang.RuntimeException: Hive Runtime Error while closing operators:
java.lang.ClassCastException: org.apache.hadoop.io.NullWritable cannot be
cast to org.apache.hadoop.hive.serde2.io.DoubleWritable
-
Key: HIVE-4745
URL: https://issues.apache.org/jira/browse/HIVE-4745
Project: Hive
Issue Type: Sub-task
Affects Versions: vectorization-branch
Reporter: Tony Murphy
Assignee: Jitendra Nath Pandey
Fix For: vectorization-branch
Attachments: HIVE-4745.2.patch, HIVE-4745.3.patch
{noformat}
SELECT SUM(L_QUANTITY),
(SUM(L_QUANTITY) + -1.3000E+000),
(-2.2002E+000 % (SUM(L_QUANTITY) +
-1.3000E+000)),
MIN(L_EXTENDEDPRICE)
FROM lineitem_orc
WHERE ((L_EXTENDEDPRICE = L_LINENUMBER)
OR (L_TAX L_EXTENDEDPRICE));
{noformat}
executed over tpch line item with scale factor 1gb
{noformat}
13/06/15 11:19:17 WARN conf.HiveConf: DEPRECATED: Configuration property
hive.metastore.local no longer has any effect. Make sure to provide a valid
value for hive.metastore.uris if you are connecting to a remote metastore.
Logging initialized using configuration in
file:/C:/Hadoop/hive-0.9.0/conf/hive-log4j.properties
Hive history
file=c:\hadoop\hive-0.9.0\logs\history/hive_job_log_jenkinsuser_5292@SLAVE23-WIN_201306151119_1652846565.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapred.reduce.tasks=number
Starting Job = job_201306142329_0098, Tracking URL =
http://localhost:50030/jobdetails.jsp?jobid=job_201306142329_0098
Kill Command = c:\Hadoop\hadoop-1.1.0-SNAPSHOT\bin\hadoop.cmd job -kill
job_201306142329_0098
Hadoop job information for Stage-1: number of mappers: 1; number of reducers:
1
2013-06-15 11:19:47,490 Stage-1 map = 0%, reduce = 0%
2013-06-15 11:20:29,801 Stage-1 map = 76%, reduce = 0%
2013-06-15 11:20:32,849 Stage-1 map = 0%, reduce = 0%
2013-06-15 11:20:35,880 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201306142329_0098 with errors
Error during job, obtaining debugging information...
Job Tracking URL:
http://localhost:50030/jobdetails.jsp?jobid=job_201306142329_0098
Examining task ID: task_201306142329_0098_m_02 (and more) from job
job_201306142329_0098
Task with the most failures(4):
-
Task ID:
task_201306142329_0098_m_00
URL:
http://localhost:50030/taskdetails.jsp?jobid=job_201306142329_0098tipid=task_201306142329_0098_m_00
-
Diagnostic Messages for this Task:
java.lang.RuntimeException: Hive Runtime Error while closing operators
at
org.apache.hadoop.hive.ql.exec.vector.VectorExecMapper.close(VectorExecMapper.java:229)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.mapred.Child.main(Child.java:265)
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.NullWritable
cannot be cast to org.apache.hadoop.hive.serde2.io.DoubleWritable
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableDoubleObjectInspector.get(WritableDoubleObjectInspector.java:35)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:340)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serializeStruct(LazyBinarySerDe.java:257)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:204)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:245)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at
[jira] [Commented] (HIVE-4172) JDBC2 does not support VOID type
[
https://issues.apache.org/jira/browse/HIVE-4172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689075#comment-13689075
]
Phabricator commented on HIVE-4172:
---
navis has commented on the revision HIVE-4172 [jira] JDBC2 does not support
VOID type.
INLINE COMMENTS
jdbc/src/java/org/apache/hive/jdbc/HiveResultSetMetaData.java:119 It's
constant value in serdeConstants, which cannot be changed.
serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyVoid.java:23 There is
LazyBinaryVoid, already. Shouldn't it be conform?
service/src/java/org/apache/hive/service/cli/Type.java:30 Again, it's
constant in serdeConstant.
REVISION DETAIL
https://reviews.facebook.net/D9555
BRANCH
HIVE-4172
ARCANIST PROJECT
hive
To: JIRA, ashutoshc, navis
Cc: prasadm
JDBC2 does not support VOID type
Key: HIVE-4172
URL: https://issues.apache.org/jira/browse/HIVE-4172
Project: Hive
Issue Type: Improvement
Components: HiveServer2, JDBC
Affects Versions: 0.11.0
Reporter: Navis
Assignee: Navis
Priority: Minor
Labels: HiveServer2
Attachments: HIVE-4172.D9555.1.patch, HIVE-4172.D9555.2.patch,
HIVE-4172.D9555.3.patch, HIVE-4172.D9555.4.patch, HIVE-4172.D9555.5.patch
In beeline, select key, null from src fails with exception,
{noformat}
org.apache.hive.service.cli.HiveSQLException: Error running query:
java.lang.NullPointerException
at
org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:112)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:166)
at
org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:148)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:183)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1133)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1118)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:39)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Server2 client authentication type CUSTOM problem (0.11.0)
Hi, I work on PAM authentication for Hive and I need to use org.apache.hive.service.auth.PasswdAuthenticationProvider interface. I configured my provider class in hive-site.xml correctly, but Hive threw exception (see below). I think, there is problem in CustomAuthenticationProviderImpl class, because first argument of getClass method is not configuration key, but enum key. I’m sending patch, that helps me to solve this problem. Regards, Zdenek java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hive.service.auth.PasswdAuthenticationProvider.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:128) at org.apache.hive.service.auth.CustomAuthenticationProviderImpl.init(CustomAuthenticationProviderImpl.java:20) at org.apache.hive.service.auth.AuthenticationProviderFactory.getAuthenticationProvider(AuthenticationProviderFactory.java:57) at org.apache.hive.service.auth.PlainSaslHelper$PlainServerCallbackHandler.handle(PlainSaslHelper.java:61) at org.apache.hive.service.auth.PlainSaslServer.evaluateResponse(PlainSaslServer.java:127) at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:509) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:264) at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.NoSuchMethodException: org.apache.hive.service.auth.PasswdAuthenticationProvider.init() at java.lang.Class.getConstructor0(Class.java:2706) at java.lang.Class.getDeclaredConstructor(Class.java:1985) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:122) ... 12 more
[jira] [Commented] (HIVE-4586) [HCatalog] WebHCat should return 404 error for undefined resource
[ https://issues.apache.org/jira/browse/HIVE-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689270#comment-13689270 ] Scott Preece commented on HIVE-4586: +1 [HCatalog] WebHCat should return 404 error for undefined resource - Key: HIVE-4586 URL: https://issues.apache.org/jira/browse/HIVE-4586 Project: Hive Issue Type: Bug Reporter: Daniel Dai Attachments: HIVE-4586-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4756) Upgrade Hadoop 0.23 profile to 2.0.5-alpha
[
https://issues.apache.org/jira/browse/HIVE-4756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689290#comment-13689290
]
Brock Noland commented on HIVE-4756:
I ran a full run again, twice, and input42 didn't fail either time. Here is the
list of failures I am seeing on these two runs:
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_npath
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_multiskew_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_1to1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_external_table_ppd
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_external_table_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_map_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_storage_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_joins
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_key_range
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_pushdown
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats3
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats_empty_partition
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_ppd_key_ranges
org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_cascade_dbdrop
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_cascade_dbdrop_hadoop20
{noformat}
Upgrade Hadoop 0.23 profile to 2.0.5-alpha
--
Key: HIVE-4756
URL: https://issues.apache.org/jira/browse/HIVE-4756
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Assignee: Brock Noland
Fix For: 0.12.0
Attachments: HIVE-4756.patch
The minimr tests fail at present with the 0.23 profile. In my tests upgrading
to 2.0.5-alpha fixes this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4756) Upgrade Hadoop 0.23 profile to 2.0.5-alpha
[ https://issues.apache.org/jira/browse/HIVE-4756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4756: --- Status: Patch Available (was: Open) Upgrade Hadoop 0.23 profile to 2.0.5-alpha -- Key: HIVE-4756 URL: https://issues.apache.org/jira/browse/HIVE-4756 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Brock Noland Fix For: 0.12.0 Attachments: HIVE-4756.patch The minimr tests fail at present with the 0.23 profile. In my tests upgrading to 2.0.5-alpha fixes this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4766) HS2 login timeout
Chaoyu Tang created HIVE-4766: - Summary: HS2 login timeout Key: HIVE-4766 URL: https://issues.apache.org/jira/browse/HIVE-4766 Project: Hive Issue Type: Bug Reporter: Chaoyu Tang -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4767) ObjectStore. getPMF has concurrency problems
Brock Noland created HIVE-4767: -- Summary: ObjectStore. getPMF has concurrency problems Key: HIVE-4767 URL: https://issues.apache.org/jira/browse/HIVE-4767 Project: Hive Issue Type: Sub-task Affects Versions: 0.11.0 Reporter: Brock Noland Assignee: Brock Noland Fix For: 0.12.0 ObjectStore.getPMF is a static factory method but isn't thread safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4766) Support HS2 client login timeout when the thrift thread max# is reached
[ https://issues.apache.org/jira/browse/HIVE-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-4766: -- Component/s: HiveServer2 Description: HiveServer2 client (beeline) hangs in login if the thrift max thread# has been reached. It is because the server crashes due to a defect in currently used thrift 0.9.0. When hive is upgraded to use a new version of Thrift (say thrift 1.0), HS2 should support client login timeout instead of current hanging. Affects Version/s: 0.10.0 Issue Type: Improvement (was: Bug) Summary: Support HS2 client login timeout when the thrift thread max# is reached (was: HS2 login timeout ) Support HS2 client login timeout when the thrift thread max# is reached --- Key: HIVE-4766 URL: https://issues.apache.org/jira/browse/HIVE-4766 Project: Hive Issue Type: Improvement Components: HiveServer2 Affects Versions: 0.10.0 Reporter: Chaoyu Tang HiveServer2 client (beeline) hangs in login if the thrift max thread# has been reached. It is because the server crashes due to a defect in currently used thrift 0.9.0. When hive is upgraded to use a new version of Thrift (say thrift 1.0), HS2 should support client login timeout instead of current hanging. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Storing data into hive table having columns from multiple files
Do you mean the merge operation? Sent from my iPad On 20-Jun-2013, at 2:51 PM, varaprasad.bh...@polarisft.com wrote: Hi All, I have a query on a scenario which is below: There are 2 files file1, file2. And a hive table test which is created with 15 columns with , delimiter. file1 contains 25 columns and file2 contains 15 columns having seperator as ,: But, out of 25 columns, 10 columns of data from file1 are inserted to hive table test and out of 15 columns, 5 columns of data from file2 are inserted to the same hive table test (lets forget about the remaining 15 columns from file1 and 10 columns from file2) First, file1 is processed and then file2 is processed. When file1 is processed, 10 columns gets inserted into test table for a row. When file2 is processed, 5 columns have to be inserted into test table of the same row. Could you please help me on how to insert the 5 columns of file2 to the same row in test table? (which has 10 columns of data inserted from file1 for that row) Thanks Regards, Varaprasada Reddy This e-Mail may contain proprietary and confidential information and is sent for the intended recipient(s) only. If by an addressing or transmission error this mail has been misdirected to you, you are requested to delete this mail immediately. You are also hereby notified that any use, any form of reproduction, dissemination, copying, disclosure, modification, distribution and/or publication of this e-mail message, contents or its attachment other than by its intended recipient/s is strictly prohibited. Visit us at http://www.polarisFT.com
[jira] [Commented] (HIVE-4609) Allow hive tests to specify an alternative to /tmp
[ https://issues.apache.org/jira/browse/HIVE-4609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689319#comment-13689319 ] Hudson commented on HIVE-4609: -- Integrated in Hive-trunk-h0.21 #2158 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2158/]) HIVE-4609 : Allow hive tests to specify an alternative to /tmp (Brock Noland via Ashutosh Chauhan) (Revision 1494860) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1494860 Files : * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TestPlan.java Allow hive tests to specify an alternative to /tmp -- Key: HIVE-4609 URL: https://issues.apache.org/jira/browse/HIVE-4609 Project: Hive Issue Type: Improvement Components: Tests Affects Versions: 0.11.0 Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4609.patch, HIVE-4609.patch It'd be nice to be able to force hive not to use /tmp/. This is particularly useful when using the ptest framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Storing data into hive table having columns from multiple files
Hi All, I have a query on a scenario which is below: There are 2 files file1, file2. And a hive table test which is created with 15 columns with , delimiter. file1 contains 25 columns and file2 contains 15 columns having seperator as ,: But, out of 25 columns, 10 columns of data from file1 are inserted to hive table test and out of 15 columns, 5 columns of data from file2 are inserted to the same hive table test (lets forget about the remaining 15 columns from file1 and 10 columns from file2) First, file1 is processed and then file2 is processed. When file1 is processed, 10 columns gets inserted into test table for a row. When file2 is processed, 5 columns have to be inserted into test table of the same row. Could you please help me on how to insert the 5 columns of file2 to the same row in test table? (which has 10 columns of data inserted from file1 for that row) Thanks Regards, Varaprasada Reddy This e-Mail may contain proprietary and confidential information and is sent for the intended recipient(s) only. If by an addressing or transmission error this mail has been misdirected to you, you are requested to delete this mail immediately. You are also hereby notified that any use, any form of reproduction, dissemination, copying, disclosure, modification, distribution and/or publication of this e-mail message, contents or its attachment other than by its intended recipient/s is strictly prohibited. Visit us at http://www.polarisFT.com
Hive-trunk-h0.21 - Build # 2158 - Still Failing
Changes for Build #2124
[gates] HIVE-4543 Broken link in HCat doc (Reader and Writer Interfaces) (Lefty
Leverenz via gates)
Changes for Build #2125
[daijy] PIG-3337: Fix remaining Window e2e tests
Changes for Build #2126
[hashutosh] HIVE-4615 : Invalid column names allowed when created dynamically
by a SerDe (Gabriel Reid via Ashutosh Chauhan)
[hashutosh] HIVE-3846 : alter view rename NPEs with authorization on. (Teddy
Choi via Ashutosh Chauhan)
[hashutosh] HIVE-4403 : Running Hive queries on Yarn (MR2) gives warnings
related to overriding final parameters (Chu Tong via Ashutosh Chauhan)
[hashutosh] HIVE-4610 : HCatalog checkstyle violation after HIVE4578 (Brock
Noland via Ashutosh Chauhan)
[hashutosh] HIVE-4636 : Failing on TestSemanticAnalysis.testAddReplaceCols in
trunk (Navis via Ashutosh Chauhan)
[hashutosh] HIVE-4626 : join_vc.q is not deterministic (Navis via Ashutosh
Chauhan)
[hashutosh] HIVE-4562 : HIVE3393 brought in Jackson library,and these four jars
should be packed into hive-exec.jar (caofangkun via Ashutosh Chauhan)
[hashutosh] HIVE-4489 : beeline always return the same error message twice
(Chaoyu Tang via Ashutosh Chauhan)
[hashutosh] HIVE-4510 : HS2 doesn't nest exceptions properly (fun debug times)
(Thejas Nair via Ashutosh Chauhan)
[hashutosh] HIVE-4535 : hive build fails with hadoop 0.20 (Thejas Nair via
Ashutosh Chauhan)
Changes for Build #2127
[hashutosh] HIVE-4585 : Remove unused MR Temp file localization from Tasks
(Gunther Hagleitner via Ashutosh Chauhan)
[hashutosh] HIVE-4418 : TestNegativeCliDriver failure message if cmd succeeds
is misleading (Thejas Nair via Ashutosh Chauhan)
[navis] HIVE-4620 MR temp directory conflicts in case of parallel execution
mode (Prasad Mujumdar via Navis)
Changes for Build #2128
[hashutosh] HIVE-4646 : skewjoin.q is failing in hadoop2 (Navis via Ashutosh
Chauhan)
[hashutosh] HIVE-4377 : Add more comment to https://reviews.facebook.net/D1209
(HIVE2340) : (Navis via Ashutosh Chauhan)
[hashutosh] HIVE-4546 : Hive CLI leaves behind the per session resource
directory on non-interactive invocation (Prasad Mujumdar via Ashutosh Chauhan)
[gates] HIVE-2670 A cluster test utility for Hive (gates and Johnny Zhang via
gates)
Changes for Build #2129
[hashutosh] HIVE-2304 : Support PreparedStatement.setObject (Ido Hadanny via
Ashutosh Chauhan)
[hashutosh] HIVE-4526 : auto_sortmerge_join_9.q throws NPE but test is
succeeded (Navis via Ashutosh Chauhan)
[hashutosh] HIVE-4516 : Fix concurrency bug in
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java (Jon
Hartlaub and Navis via Ashutosh Chauhan)
[hashutosh] HIVE-4566 : NullPointerException if typeinfo and nativesql commands
are executed at beeline before a DB connection is established (Xuefu Zhang via
Ashutosh Chauhan)
Changes for Build #2130
Changes for Build #2131
[hashutosh] HIVE-4422 : Test output need to be updated for Windows only unit
test in TestCliDriver (Shuaishuai Nie via Ashutosh Chauhan)
[hashutosh] HIVE-4657 : HCatalog checkstyle violation after HIVE-2670
(Shreepadma Venugopalan via Ashutosh Chauhan)
[hashutosh] HIVE-4348 : Unit test compile fail at hbase-handler project on
Windows becuase of illegal escape character (Shuaishuai Nie via Ashutosh
Chauhan)
[hashutosh] HIVE-4554 : Failed to create a table from existing file if file
path has spaces (Xuefu Zhang via Ashutosh Chauhan)
[hashutosh] HIVE-4459 : Script hcat is overriding HIVE_CONF_DIR variable (Jarek
Jarcec Cecho via Ashutosh Chauhan)
Changes for Build #2132
[hashutosh] HIVE-4435 : Column stats: Distinct value estimator should use hash
functions that are pairwise independent (Shreepadma Venugopalan via Ashutosh
Chauhan)
Changes for Build #2133
Changes for Build #2134
Changes for Build #2135
Changes for Build #2136
Changes for Build #2137
[navis] HIVE-4549 : JDBC compliance change TABLE_SCHEMA to TABLE_SCHEM (Prasad
Mujumdar via Navis)
Changes for Build #2138
[navis] HIVE-4691 : orc_createas1.q has minor inconsistency (Brock Noland via
Navis)
Changes for Build #2139
[hashutosh] HIVE-4712 : Fix TestCliDriver.truncate_* on 0.23 (Brock Noland via
Ashutosh Chauhan)
Changes for Build #2140
[navis] HIVE-4708 : Fix TestCliDriver.combine2.q on 0.23 (Brock Noland via
Navis)
Changes for Build #2141
Changes for Build #2142
[hashutosh] HIVE-4717 : Fix non-deterministic TestCliDriver on 0.23 (Brock
Noland via Ashutosh Chauhan)
Changes for Build #2143
Changes for Build #2144
[hashutosh] HIVE-4713 : Fix TestCliDriver.skewjoin_union_remove_{1,2}.q on 0.23
(Brock Noland via Ashutosh Chauhan)
[hashutosh] HIVE-4715 : Fix
TestCliDriver.{recursive_dir.q,sample_islocalmode_hook.q,input12.q,input39.q,auto_join14.q}
on 0.23 (Brock Noland via Ashutosh Chauhan)
Changes for Build #2145
Changes for Build #2146
[ecapriolo] Submitted by: Brock Noland Jonathon Chang
Reviewed by: Edward Capriolo
Approved by: Edward Capriolo
Changes for Build #2147
[jira] [Updated] (HIVE-4761) ZooKeeperHiveLockManage.unlockPrimitive has race condition with threads
[ https://issues.apache.org/jira/browse/HIVE-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4761: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Brock! ZooKeeperHiveLockManage.unlockPrimitive has race condition with threads --- Key: HIVE-4761 URL: https://issues.apache.org/jira/browse/HIVE-4761 Project: Hive Issue Type: Sub-task Affects Versions: 0.11.0 Reporter: Brock Noland Assignee: Brock Noland Priority: Critical Fix For: 0.12.0 Attachments: HIVE-4761.patch, HIVE-4761.patch In unlockPrimitive, we check to see if children exist and if not delete the parent node. If two threads do this at the same time it's possible for two threads to call Zookeeper.delete() on the same node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Subscribe for dev hive group
There are instructions for subscribing to the mailing lists at http://hive.apache.org/mailing_lists.html. You need to send mail to dev-subscr...@hive.apache.org. scott On 6/20/13 4:08 AM, varaprasad.bh...@polarisft.com varaprasad.bh...@polarisft.com wrote: Hi, I would like to subscribe for dev hive group. Please add me to this group. Thanks in advance. Thanks Regards, Varaprasada Reddy This e-Mail may contain proprietary and confidential information and is sent for the intended recipient(s) only. If by an addressing or transmission error this mail has been misdirected to you, you are requested to delete this mail immediately. You are also hereby notified that any use, any form of reproduction, dissemination, copying, disclosure, modification, distribution and/or publication of this e-mail message, contents or its attachment other than by its intended recipient/s is strictly prohibited. Visit us at http://www.polarisFT.com
Subscribe for dev hive group
Hi, I would like to subscribe for dev hive group. Please add me to this group. Thanks in advance. Thanks Regards, Varaprasada Reddy This e-Mail may contain proprietary and confidential information and is sent for the intended recipient(s) only. If by an addressing or transmission error this mail has been misdirected to you, you are requested to delete this mail immediately. You are also hereby notified that any use, any form of reproduction, dissemination, copying, disclosure, modification, distribution and/or publication of this e-mail message, contents or its attachment other than by its intended recipient/s is strictly prohibited. Visit us at http://www.polarisFT.com
[jira] [Commented] (HIVE-4761) ZooKeeperHiveLockManage.unlockPrimitive has race condition with threads
[ https://issues.apache.org/jira/browse/HIVE-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689343#comment-13689343 ] Brock Noland commented on HIVE-4761: Thanks! ZooKeeperHiveLockManage.unlockPrimitive has race condition with threads --- Key: HIVE-4761 URL: https://issues.apache.org/jira/browse/HIVE-4761 Project: Hive Issue Type: Sub-task Affects Versions: 0.11.0 Reporter: Brock Noland Assignee: Brock Noland Priority: Critical Fix For: 0.12.0 Attachments: HIVE-4761.patch, HIVE-4761.patch In unlockPrimitive, we check to see if children exist and if not delete the parent node. If two threads do this at the same time it's possible for two threads to call Zookeeper.delete() on the same node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4733) HiveLockObjectData is not compared properly
[
https://issues.apache.org/jira/browse/HIVE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689342#comment-13689342
]
Phabricator commented on HIVE-4733:
---
ashutoshc has accepted the revision HIVE-4733 [jira] HiveLockObjectData is not
compared properly.
+1
REVISION DETAIL
https://reviews.facebook.net/D11277
BRANCH
HIVE-4733
ARCANIST PROJECT
hive
To: JIRA, ashutoshc, navis
HiveLockObjectData is not compared properly
---
Key: HIVE-4733
URL: https://issues.apache.org/jira/browse/HIVE-4733
Project: Hive
Issue Type: Bug
Components: Locking
Reporter: Navis
Assignee: Navis
Priority: Trivial
Fix For: 0.12.0
Attachments: HIVE-4733.D11277.1.patch, HIVE-4733.D11277.2.patch
{noformat}
ret = ret (clientIp == null) ? target.getClientIp() == null :
clientIp.equals(target.getClientIp());
{noformat}
seemed intended to be
{noformat}
ret = ret (clientIp == null ? target.getClientIp() == null :
clientIp.equals(target.getClientIp()));
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4733) HiveLockObjectData is not compared properly
[
https://issues.apache.org/jira/browse/HIVE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689354#comment-13689354
]
Phabricator commented on HIVE-4733:
---
brock has commented on the revision HIVE-4733 [jira] HiveLockObjectData is not
compared properly.
Hi Navis,
Great catch on the ordering issue! I do have one minor question about the
change to ZooKeeperHiveLockManager. I am not expert there so I might be
totally off base. :)
Brock
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java:448
Ignoring this exception I am less sure about. Too be sure I am no expert on
this class! :) But I read this method as:
delete mylock
if not parent.hasChildren():
delete parent
Is that your reading? If so, I wonder if we should let this exception be
thrown because it could be an indication of correctness issues?
REVISION DETAIL
https://reviews.facebook.net/D11277
BRANCH
HIVE-4733
ARCANIST PROJECT
hive
To: JIRA, ashutoshc, navis
Cc: brock
HiveLockObjectData is not compared properly
---
Key: HIVE-4733
URL: https://issues.apache.org/jira/browse/HIVE-4733
Project: Hive
Issue Type: Bug
Components: Locking
Reporter: Navis
Assignee: Navis
Priority: Trivial
Fix For: 0.12.0
Attachments: HIVE-4733.D11277.1.patch, HIVE-4733.D11277.2.patch
{noformat}
ret = ret (clientIp == null) ? target.getClientIp() == null :
clientIp.equals(target.getClientIp());
{noformat}
seemed intended to be
{noformat}
ret = ret (clientIp == null ? target.getClientIp() == null :
clientIp.equals(target.getClientIp()));
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4172) JDBC2 does not support VOID type
[
https://issues.apache.org/jira/browse/HIVE-4172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689357#comment-13689357
]
Phabricator commented on HIVE-4172:
---
ashutoshc has accepted the revision HIVE-4172 [jira] JDBC2 does not support
VOID type.
+1
REVISION DETAIL
https://reviews.facebook.net/D9555
BRANCH
HIVE-4172
ARCANIST PROJECT
hive
To: JIRA, ashutoshc, navis
Cc: prasadm
JDBC2 does not support VOID type
Key: HIVE-4172
URL: https://issues.apache.org/jira/browse/HIVE-4172
Project: Hive
Issue Type: Improvement
Components: HiveServer2, JDBC
Affects Versions: 0.11.0
Reporter: Navis
Assignee: Navis
Priority: Minor
Labels: HiveServer2
Attachments: HIVE-4172.D9555.1.patch, HIVE-4172.D9555.2.patch,
HIVE-4172.D9555.3.patch, HIVE-4172.D9555.4.patch, HIVE-4172.D9555.5.patch
In beeline, select key, null from src fails with exception,
{noformat}
org.apache.hive.service.cli.HiveSQLException: Error running query:
java.lang.NullPointerException
at
org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:112)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:166)
at
org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:148)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:183)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1133)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1118)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:39)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4743) Improve test coverage of package org.apache.hadoop.hive.ql.io
[ https://issues.apache.org/jira/browse/HIVE-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HIVE-4743: - Affects Version/s: 0.12.0 0.11.1 0.10.1 Status: Patch Available (was: Open) Improve test coverage of package org.apache.hadoop.hive.ql.io - Key: HIVE-4743 URL: https://issues.apache.org/jira/browse/HIVE-4743 Project: Hive Issue Type: Test Affects Versions: 0.10.1, 0.11.1, 0.12.0 Reporter: Ivan A. Veselovsky Attachments: HIVE-4743-trunk--N1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4743) Improve test coverage of package org.apache.hadoop.hive.ql.io
[ https://issues.apache.org/jira/browse/HIVE-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HIVE-4743: - Attachment: HIVE-4743-trunk--N1.patch The patch intended to be applied to branches trunk, 0.11, 0.10. Improve test coverage of package org.apache.hadoop.hive.ql.io - Key: HIVE-4743 URL: https://issues.apache.org/jira/browse/HIVE-4743 Project: Hive Issue Type: Test Reporter: Ivan A. Veselovsky Attachments: HIVE-4743-trunk--N1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4743) Improve test coverage of package org.apache.hadoop.hive.ql.io
[ https://issues.apache.org/jira/browse/HIVE-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HIVE-4743: - Description: The patch improves unit test coverage of package org.apache.hadoop.hive.ql.io up to 80%. (was: the patch improves test coverage of package up to 80%.) Improve test coverage of package org.apache.hadoop.hive.ql.io - Key: HIVE-4743 URL: https://issues.apache.org/jira/browse/HIVE-4743 Project: Hive Issue Type: Test Affects Versions: 0.10.1, 0.11.1, 0.12.0 Reporter: Ivan A. Veselovsky Attachments: HIVE-4743-trunk--N1.patch The patch improves unit test coverage of package org.apache.hadoop.hive.ql.io up to 80%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4743) Improve test coverage of package org.apache.hadoop.hive.ql.io
[ https://issues.apache.org/jira/browse/HIVE-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HIVE-4743: - Description: the patch improves test coverage of package Improve test coverage of package org.apache.hadoop.hive.ql.io - Key: HIVE-4743 URL: https://issues.apache.org/jira/browse/HIVE-4743 Project: Hive Issue Type: Test Affects Versions: 0.10.1, 0.11.1, 0.12.0 Reporter: Ivan A. Veselovsky Attachments: HIVE-4743-trunk--N1.patch the patch improves test coverage of package -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4743) Improve test coverage of package org.apache.hadoop.hive.ql.io
[ https://issues.apache.org/jira/browse/HIVE-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HIVE-4743: - Description: the patch improves test coverage of package up to 80%. (was: the patch improves test coverage of package ) Improve test coverage of package org.apache.hadoop.hive.ql.io - Key: HIVE-4743 URL: https://issues.apache.org/jira/browse/HIVE-4743 Project: Hive Issue Type: Test Affects Versions: 0.10.1, 0.11.1, 0.12.0 Reporter: Ivan A. Veselovsky Attachments: HIVE-4743-trunk--N1.patch the patch improves test coverage of package up to 80%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3953) Reading of partitioned Avro data fails because of missing properties
[ https://issues.apache.org/jira/browse/HIVE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689380#comment-13689380 ] Hudson commented on HIVE-3953: -- Integrated in Hive-trunk-hadoop2 #249 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/249/]) HIVE-3953 : Reading of partitioned Avro data fails because of missing properties (Mark Wagner via Ashutosh Chauhan) (Revision 1494833) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1494833 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java * /hive/trunk/ql/src/test/queries/clientpositive/avro_partitioned.q * /hive/trunk/ql/src/test/results/clientpositive/avro_partitioned.q.out Reading of partitioned Avro data fails because of missing properties Key: HIVE-3953 URL: https://issues.apache.org/jira/browse/HIVE-3953 Project: Hive Issue Type: Bug Affects Versions: 0.11.0, 0.11.1, 0.12.0 Reporter: Mark Wagner Assignee: Mark Wagner Priority: Blocker Fix For: 0.12.0 Attachments: avro_partition_test.q, HIVE-3953.1.patch After HIVE-3833, reading partitioned Avro data fails due to missing properties. The avro.schema.(url|literal) properties are not making it all the way to the SerDe. Non-partitioned data can still be read. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4743) Improve test coverage of package org.apache.hadoop.hive.ql.io
[ https://issues.apache.org/jira/browse/HIVE-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689382#comment-13689382 ] Ashutosh Chauhan commented on HIVE-4743: [~iveselovsky] How are you measuring test coverage ? Improve test coverage of package org.apache.hadoop.hive.ql.io - Key: HIVE-4743 URL: https://issues.apache.org/jira/browse/HIVE-4743 Project: Hive Issue Type: Test Affects Versions: 0.10.1, 0.11.1, 0.12.0 Reporter: Ivan A. Veselovsky Attachments: HIVE-4743-trunk--N1.patch The patch improves unit test coverage of package org.apache.hadoop.hive.ql.io up to 80%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4767) ObjectStore.getPMF has concurrency problems
[ https://issues.apache.org/jira/browse/HIVE-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4767: --- Summary: ObjectStore.getPMF has concurrency problems (was: ObjectStore. getPMF has concurrency problems) ObjectStore.getPMF has concurrency problems --- Key: HIVE-4767 URL: https://issues.apache.org/jira/browse/HIVE-4767 Project: Hive Issue Type: Sub-task Affects Versions: 0.11.0 Reporter: Brock Noland Assignee: Brock Noland Fix For: 0.12.0 ObjectStore.getPMF is a static factory method but isn't thread safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4767) ObjectStore.getPMF has concurrency problems
[ https://issues.apache.org/jira/browse/HIVE-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4767: --- Attachment: HIVE-4767.patch Trivial patch. ObjectStore.getPMF has concurrency problems --- Key: HIVE-4767 URL: https://issues.apache.org/jira/browse/HIVE-4767 Project: Hive Issue Type: Sub-task Affects Versions: 0.11.0 Reporter: Brock Noland Assignee: Brock Noland Fix For: 0.12.0 Attachments: HIVE-4767.patch ObjectStore.getPMF is a static factory method but isn't thread safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4743) Improve test coverage of package org.apache.hadoop.hive.ql.io
[ https://issues.apache.org/jira/browse/HIVE-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689394#comment-13689394 ] Ivan A. Veselovsky commented on HIVE-4743: -- We measure coverage with Clover (http://www.atlassian.com/software/clover/overview). Clover integration with Hive described in https://issues.apache.org/jira/browse/HIVE-2991 . We use somewhat updated version of those changes, will update the patches in 2991 soon. Improve test coverage of package org.apache.hadoop.hive.ql.io - Key: HIVE-4743 URL: https://issues.apache.org/jira/browse/HIVE-4743 Project: Hive Issue Type: Test Affects Versions: 0.10.1, 0.11.1, 0.12.0 Reporter: Ivan A. Veselovsky Attachments: HIVE-4743-trunk--N1.patch The patch improves unit test coverage of package org.apache.hadoop.hive.ql.io up to 80%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4767) ObjectStore.getPMF has concurrency problems
[ https://issues.apache.org/jira/browse/HIVE-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4767: --- Status: Patch Available (was: Open) https://reviews.facebook.net/D11421 ObjectStore.getPMF has concurrency problems --- Key: HIVE-4767 URL: https://issues.apache.org/jira/browse/HIVE-4767 Project: Hive Issue Type: Sub-task Affects Versions: 0.11.0 Reporter: Brock Noland Assignee: Brock Noland Fix For: 0.12.0 Attachments: HIVE-4767.patch ObjectStore.getPMF is a static factory method but isn't thread safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4762) HMS cannot handle concurrent requests
[
https://issues.apache.org/jira/browse/HIVE-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689403#comment-13689403
]
Brock Noland commented on HIVE-4762:
I documented this
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin in
revision
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=27362076selectedPageVersions=12selectedPageVersions=11.
With the three fixes in HIVE-4759 I ran a long test overnight and I didn't have
the same Write concurrency issues.
HMS cannot handle concurrent requests
-
Key: HIVE-4762
URL: https://issues.apache.org/jira/browse/HIVE-4762
Project: Hive
Issue Type: Sub-task
Affects Versions: 0.11.0
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Critical
Fix For: 0.12.0
It appears our use of DataNucleaus is not correct or perhaps there is a bug
in the ancient version of DN we are using. On startup having multiple threads
performing show tables results in failures. Additionally concurrent DML
will fail event after startup. I used the program below to demonstrate this.
{noformat}
package org.apache.hadoop.hive.ql;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.Statement;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import org.apache.hive.jdbc.HiveDriver;
public class MultiThreadTest {
public static class QueryRunner implements Runnable {
int id;
double averageElapsedTime;
Connection connection;
Statement statement;
QueryRunner(int id) {
this.id = id;
}
@Override
public void run() {
long count = 0;
double elapsedTime = 0;
try {
connection =
DriverManager.getConnection(jdbc:hive2://localhost:1/default, brock,
password);
statement = connection.createStatement();
//statement.execute(DROP TABLE IF EXISTS t + id);
for (int i = 0; i 10; i++) {
// statement.execute(CREATE TABLE t + id + (key int));
long start = System.currentTimeMillis();
// statement.execute(DROP TABLE t + id);
statement.execute(SHOW TABLES);
elapsedTime += System.currentTimeMillis() - start;
count++;
}
if(statement != null) {
statement.close();
}
if(connection != null) {
connection.close();
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if(count 0) {
averageElapsedTime = elapsedTime / (double)count;
}
}
}
}
public static void main(String[] args) throws Exception {
int numThreads = 50;
Class.forName(HiveDriver.class.getName());
ExecutorService executor = Executors.newFixedThreadPool(numThreads);
QueryRunner[] queryRunners = new QueryRunner[numThreads];
for (int i = 0; i numThreads; i++) {
queryRunners[i] = new QueryRunner(i);
executor.execute(queryRunners[i]);
}
executor.shutdown();
while(!executor.isTerminated()) {
System.out.println(Waiting...);
Thread.sleep(1000L);
}
for (int i = 0; i numThreads; i++) {
System.out.println(Math.round(queryRunners[i].averageElapsedTime));
}
}
}
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-4762) HMS cannot handle concurrent requests
[
https://issues.apache.org/jira/browse/HIVE-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland resolved HIVE-4762.
Resolution: Fixed
HMS cannot handle concurrent requests
-
Key: HIVE-4762
URL: https://issues.apache.org/jira/browse/HIVE-4762
Project: Hive
Issue Type: Sub-task
Affects Versions: 0.11.0
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Critical
Fix For: 0.12.0
It appears our use of DataNucleaus is not correct or perhaps there is a bug
in the ancient version of DN we are using. On startup having multiple threads
performing show tables results in failures. Additionally concurrent DML
will fail event after startup. I used the program below to demonstrate this.
{noformat}
package org.apache.hadoop.hive.ql;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.Statement;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import org.apache.hive.jdbc.HiveDriver;
public class MultiThreadTest {
public static class QueryRunner implements Runnable {
int id;
double averageElapsedTime;
Connection connection;
Statement statement;
QueryRunner(int id) {
this.id = id;
}
@Override
public void run() {
long count = 0;
double elapsedTime = 0;
try {
connection =
DriverManager.getConnection(jdbc:hive2://localhost:1/default, brock,
password);
statement = connection.createStatement();
//statement.execute(DROP TABLE IF EXISTS t + id);
for (int i = 0; i 10; i++) {
// statement.execute(CREATE TABLE t + id + (key int));
long start = System.currentTimeMillis();
// statement.execute(DROP TABLE t + id);
statement.execute(SHOW TABLES);
elapsedTime += System.currentTimeMillis() - start;
count++;
}
if(statement != null) {
statement.close();
}
if(connection != null) {
connection.close();
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if(count 0) {
averageElapsedTime = elapsedTime / (double)count;
}
}
}
}
public static void main(String[] args) throws Exception {
int numThreads = 50;
Class.forName(HiveDriver.class.getName());
ExecutorService executor = Executors.newFixedThreadPool(numThreads);
QueryRunner[] queryRunners = new QueryRunner[numThreads];
for (int i = 0; i numThreads; i++) {
queryRunners[i] = new QueryRunner(i);
executor.execute(queryRunners[i]);
}
executor.shutdown();
while(!executor.isTerminated()) {
System.out.println(Waiting...);
Thread.sleep(1000L);
}
for (int i = 0; i numThreads; i++) {
System.out.println(Math.round(queryRunners[i].averageElapsedTime));
}
}
}
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4767) ObjectStore.getPMF has concurrency problems
[ https://issues.apache.org/jira/browse/HIVE-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689414#comment-13689414 ] Ashutosh Chauhan commented on HIVE-4767: Patch looks alright and I can see the concurrency issue. But can you elaborate on what was your test setup where this was evident and after this patch problem went away. ObjectStore.getPMF has concurrency problems --- Key: HIVE-4767 URL: https://issues.apache.org/jira/browse/HIVE-4767 Project: Hive Issue Type: Sub-task Affects Versions: 0.11.0 Reporter: Brock Noland Assignee: Brock Noland Fix For: 0.12.0 Attachments: HIVE-4767.patch ObjectStore.getPMF is a static factory method but isn't thread safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4767) ObjectStore.getPMF has concurrency problems
[ https://issues.apache.org/jira/browse/HIVE-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689436#comment-13689436 ] Brock Noland commented on HIVE-4767: Hi, I cannot say definitively I was impacted by this bug. I noted that this method has correctness issues while looking into HIVE-4762. My test was 50 client threads (HIVE-4762) - HS2 - HMS - MySQL. I ran it overnight and this patch introduced no new issues. Brock ObjectStore.getPMF has concurrency problems --- Key: HIVE-4767 URL: https://issues.apache.org/jira/browse/HIVE-4767 Project: Hive Issue Type: Sub-task Affects Versions: 0.11.0 Reporter: Brock Noland Assignee: Brock Noland Fix For: 0.12.0 Attachments: HIVE-4767.patch ObjectStore.getPMF is a static factory method but isn't thread safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4763) add support for thrift over http transport in HS2
[ https://issues.apache.org/jira/browse/HIVE-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689455#comment-13689455 ] Thejas M Nair commented on HIVE-4763: - Thanks to [~mikelid] for his contributions for this feature. Most of the changes in this patch were contributed by him (I reviewed the changes, added test cases, moved some classes around, added error handling changes). add support for thrift over http transport in HS2 - Key: HIVE-4763 URL: https://issues.apache.org/jira/browse/HIVE-4763 Project: Hive Issue Type: Sub-task Components: HiveServer2 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.12.0 Attachments: HIVE-4763.1.patch Subtask for adding support for http transport mode for thrift api in hive server2. Support for the different authentication modes will be part of another subtask. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4580) Change DDLTask to report errors using canonical error messages rather than http status codes
[ https://issues.apache.org/jira/browse/HIVE-4580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689467#comment-13689467 ] Eugene Koifman commented on HIVE-4580: -- List of files in HIVE-4580.1.patch git diff-tree --no-commit-id --name-only -r 91a702fd80f044a3d71efd3ba41a27b05f2a240a 8d0568474a8f734d59304c4d4165ae7fd8beeb75 contrib/src/test/results/clientnegative/serde_regex.q.out contrib/src/test/results/clientnegative/url_hook.q.out hcatalog/src/test/e2e/templeton/tests/ddl.conf hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/AppConfig.java hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/BadParam.java hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/BusyException.java hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/CallbackFailedException.java hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/CatchallExceptionMapper.java hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/HcatDelegator.java hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/HcatException.java hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/JsonBuilder.java hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/Main.java hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/NotAuthorizedException.java hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/QueueException.java hcatalog/webhcat/svr/src/test/java/org/apache/hcatalog/templeton/TestWebHCatE2e.java ql/src/java/org/apache/hadoop/hive/ql/Driver.java ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java ql/src/java/org/apache/hadoop/hive/ql/exec/TaskResult.java ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java ql/src/java/org/apache/hadoop/hive/ql/metadata/InvalidTableException.java ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatter.java ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java ql/src/test/results/clientnegative/add_partition_with_whitelist.q.out ql/src/test/results/clientnegative/addpart1.q.out ql/src/test/results/clientnegative/alter_partition_nodrop_table.q.out ql/src/test/results/clientnegative/alter_partition_with_whitelist.q.out ql/src/test/results/clientnegative/alter_rename_partition_failure.q.out ql/src/test/results/clientnegative/alter_rename_partition_failure2.q.out ql/src/test/results/clientnegative/alter_rename_partition_failure3.q.out ql/src/test/results/clientnegative/alter_table_wrong_regex.q.out ql/src/test/results/clientnegative/alter_view_failure4.q.out ql/src/test/results/clientnegative/altern1.q.out ql/src/test/results/clientnegative/archive1.q.out ql/src/test/results/clientnegative/archive2.q.out ql/src/test/results/clientnegative/archive_multi1.q.out ql/src/test/results/clientnegative/archive_multi2.q.out ql/src/test/results/clientnegative/archive_multi3.q.out ql/src/test/results/clientnegative/archive_multi4.q.out ql/src/test/results/clientnegative/archive_multi5.q.out ql/src/test/results/clientnegative/archive_multi6.q.out ql/src/test/results/clientnegative/archive_multi7.q.out ql/src/test/results/clientnegative/authorization_fail_1.q.out ql/src/test/results/clientnegative/column_rename1.q.out ql/src/test/results/clientnegative/column_rename2.q.out ql/src/test/results/clientnegative/column_rename4.q.out ql/src/test/results/clientnegative/create_table_failure3.q.out ql/src/test/results/clientnegative/create_table_failure4.q.out ql/src/test/results/clientnegative/create_table_wrong_regex.q.out ql/src/test/results/clientnegative/create_view_failure1.q.out ql/src/test/results/clientnegative/create_view_failure2.q.out ql/src/test/results/clientnegative/create_view_failure4.q.out ql/src/test/results/clientnegative/database_create_already_exists.q.out ql/src/test/results/clientnegative/database_create_invalid_name.q.out ql/src/test/results/clientnegative/database_drop_does_not_exist.q.out ql/src/test/results/clientnegative/database_drop_not_empty.q.out ql/src/test/results/clientnegative/database_drop_not_empty_restrict.q.out ql/src/test/results/clientnegative/database_switch_does_not_exist.q.out ql/src/test/results/clientnegative/deletejar.q.out ql/src/test/results/clientnegative/describe_xpath1.q.out ql/src/test/results/clientnegative/describe_xpath2.q.out ql/src/test/results/clientnegative/describe_xpath3.q.out ql/src/test/results/clientnegative/describe_xpath4.q.out
[jira] [Created] (HIVE-4768) fix multiple problems running parallel unit test framework
Thejas M Nair created HIVE-4768: --- Summary: fix multiple problems running parallel unit test framework Key: HIVE-4768 URL: https://issues.apache.org/jira/browse/HIVE-4768 Project: Hive Issue Type: Bug Components: Testing Infrastructure Affects Versions: 0.11.0 Reporter: Thejas M Nair Assignee: Thejas M Nair There are a few problems with the [parallel unit test scripts| https://cwiki.apache.org/confluence/display/Hive/Unit+Test+Parallel+Execution] - It tries to download apache-ant from a mirror that does not have it anymore - The git pull command without branch name does not work in some setups - It tries to cd to a dir which might exist - It runs tests using sudo, which is not necessary -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4763) add support for thrift over http transport in HS2
[ https://issues.apache.org/jira/browse/HIVE-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689510#comment-13689510 ] Carl Steinbach commented on HIVE-4763: -- Quick question: can we make it possible for an HS2 process to support HTTP and binary serialization on different ports at the same time? Seems like we should be able to do this by subclassing ThriftCLIService. add support for thrift over http transport in HS2 - Key: HIVE-4763 URL: https://issues.apache.org/jira/browse/HIVE-4763 Project: Hive Issue Type: Sub-task Components: HiveServer2 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.12.0 Attachments: HIVE-4763.1.patch Subtask for adding support for http transport mode for thrift api in hive server2. Support for the different authentication modes will be part of another subtask. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4733) HiveLockObjectData is not compared properly
[
https://issues.apache.org/jira/browse/HIVE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689516#comment-13689516
]
Phabricator commented on HIVE-4733:
---
brock has commented on the revision HIVE-4733 [jira] HiveLockObjectData is not
compared properly.
One comment about a new exception I saw.
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java:456
Looks like we still have an issue with getChildren(). I am able to get get the
following exception on trunk.
2013-06-20 13:42:05,313 ERROR ZooKeeperHiveLockManager pool-1-thread-17
(ZooKeeperHiveLockManager.java:unlockPrimitive(465)) - Failed to release
ZooKeeper lock:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /hive_zookeeper_namespace/default
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1448)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1476)
at
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.unlockPrimitive(ZooKeeperHiveLockManager.java:455)
REVISION DETAIL
https://reviews.facebook.net/D11277
BRANCH
HIVE-4733
ARCANIST PROJECT
hive
To: JIRA, ashutoshc, navis
Cc: brock
HiveLockObjectData is not compared properly
---
Key: HIVE-4733
URL: https://issues.apache.org/jira/browse/HIVE-4733
Project: Hive
Issue Type: Bug
Components: Locking
Reporter: Navis
Assignee: Navis
Priority: Trivial
Fix For: 0.12.0
Attachments: HIVE-4733.D11277.1.patch, HIVE-4733.D11277.2.patch
{noformat}
ret = ret (clientIp == null) ? target.getClientIp() == null :
clientIp.equals(target.getClientIp());
{noformat}
seemed intended to be
{noformat}
ret = ret (clientIp == null ? target.getClientIp() == null :
clientIp.equals(target.getClientIp()));
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4763) add support for thrift over http transport in HS2
[ https://issues.apache.org/jira/browse/HIVE-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689518#comment-13689518 ] Thejas M Nair commented on HIVE-4763: - Yes, that should be possible. But I am not sure if that is going to be a common use case. The config params introduced in this patch would work with that change as well - there is a separate config param for the http port. We would just need to add support for new value for the server mode, something like thrift+http. add support for thrift over http transport in HS2 - Key: HIVE-4763 URL: https://issues.apache.org/jira/browse/HIVE-4763 Project: Hive Issue Type: Sub-task Components: HiveServer2 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.12.0 Attachments: HIVE-4763.1.patch Subtask for adding support for http transport mode for thrift api in hive server2. Support for the different authentication modes will be part of another subtask. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4675) Create new parallel unit test environment
[ https://issues.apache.org/jira/browse/HIVE-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4675: --- Fix Version/s: 0.12.0 Status: Patch Available (was: Open) Create new parallel unit test environment - Key: HIVE-4675 URL: https://issues.apache.org/jira/browse/HIVE-4675 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Reporter: Brock Noland Assignee: Brock Noland Fix For: 0.12.0 Attachments: HIVE-4675.patch The current ptest tool is great, but it has the following limitations: -Requires an NFS filer -Unless the NFS filer is dedicated ptests can become IO bound easily -Investigating of failures is troublesome because the source directory for the failure is not saved -Ignoring or isolated tests is not supported -No unit tests for the ptest framework exist It'd be great to have a ptest tool that addresses this limitations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4675) Create new parallel unit test environment
[ https://issues.apache.org/jira/browse/HIVE-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4675: --- Attachment: HIVE-4675.patch https://reviews.facebook.net/D11427 Create new parallel unit test environment - Key: HIVE-4675 URL: https://issues.apache.org/jira/browse/HIVE-4675 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-4675.patch The current ptest tool is great, but it has the following limitations: -Requires an NFS filer -Unless the NFS filer is dedicated ptests can become IO bound easily -Investigating of failures is troublesome because the source directory for the failure is not saved -Ignoring or isolated tests is not supported -No unit tests for the ptest framework exist It'd be great to have a ptest tool that addresses this limitations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4769) Vectorized ORC reader does not handle absence of column present stream correctly.
Sarvesh Sakalanaga created HIVE-4769: Summary: Vectorized ORC reader does not handle absence of column present stream correctly. Key: HIVE-4769 URL: https://issues.apache.org/jira/browse/HIVE-4769 Project: Hive Issue Type: Sub-task Reporter: Sarvesh Sakalanaga Assignee: Sarvesh Sakalanaga If the present stream is not present for a column then by design all the rows are present for that column. Currently ORC vectorized reader does initialize the IsNull vector correctly in this case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4769) Vectorized ORC reader does not handle absence of column present stream correctly.
[ https://issues.apache.org/jira/browse/HIVE-4769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarvesh Sakalanaga updated HIVE-4769: - Status: Patch Available (was: Open) Vectorized ORC reader does not handle absence of column present stream correctly. -- Key: HIVE-4769 URL: https://issues.apache.org/jira/browse/HIVE-4769 Project: Hive Issue Type: Sub-task Reporter: Sarvesh Sakalanaga Assignee: Sarvesh Sakalanaga Attachments: Hive-4769.0.patch If the present stream is not present for a column then by design all the rows are present for that column. Currently ORC vectorized reader does initialize the IsNull vector correctly in this case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4769) Vectorized ORC reader does not handle absence of column present stream correctly.
[ https://issues.apache.org/jira/browse/HIVE-4769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarvesh Sakalanaga updated HIVE-4769: - Attachment: Hive-4769.0.patch Patch available. Vectorized ORC reader does not handle absence of column present stream correctly. -- Key: HIVE-4769 URL: https://issues.apache.org/jira/browse/HIVE-4769 Project: Hive Issue Type: Sub-task Reporter: Sarvesh Sakalanaga Assignee: Sarvesh Sakalanaga Attachments: Hive-4769.0.patch If the present stream is not present for a column then by design all the rows are present for that column. Currently ORC vectorized reader does initialize the IsNull vector correctly in this case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4763) add support for thrift over http transport in HS2
[ https://issues.apache.org/jira/browse/HIVE-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689643#comment-13689643 ] Carl Steinbach commented on HIVE-4763: -- Any idea how http compares to binary performance wise? add support for thrift over http transport in HS2 - Key: HIVE-4763 URL: https://issues.apache.org/jira/browse/HIVE-4763 Project: Hive Issue Type: Sub-task Components: HiveServer2 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.12.0 Attachments: HIVE-4763.1.patch Subtask for adding support for http transport mode for thrift api in hive server2. Support for the different authentication modes will be part of another subtask. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2055) Hive HBase Integration issue
[ https://issues.apache.org/jira/browse/HIVE-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689662#comment-13689662 ] Nick Dimiduk commented on HIVE-2055: The topic of this issue came up in conversation with [~cnauroth]. He pointed me toward a Hadoop feature used by YARN to get around this very issue. Perhaps we can make use of the same. https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java#L1140 Hive HBase Integration issue Key: HIVE-2055 URL: https://issues.apache.org/jira/browse/HIVE-2055 Project: Hive Issue Type: Bug Components: HBase Handler Affects Versions: 0.10.0 Reporter: sajith v Attachments: 0001-HIVE-2055-include-hbase-dependencies-in-launch-scrip.patch, HIVE-2055.patch Created an external table in hive , which points to the HBase table. When tried to query a column using the column name in select clause got the following exception : ( java.lang.ClassNotFoundException: org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat), errorCode:12, SQLState:42000) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4763) add support for thrift over http transport in HS2
[ https://issues.apache.org/jira/browse/HIVE-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689708#comment-13689708 ] Thejas M Nair commented on HIVE-4763: - No, haven't run any performance benchmarks on it yet. add support for thrift over http transport in HS2 - Key: HIVE-4763 URL: https://issues.apache.org/jira/browse/HIVE-4763 Project: Hive Issue Type: Sub-task Components: HiveServer2 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.12.0 Attachments: HIVE-4763.1.patch Subtask for adding support for http transport mode for thrift api in hive server2. Support for the different authentication modes will be part of another subtask. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4770) java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row
Tony Murphy created HIVE-4770:
-
Summary: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row
Key: HIVE-4770
URL: https://issues.apache.org/jira/browse/HIVE-4770
Project: Hive
Issue Type: Sub-task
Affects Versions: vectorization-branch
Reporter: Tony Murphy
Fix For: vectorization-branch
Attachments: output.txt, tableAndData.zip
Table and data attached.
{noformat}
SELECT cfloat,
csmallint,
cint,
ctimestamp,
(cfloat + 10),
STDDEV_SAMP(cfloat),
(-((cfloat + 10))),
(cint / cfloat),
MAX(cint),
(-(cint)),
(cint * STDDEV_SAMP(cfloat)),
STDDEV_SAMP(cint),
VAR_SAMP(cint),
(-(MAX(cint))),
((-(MAX(cint))) / 0.E+000)
FROM alltypes_orc
WHERE(((1 = cfloat)
OR (cstring2 LIKE '%b'))
OR ((cint = csmallint)
OR (cstring2 LIKE '%ss')))
GROUP BY cfloat, csmallint, cint, ctimestamp
ORDER BY cint, cfloat;
{noformat}
{noformat}
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
{ctinyint:null,csmallint:-3806,cint:-66533315,cbigint:null,cdouble:null,cfloat:152.95706,cstring1:null,cstring2:null,ctimestamp:9131-01-01
16:52:03.53,cboolean:null}
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.mapred.Child.main(Child.java:265)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row
{ctinyint:null,csmallint:-3806,cint:-66533315,cbigint:null,cdouble:null,cfloat:152.95706,cstring1:null,cstring2:null,ctimestamp:9131-01-01
16:52:03.53,cboolean:null}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:671)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:796)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
at
org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:136)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:652)
... 9 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.shouldBeFlushed(GroupByOperator.java:941)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:836)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:723)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:791)
... 21 more
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4770) java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row
[
https://issues.apache.org/jira/browse/HIVE-4770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tony Murphy updated HIVE-4770:
--
Attachment: output.txt
tableAndData.zip
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
--
Key: HIVE-4770
URL: https://issues.apache.org/jira/browse/HIVE-4770
Project: Hive
Issue Type: Sub-task
Affects Versions: vectorization-branch
Reporter: Tony Murphy
Fix For: vectorization-branch
Attachments: output.txt, tableAndData.zip
Table and data attached.
{noformat}
SELECT cfloat,
csmallint,
cint,
ctimestamp,
(cfloat + 10),
STDDEV_SAMP(cfloat),
(-((cfloat + 10))),
(cint / cfloat),
MAX(cint),
(-(cint)),
(cint * STDDEV_SAMP(cfloat)),
STDDEV_SAMP(cint),
VAR_SAMP(cint),
(-(MAX(cint))),
((-(MAX(cint))) / 0.E+000)
FROM alltypes_orc
WHERE(((1 = cfloat)
OR (cstring2 LIKE '%b'))
OR ((cint = csmallint)
OR (cstring2 LIKE '%ss')))
GROUP BY cfloat, csmallint, cint, ctimestamp
ORDER BY cint, cfloat;
{noformat}
{noformat}
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
{ctinyint:null,csmallint:-3806,cint:-66533315,cbigint:null,cdouble:null,cfloat:152.95706,cstring1:null,cstring2:null,ctimestamp:9131-01-01
16:52:03.53,cboolean:null}
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.mapred.Child.main(Child.java:265)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row
{ctinyint:null,csmallint:-3806,cint:-66533315,cbigint:null,cdouble:null,cfloat:152.95706,cstring1:null,cstring2:null,ctimestamp:9131-01-01
16:52:03.53,cboolean:null}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:671)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:796)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
at
org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:136)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:652)
... 9 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.shouldBeFlushed(GroupByOperator.java:941)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:836)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:723)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:791)
... 21 more
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4771) Support subqueries in INSERT for array types
Michael Malak created HIVE-4771: --- Summary: Support subqueries in INSERT for array types Key: HIVE-4771 URL: https://issues.apache.org/jira/browse/HIVE-4771 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Michael Malak Since Hive supports SQL1999-style array types for columns, it would be nice for there to be a way to INSERT non-static data into such columns -- i.e. from another table based on a complex query as opposed to loading from a static file, loading from hard-coded values within the INSERT query, or copying complete arrays verbatim from another table. An example can be found at: http://www.postgresql.org/message-id/20041028232152.ga76...@winnie.fuhr.org CREATE TABLE table_a(a int, b int, c int[]); INSERT INTO table_a SELECT a, b, ARRAY(SELECT c FROM table_c WHERE table_c.parent = table_b.id) FROM table_b This should be implemented after regular correlated and uncorrelated subqueries are implemented: https://issues.apache.org/jira/browse/HIVE-784 Support uncorrelated subqueries in the WHERE clause https://issues.apache.org/jira/browse/HIVE-1799 Support correlated subqueries in the WHERE clause -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4772) Enable parallel execution of various E2E tests
Deepesh Khandelwal created HIVE-4772: Summary: Enable parallel execution of various E2E tests Key: HIVE-4772 URL: https://issues.apache.org/jira/browse/HIVE-4772 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal The e2e tests currently run in the sequential mode but the test harness that is used from the pig codebase allows parallel execution (PIG-2898). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4772) Enable parallel execution of various E2E tests
[ https://issues.apache.org/jira/browse/HIVE-4772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepesh Khandelwal updated HIVE-4772: - Status: Patch Available (was: Open) Enable parallel execution of various E2E tests -- Key: HIVE-4772 URL: https://issues.apache.org/jira/browse/HIVE-4772 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal Attachments: HIVE-4772.patch The e2e tests currently run in the sequential mode but the test harness that is used from the pig codebase allows parallel execution (PIG-2898). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4772) Enable parallel execution of various E2E tests
[ https://issues.apache.org/jira/browse/HIVE-4772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepesh Khandelwal updated HIVE-4772: - Attachment: HIVE-4772.patch Attached is the patch that changes the drivers and build script to enable parallel execution of e2e tests. Enable parallel execution of various E2E tests -- Key: HIVE-4772 URL: https://issues.apache.org/jira/browse/HIVE-4772 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Reporter: Deepesh Khandelwal Assignee: Deepesh Khandelwal Attachments: HIVE-4772.patch The e2e tests currently run in the sequential mode but the test harness that is used from the pig codebase allows parallel execution (PIG-2898). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4773) Templeton intermittently fail to commit output to file system
Shuaishuai Nie created HIVE-4773: Summary: Templeton intermittently fail to commit output to file system Key: HIVE-4773 URL: https://issues.apache.org/jira/browse/HIVE-4773 Project: Hive Issue Type: Bug Reporter: Shuaishuai Nie With ASV as a default FS, we saw instances where output is not fully flushed to storage before the Templeton controller process exits. This results in stdout and stderr being empty even though the job completed successfully. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4773) Templeton intermittently fail to commit output to file system
[ https://issues.apache.org/jira/browse/HIVE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuaishuai Nie updated HIVE-4773: - Attachment: HIVE-4773.1.patch Templeton intermittently fail to commit output to file system - Key: HIVE-4773 URL: https://issues.apache.org/jira/browse/HIVE-4773 Project: Hive Issue Type: Bug Reporter: Shuaishuai Nie Attachments: HIVE-4773.1.patch With ASV as a default FS, we saw instances where output is not fully flushed to storage before the Templeton controller process exits. This results in stdout and stderr being empty even though the job completed successfully. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4773) Templeton intermittently fail to commit output to file system
[ https://issues.apache.org/jira/browse/HIVE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuaishuai Nie updated HIVE-4773: - Assignee: Shuaishuai Nie Status: Patch Available (was: Open) Templeton intermittently fail to commit output to file system - Key: HIVE-4773 URL: https://issues.apache.org/jira/browse/HIVE-4773 Project: Hive Issue Type: Bug Reporter: Shuaishuai Nie Assignee: Shuaishuai Nie Attachments: HIVE-4773.1.patch With ASV as a default FS, we saw instances where output is not fully flushed to storage before the Templeton controller process exits. This results in stdout and stderr being empty even though the job completed successfully. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4774) Hive Runtime Error while closing operators: java.lang.NullPointerException
Tony Murphy created HIVE-4774:
-
Summary: Hive Runtime Error while closing operators:
java.lang.NullPointerException
Key: HIVE-4774
URL: https://issues.apache.org/jira/browse/HIVE-4774
Project: Hive
Issue Type: Sub-task
Affects Versions: vectorization-branch
Reporter: Tony Murphy
Fix For: vectorization-branch
Attachments: output.txt, tableAndData.zip
table and data attached
{noformat}
SELECT cdouble,
cint,
(cdouble - -1),
(-((cdouble - -1))),
MIN((-((cdouble - -1,
STDDEV_SAMP(cdouble),
(0.E+000 / (cdouble - -1))
FROM alltypes_orc
WHERE((cdouble cint)
OR (cdouble = cint))
GROUP BY cdouble, cint
ORDER BY cint;
{noformat}
{noformat}
java.lang.RuntimeException: Hive Runtime Error while closing operators
at
org.apache.hadoop.hive.ql.exec.vector.VectorExecMapper.close(VectorExecMapper.java:229)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.mapred.Child.main(Child.java:265)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:292)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serializeStruct(LazyBinarySerDe.java:257)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:204)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:245)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.flush(VectorGroupByOperator.java:302)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.closeOp(VectorGroupByOperator.java:423)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at
org.apache.hadoop.hive.ql.exec.vector.VectorExecMapper.close(VectorExecMapper.java:196)
... 8 more
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4774) Hive Runtime Error while closing operators: java.lang.NullPointerException
[
https://issues.apache.org/jira/browse/HIVE-4774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tony Murphy updated HIVE-4774:
--
Attachment: output.txt
tableAndData.zip
Hive Runtime Error while closing operators: java.lang.NullPointerException
--
Key: HIVE-4774
URL: https://issues.apache.org/jira/browse/HIVE-4774
Project: Hive
Issue Type: Sub-task
Affects Versions: vectorization-branch
Reporter: Tony Murphy
Fix For: vectorization-branch
Attachments: output.txt, tableAndData.zip
table and data attached
{noformat}
SELECT cdouble,
cint,
(cdouble - -1),
(-((cdouble - -1))),
MIN((-((cdouble - -1,
STDDEV_SAMP(cdouble),
(0.E+000 / (cdouble - -1))
FROM alltypes_orc
WHERE((cdouble cint)
OR (cdouble = cint))
GROUP BY cdouble, cint
ORDER BY cint;
{noformat}
{noformat}
java.lang.RuntimeException: Hive Runtime Error while closing operators
at
org.apache.hadoop.hive.ql.exec.vector.VectorExecMapper.close(VectorExecMapper.java:229)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:271)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.mapred.Child.main(Child.java:265)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:292)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serializeStruct(LazyBinarySerDe.java:257)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:204)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:245)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.flush(VectorGroupByOperator.java:302)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.closeOp(VectorGroupByOperator.java:423)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at
org.apache.hadoop.hive.ql.exec.vector.VectorExecMapper.close(VectorExecMapper.java:196)
... 8 more
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4775) getColumnTypeName returns string for int partition columns
Miklos Christine created HIVE-4775:
--
Summary: getColumnTypeName returns string for int partition columns
Key: HIVE-4775
URL: https://issues.apache.org/jira/browse/HIVE-4775
Project: Hive
Issue Type: Bug
Components: JDBC
Affects Versions: 0.9.0
Reporter: Miklos Christine
For an 'int' partition column, the 'getColumnTypeName' metadata call is
returning type 'string'. However, a DESCRIBE correctly returns type 'int'.
Doug Sedlak mentioned this issue in a comment of Hive-2315.
https://issues.apache.org/jira/browse/HIVE-2315
A patch was submitted, but the ticket was closed before then.
This appears to be a bug.
Similar issue here to address the consistency of the call:
https://issues.apache.org/jira/browse/HIVE-4519
Sample table is:
create EXTERNAL table partition_by_int (str_val string) PARTITIONED BY (int_col
INT) ROW FORMAT delimited fields terminated by ',' STORED AS TEXTFILE;
And (shortened) JDBC code is:
res = stmt.executeQuery(select * from partition_by_int);
ResultSetMetaData metaData = res.getMetaData();
for (int i = 0; i metaData.getColumnCount(); i++)
{ System.out.print(metaData.getColumnName(i + 1) + \t);
System.out.print(metaData.getColumnDisplaySize(i + 1) + \t);
System.out.println(metaData.getColumnTypeName(i + 1)); }
With output:
str_val 2147483647 string
int_col 2147483647 string
In contrast, DESCRIBE partition_by_int generates:
str_val string
int_col int
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4703) WebHcat looses column comments and returns 'from deserializer'
[
https://issues.apache.org/jira/browse/HIVE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689878#comment-13689878
]
Eugene Koifman commented on HIVE-4703:
--
hive show tables;
OK
Time taken: 2.016 seconds
hive CREATE TABLE if not exists pokes (foo INT COMMENT my column comment)
COMMENT table comment;
OK
Time taken: 0.256 seconds
hive desc pokes;
OK
foo int my column comment
Time taken: 0.14 seconds, Fetched: 1 row(s)
hive show create table pokes;
OK
CREATE TABLE pokes(
foo int COMMENT 'my column comment')
COMMENT 'table comment'
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'file:/user/hive/warehouse/pokes'
TBLPROPERTIES (
'transient_lastDdlTime'='1371773134')
Time taken: 0.083 seconds, Fetched: 13 row(s)
hive
WebHcat looses column comments and returns 'from deserializer'
--
Key: HIVE-4703
URL: https://issues.apache.org/jira/browse/HIVE-4703
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.12.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Fix For: 0.12.0
Attachments: webhcatMini.sh
This causes Templeton e2e tests to fail.
start WeHcat server (build/dist/hcatalog/sbin/webhcat_server.sh start)
run commands in the attached webhcatMini.sh.
It creates a table with some comments on columns.
When executing describe (GET) on this table, original comments are lost and
are replaced with from deserializer string.
Here is the output of these commands:
localhost:dev ekoifman$ webhcatMini.sh
Running delete test_table ifExists
HTTP/1.1 200 OK
Set-Cookie:
hadoop.auth=u=ekoifmanp=ekoifmant=simplee=1370945567179s=vIBKhGQwzs5pPAY3IkhyPpDkWrY=;Version=1;Path=/;Discard
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(7.6.0.v20120127)
{table:test_table,database:default}
Running create test_table
HTTP/1.1 200 OK
Set-Cookie:
hadoop.auth=u=ekoifmanp=ekoifmant=simplee=1370945569788s=g37NbyyRnf667IciUiIpIQNYGOo=;Version=1;Path=/;Discard
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(7.6.0.v20120127)
{table:test_table,database:default}
Running describe test_table
HTTP/1.1 200 OK
Set-Cookie:
hadoop.auth=u=ekoifmanp=ekoifmant=simplee=1370945572423s=7kE1FOn1Co2JQzZfW0V1myqulw0=;Version=1;Path=/;Discard
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(7.6.0.v20120127)
{columns:[{name:int,comment:from
deserializer,type:string},{name:int2,comment:from
deserializer,type:int}],database:default,table:test_table}
Mon Jun 10 17:12:55 PDT 2013
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 2159 - Still Failing
Changes for Build #2125
[daijy] PIG-3337: Fix remaining Window e2e tests
Changes for Build #2126
[hashutosh] HIVE-4615 : Invalid column names allowed when created dynamically
by a SerDe (Gabriel Reid via Ashutosh Chauhan)
[hashutosh] HIVE-3846 : alter view rename NPEs with authorization on. (Teddy
Choi via Ashutosh Chauhan)
[hashutosh] HIVE-4403 : Running Hive queries on Yarn (MR2) gives warnings
related to overriding final parameters (Chu Tong via Ashutosh Chauhan)
[hashutosh] HIVE-4610 : HCatalog checkstyle violation after HIVE4578 (Brock
Noland via Ashutosh Chauhan)
[hashutosh] HIVE-4636 : Failing on TestSemanticAnalysis.testAddReplaceCols in
trunk (Navis via Ashutosh Chauhan)
[hashutosh] HIVE-4626 : join_vc.q is not deterministic (Navis via Ashutosh
Chauhan)
[hashutosh] HIVE-4562 : HIVE3393 brought in Jackson library,and these four jars
should be packed into hive-exec.jar (caofangkun via Ashutosh Chauhan)
[hashutosh] HIVE-4489 : beeline always return the same error message twice
(Chaoyu Tang via Ashutosh Chauhan)
[hashutosh] HIVE-4510 : HS2 doesn't nest exceptions properly (fun debug times)
(Thejas Nair via Ashutosh Chauhan)
[hashutosh] HIVE-4535 : hive build fails with hadoop 0.20 (Thejas Nair via
Ashutosh Chauhan)
Changes for Build #2127
[hashutosh] HIVE-4585 : Remove unused MR Temp file localization from Tasks
(Gunther Hagleitner via Ashutosh Chauhan)
[hashutosh] HIVE-4418 : TestNegativeCliDriver failure message if cmd succeeds
is misleading (Thejas Nair via Ashutosh Chauhan)
[navis] HIVE-4620 MR temp directory conflicts in case of parallel execution
mode (Prasad Mujumdar via Navis)
Changes for Build #2128
[hashutosh] HIVE-4646 : skewjoin.q is failing in hadoop2 (Navis via Ashutosh
Chauhan)
[hashutosh] HIVE-4377 : Add more comment to https://reviews.facebook.net/D1209
(HIVE2340) : (Navis via Ashutosh Chauhan)
[hashutosh] HIVE-4546 : Hive CLI leaves behind the per session resource
directory on non-interactive invocation (Prasad Mujumdar via Ashutosh Chauhan)
[gates] HIVE-2670 A cluster test utility for Hive (gates and Johnny Zhang via
gates)
Changes for Build #2129
[hashutosh] HIVE-2304 : Support PreparedStatement.setObject (Ido Hadanny via
Ashutosh Chauhan)
[hashutosh] HIVE-4526 : auto_sortmerge_join_9.q throws NPE but test is
succeeded (Navis via Ashutosh Chauhan)
[hashutosh] HIVE-4516 : Fix concurrency bug in
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java (Jon
Hartlaub and Navis via Ashutosh Chauhan)
[hashutosh] HIVE-4566 : NullPointerException if typeinfo and nativesql commands
are executed at beeline before a DB connection is established (Xuefu Zhang via
Ashutosh Chauhan)
Changes for Build #2130
Changes for Build #2131
[hashutosh] HIVE-4422 : Test output need to be updated for Windows only unit
test in TestCliDriver (Shuaishuai Nie via Ashutosh Chauhan)
[hashutosh] HIVE-4657 : HCatalog checkstyle violation after HIVE-2670
(Shreepadma Venugopalan via Ashutosh Chauhan)
[hashutosh] HIVE-4348 : Unit test compile fail at hbase-handler project on
Windows becuase of illegal escape character (Shuaishuai Nie via Ashutosh
Chauhan)
[hashutosh] HIVE-4554 : Failed to create a table from existing file if file
path has spaces (Xuefu Zhang via Ashutosh Chauhan)
[hashutosh] HIVE-4459 : Script hcat is overriding HIVE_CONF_DIR variable (Jarek
Jarcec Cecho via Ashutosh Chauhan)
Changes for Build #2132
[hashutosh] HIVE-4435 : Column stats: Distinct value estimator should use hash
functions that are pairwise independent (Shreepadma Venugopalan via Ashutosh
Chauhan)
Changes for Build #2133
Changes for Build #2134
Changes for Build #2135
Changes for Build #2136
Changes for Build #2137
[navis] HIVE-4549 : JDBC compliance change TABLE_SCHEMA to TABLE_SCHEM (Prasad
Mujumdar via Navis)
Changes for Build #2138
[navis] HIVE-4691 : orc_createas1.q has minor inconsistency (Brock Noland via
Navis)
Changes for Build #2139
[hashutosh] HIVE-4712 : Fix TestCliDriver.truncate_* on 0.23 (Brock Noland via
Ashutosh Chauhan)
Changes for Build #2140
[navis] HIVE-4708 : Fix TestCliDriver.combine2.q on 0.23 (Brock Noland via
Navis)
Changes for Build #2141
Changes for Build #2142
[hashutosh] HIVE-4717 : Fix non-deterministic TestCliDriver on 0.23 (Brock
Noland via Ashutosh Chauhan)
Changes for Build #2143
Changes for Build #2144
[hashutosh] HIVE-4713 : Fix TestCliDriver.skewjoin_union_remove_{1,2}.q on 0.23
(Brock Noland via Ashutosh Chauhan)
[hashutosh] HIVE-4715 : Fix
TestCliDriver.{recursive_dir.q,sample_islocalmode_hook.q,input12.q,input39.q,auto_join14.q}
on 0.23 (Brock Noland via Ashutosh Chauhan)
Changes for Build #2145
Changes for Build #2146
[ecapriolo] Submitted by: Brock Noland Jonathon Chang
Reviewed by: Edward Capriolo
Approved by: Edward Capriolo
Changes for Build #2147
Changes for Build #2148
Changes for Build #2149
[navis] HIVE-4708 : Temporary files are not closed in PTFPersistence on jvm
[jira] [Commented] (HIVE-4761) ZooKeeperHiveLockManage.unlockPrimitive has race condition with threads
[ https://issues.apache.org/jira/browse/HIVE-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689924#comment-13689924 ] Hudson commented on HIVE-4761: -- Integrated in Hive-trunk-h0.21 #2159 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2159/]) HIVE-4761 : ZooKeeperHiveLockManage.unlockPrimitive has race condition with threads (Brock Noland via Ashutosh Chauhan) (Revision 1495055) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1495055 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java ZooKeeperHiveLockManage.unlockPrimitive has race condition with threads --- Key: HIVE-4761 URL: https://issues.apache.org/jira/browse/HIVE-4761 Project: Hive Issue Type: Sub-task Affects Versions: 0.11.0 Reporter: Brock Noland Assignee: Brock Noland Priority: Critical Fix For: 0.12.0 Attachments: HIVE-4761.patch, HIVE-4761.patch In unlockPrimitive, we check to see if children exist and if not delete the parent node. If two threads do this at the same time it's possible for two threads to call Zookeeper.delete() on the same node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4703) WebHcat looses column comments and returns 'from deserializer'
[
https://issues.apache.org/jira/browse/HIVE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689945#comment-13689945
]
Eugene Koifman commented on HIVE-4703:
--
When table is create like so:
curl -i -X PUT -HContent-type:application/json \
-d '{ comment:test comment, columns: [ { name: int, type:
string, comment:1st column comment }, {name:int2, type:int} ],
format: { storedAs: rcfile } }' \
'http://localhost:50111/templeton/v1/ddl/database/default/table/test_table?user.name=ekoifman'
and then 'desc test_table' from Hive CLI the output looks like:
hive desc test_table;
OK
int string from deserializer
int2int from deserializer
Time taken: 0.099 seconds, Fetched: 2 row(s)
or curl -i -X GET
'http://localhost:50111/templeton/v1/ddl/database/default/table/test_table?user.name=ekoifman'
ql.exec.DDLTask.describeTable() calls
ql.metadata.Table.getCols()
getCols() has access to a StorageDescriptor instance which as the correct
column Comments but it's not
using it. Instead it branches to use
Hive.getFieldsFromDeserializer(getTableName(), getDeserializer())
It looks like metastore.MetaStoreUtil.determineFieldComment() produces from
deserializer message.
Also, there are dozens of .q.out and other test files are written to expect
from deserializer message.
So I'm not sure how WebHcat e2e tests worked before as written.
See for example, tests 4 8 in REST_DDL_TABLE_BASIC group in
hcatalog/src/test/e2e/templeton/tests/ddl.conf
WebHcat looses column comments and returns 'from deserializer'
--
Key: HIVE-4703
URL: https://issues.apache.org/jira/browse/HIVE-4703
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.12.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Fix For: 0.12.0
Attachments: webhcatMini.sh
This causes Templeton e2e tests to fail.
start WeHcat server (build/dist/hcatalog/sbin/webhcat_server.sh start)
run commands in the attached webhcatMini.sh.
It creates a table with some comments on columns.
When executing describe (GET) on this table, original comments are lost and
are replaced with from deserializer string.
Here is the output of these commands:
localhost:dev ekoifman$ webhcatMini.sh
Running delete test_table ifExists
HTTP/1.1 200 OK
Set-Cookie:
hadoop.auth=u=ekoifmanp=ekoifmant=simplee=1370945567179s=vIBKhGQwzs5pPAY3IkhyPpDkWrY=;Version=1;Path=/;Discard
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(7.6.0.v20120127)
{table:test_table,database:default}
Running create test_table
HTTP/1.1 200 OK
Set-Cookie:
hadoop.auth=u=ekoifmanp=ekoifmant=simplee=1370945569788s=g37NbyyRnf667IciUiIpIQNYGOo=;Version=1;Path=/;Discard
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(7.6.0.v20120127)
{table:test_table,database:default}
Running describe test_table
HTTP/1.1 200 OK
Set-Cookie:
hadoop.auth=u=ekoifmanp=ekoifmant=simplee=1370945572423s=7kE1FOn1Co2JQzZfW0V1myqulw0=;Version=1;Path=/;Discard
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(7.6.0.v20120127)
{columns:[{name:int,comment:from
deserializer,type:string},{name:int2,comment:from
deserializer,type:int}],database:default,table:test_table}
Mon Jun 10 17:12:55 PDT 2013
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4172) JDBC2 does not support VOID type
[
https://issues.apache.org/jira/browse/HIVE-4172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-4172:
---
Resolution: Fixed
Fix Version/s: 0.12.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks, Navis!
JDBC2 does not support VOID type
Key: HIVE-4172
URL: https://issues.apache.org/jira/browse/HIVE-4172
Project: Hive
Issue Type: Improvement
Components: HiveServer2, JDBC
Affects Versions: 0.11.0
Reporter: Navis
Assignee: Navis
Priority: Minor
Labels: HiveServer2
Fix For: 0.12.0
Attachments: HIVE-4172.D9555.1.patch, HIVE-4172.D9555.2.patch,
HIVE-4172.D9555.3.patch, HIVE-4172.D9555.4.patch, HIVE-4172.D9555.5.patch
In beeline, select key, null from src fails with exception,
{noformat}
org.apache.hive.service.cli.HiveSQLException: Error running query:
java.lang.NullPointerException
at
org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:112)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:166)
at
org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:148)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:183)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1133)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1118)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:39)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4743) Improve test coverage of package org.apache.hadoop.hive.ql.io
[ https://issues.apache.org/jira/browse/HIVE-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689983#comment-13689983 ] Ashutosh Chauhan commented on HIVE-4743: +1 Improve test coverage of package org.apache.hadoop.hive.ql.io - Key: HIVE-4743 URL: https://issues.apache.org/jira/browse/HIVE-4743 Project: Hive Issue Type: Test Affects Versions: 0.10.1, 0.11.1, 0.12.0 Reporter: Ivan A. Veselovsky Attachments: HIVE-4743-trunk--N1.patch The patch improves unit test coverage of package org.apache.hadoop.hive.ql.io up to 80%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3297) change hive.auto.convert.join's default value to true
[ https://issues.apache.org/jira/browse/HIVE-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13689985#comment-13689985 ] wan kun commented on HIVE-3297: --- hello,I find while the code had updated ,the configurate file hive-default.xml is the old version. this makes me confused. Should we update that??? property namehive.auto.convert.join/name valuefalse/value descriptionWhether Hive enable the optimization about converting common join into mapjoin based on the input file size/description /property change hive.auto.convert.join's default value to true - Key: HIVE-3297 URL: https://issues.apache.org/jira/browse/HIVE-3297 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Ashutosh Chauhan Fix For: 0.11.0 Attachments: HIVE-3297.patch For unit tests also, this parameter should be set to true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4743) Improve test coverage of package org.apache.hadoop.hive.ql.io
[ https://issues.apache.org/jira/browse/HIVE-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1368#comment-1368 ] Brock Noland commented on HIVE-4743: Nice, I love to see new unit test coverage! :) Improve test coverage of package org.apache.hadoop.hive.ql.io - Key: HIVE-4743 URL: https://issues.apache.org/jira/browse/HIVE-4743 Project: Hive Issue Type: Test Affects Versions: 0.10.1, 0.11.1, 0.12.0 Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Attachments: HIVE-4743-trunk--N1.patch The patch improves unit test coverage of package org.apache.hadoop.hive.ql.io up to 80%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4703) WebHcat looses column comments and returns 'from deserializer'
[
https://issues.apache.org/jira/browse/HIVE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690008#comment-13690008
]
Eugene Koifman commented on HIVE-4703:
--
This issue is only new for rcfile type tables.
It started happening with the addition of
property
namehive.default.rcfile.serde/name
valueorg.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe/value
descriptionThe default SerDe hive will use for the rcfile
format/description
/property
to hive-default.xml.template.
see org.apache.hadoop.hive.serde2.SerDeUtils.shouldGetColsFromSerDe()
The long term fix is to make sure that Table.getCols() gets the information
from StorageDescriptor (i.e. metastore) rather than from SerDe.
WebHcat looses column comments and returns 'from deserializer'
--
Key: HIVE-4703
URL: https://issues.apache.org/jira/browse/HIVE-4703
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.12.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Fix For: 0.12.0
Attachments: webhcatMini.sh
This causes Templeton e2e tests to fail.
start WeHcat server (build/dist/hcatalog/sbin/webhcat_server.sh start)
run commands in the attached webhcatMini.sh.
It creates a table with some comments on columns.
When executing describe (GET) on this table, original comments are lost and
are replaced with from deserializer string.
Here is the output of these commands:
localhost:dev ekoifman$ webhcatMini.sh
Running delete test_table ifExists
HTTP/1.1 200 OK
Set-Cookie:
hadoop.auth=u=ekoifmanp=ekoifmant=simplee=1370945567179s=vIBKhGQwzs5pPAY3IkhyPpDkWrY=;Version=1;Path=/;Discard
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(7.6.0.v20120127)
{table:test_table,database:default}
Running create test_table
HTTP/1.1 200 OK
Set-Cookie:
hadoop.auth=u=ekoifmanp=ekoifmant=simplee=1370945569788s=g37NbyyRnf667IciUiIpIQNYGOo=;Version=1;Path=/;Discard
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(7.6.0.v20120127)
{table:test_table,database:default}
Running describe test_table
HTTP/1.1 200 OK
Set-Cookie:
hadoop.auth=u=ekoifmanp=ekoifmant=simplee=1370945572423s=7kE1FOn1Co2JQzZfW0V1myqulw0=;Version=1;Path=/;Discard
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(7.6.0.v20120127)
{columns:[{name:int,comment:from
deserializer,type:string},{name:int2,comment:from
deserializer,type:int}],database:default,table:test_table}
Mon Jun 10 17:12:55 PDT 2013
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4767) ObjectStore.getPMF has concurrency problems
[ https://issues.apache.org/jira/browse/HIVE-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690021#comment-13690021 ] Ashutosh Chauhan commented on HIVE-4767: I see. Lets go ahead with patch, since there indeed is a race condition. +1 Btw, if you can format your testcase of HIVE-4762 as a junit, it will be good to have that checked-in as well. ObjectStore.getPMF has concurrency problems --- Key: HIVE-4767 URL: https://issues.apache.org/jira/browse/HIVE-4767 Project: Hive Issue Type: Sub-task Affects Versions: 0.11.0 Reporter: Brock Noland Assignee: Brock Noland Fix For: 0.12.0 Attachments: HIVE-4767.patch ObjectStore.getPMF is a static factory method but isn't thread safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4448) Fix metastore warehouse incorrect location on Windows in unit tests
[
https://issues.apache.org/jira/browse/HIVE-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13690046#comment-13690046
]
Ashutosh Chauhan commented on HIVE-4448:
You are changing following four strings, which I think is not necessary.
* HiveConf.ConfVars.METASTOREWAREHOUSE : This is a path on hdfs.
C:\\user\hive\warehouse won't make sense here.
* System.getProperty(test.tmp.dir) This is coming from build-common.xml. We
should rather fix it there, by properly using file seprators ${file.separator}
More info: http://ant.apache.org/manual/Types/mapper.html
* System.getProperty(test.src.data.dir) : Same as above.
* HiveConf.ConfVars.SCRATCHDIR : This fixed via HIVE-4609
Fix metastore warehouse incorrect location on Windows in unit tests
---
Key: HIVE-4448
URL: https://issues.apache.org/jira/browse/HIVE-4448
Project: Hive
Issue Type: Bug
Components: Testing Infrastructure
Affects Versions: 0.11.0
Environment: Windows
Reporter: Shuaishuai Nie
Assignee: Shuaishuai Nie
Attachments: HIVE-4448.1.patch
Unit test cases which not using QTestUtil will pass incompatible Windows path
of METASTOREWAREHOUSE to HiveConf which result in creating the
/test/data/warehouse folder in the wrong location in Windows. This folder
will not be deleted at the beginning of the unit test and the content will
cause failure of unit tests if run the same test case repeatedly. The root
cause of this problem is for path like this
pfile://C:\hive\build\ql/test/data/warehouse, the C:\hive\build\ part
will be parsed as authority of the path and removed from the path string. The
patch will fix this problem and make the unit test result consistent between
Windows and Linux.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4448) Fix metastore warehouse incorrect location on Windows in unit tests
[ https://issues.apache.org/jira/browse/HIVE-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4448: --- Status: Open (was: Patch Available) Fix metastore warehouse incorrect location on Windows in unit tests --- Key: HIVE-4448 URL: https://issues.apache.org/jira/browse/HIVE-4448 Project: Hive Issue Type: Bug Components: Testing Infrastructure Affects Versions: 0.11.0 Environment: Windows Reporter: Shuaishuai Nie Assignee: Shuaishuai Nie Attachments: HIVE-4448.1.patch Unit test cases which not using QTestUtil will pass incompatible Windows path of METASTOREWAREHOUSE to HiveConf which result in creating the /test/data/warehouse folder in the wrong location in Windows. This folder will not be deleted at the beginning of the unit test and the content will cause failure of unit tests if run the same test case repeatedly. The root cause of this problem is for path like this pfile://C:\hive\build\ql/test/data/warehouse, the C:\hive\build\ part will be parsed as authority of the path and removed from the path string. The patch will fix this problem and make the unit test result consistent between Windows and Linux. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
