[jira] [Commented] (HIVE-7240) Add classifier for avro-mapred jar
[
https://issues.apache.org/jira/browse/HIVE-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14033559#comment-14033559
]
Hive QA commented on HIVE-7240:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12650628/HIVE-7240.1.patch
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 5536 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_columnar
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/488/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/488/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-488/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12650628
Add classifier for avro-mapred jar
--
Key: HIVE-7240
URL: https://issues.apache.org/jira/browse/HIVE-7240
Project: Hive
Issue Type: Task
Components: File Formats
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
Attachments: HIVE-7240.1.patch, HIVE-7240.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7229) String is compared using equal in HiveMetaStore#HMSHandler#init()
[
https://issues.apache.org/jira/browse/HIVE-7229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kang updated HIVE-7229:
---
Attachment: HIVE-7229
Change string compare operator
String is compared using equal in HiveMetaStore#HMSHandler#init()
-
Key: HIVE-7229
URL: https://issues.apache.org/jira/browse/HIVE-7229
Project: Hive
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Attachments: HIVE-7229
Around line 423:
{code}
if (partitionValidationRegex != null partitionValidationRegex != )
{
partitionValidationPattern =
Pattern.compile(partitionValidationRegex);
{code}
partitionValidationRegex.isEmpty() can be used instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7229) String is compared using equal in HiveMetaStore#HMSHandler#init()
[
https://issues.apache.org/jira/browse/HIVE-7229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kang updated HIVE-7229:
---
Status: Patch Available (was: Open)
String is compared using equal in HiveMetaStore#HMSHandler#init()
-
Key: HIVE-7229
URL: https://issues.apache.org/jira/browse/HIVE-7229
Project: Hive
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Attachments: HIVE-7229
Around line 423:
{code}
if (partitionValidationRegex != null partitionValidationRegex != )
{
partitionValidationPattern =
Pattern.compile(partitionValidationRegex);
{code}
partitionValidationRegex.isEmpty() can be used instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6584) Add HiveHBaseTableSnapshotInputFormat
[
https://issues.apache.org/jira/browse/HIVE-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14033591#comment-14033591
]
zjkyly commented on HIVE-6584:
--
Teng YuTong and I are colleagues. we have a patch for HIVE-6584 and a patch for
HBASE-11163 , and we modify
org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat（line 93）
from: static class TableSnapshotRegionSplit extends InputSplit implements
Writable
to: public static class TableSnapshotRegionSplit extends InputSplit implements
Writable
we can run mapred on snapshot. mapred (count(1)) result :
2014-06-17 16:29:34,540 Stage-1 map = 100%, reduce = 32%, Cumulative CPU
2467.57 sec
2014-06-17 16:29:35,578 Stage-1 map = 100%, reduce = 100%, Cumulative CPU
2468.35 sec
MapReduce Total cumulative CPU time: 41 minutes 8 seconds 350 msec
Ended Job = job_1402970116480_0015
MapReduce Jobs Launched:
Job 0: Map: 64 Reduce: 1 Cumulative CPU: 2468.35 sec HDFS Read: 18334 HDFS
Write: 9 SUCCESS
Total MapReduce CPU Time Spent: 41 minutes 8 seconds 350 msec
OK
65497163
Time taken: 429.647 seconds, Fetched: 1 row(s)
hbase count result:
Current count: 6540, row: user987684650651905350
65497163 row(s) in 1446.2310 seconds
= 65497163
but hfile has different versions of the record. We can not solve this problem.
So, we set the version of hbase table is 1, and run major compact before
snapshot table.
Add HiveHBaseTableSnapshotInputFormat
-
Key: HIVE-6584
URL: https://issues.apache.org/jira/browse/HIVE-6584
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Fix For: 0.14.0
Attachments: HIVE-6584.0.patch, HIVE-6584.1.patch, HIVE-6584.2.patch,
HIVE-6584.3.patch, HIVE-6584.4.patch
HBASE-8369 provided mapreduce support for reading from HBase table snapsopts.
This allows a MR job to consume a stable, read-only view of an HBase table
directly off of HDFS. Bypassing the online region server API provides a nice
performance boost for the full scan. HBASE-10642 is backporting that feature
to 0.94/0.96 and also adding a {{mapred}} implementation. Once that's
available, we should add an input format. A follow-on patch could work out
how to integrate this functionality into the StorageHandler, similar to how
HIVE-6473 integrates the HFileOutputFormat into existing table definitions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6584) Add HiveHBaseTableSnapshotInputFormat
[
https://issues.apache.org/jira/browse/HIVE-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14033626#comment-14033626
]
zjkyly commented on HIVE-6584:
--
hi Nick Dimiduk and Teng Yutong. what we can do at repsent is to scan all kv
records of snapshot.
we weren't able to solve the issue that hfile had different version for record,
so we set default version of hbase table as 1, and run major compact before
doing snapshot table.
we don't know the open soure community, is it necessary to solve the problem of
same kv version? shall we find all kv version ? or just find the latest one?
We will try to solve the problem of multiple versions.
Add HiveHBaseTableSnapshotInputFormat
-
Key: HIVE-6584
URL: https://issues.apache.org/jira/browse/HIVE-6584
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Fix For: 0.14.0
Attachments: HIVE-6584.0.patch, HIVE-6584.1.patch, HIVE-6584.2.patch,
HIVE-6584.3.patch, HIVE-6584.4.patch
HBASE-8369 provided mapreduce support for reading from HBase table snapsopts.
This allows a MR job to consume a stable, read-only view of an HBase table
directly off of HDFS. Bypassing the online region server API provides a nice
performance boost for the full scan. HBASE-10642 is backporting that feature
to 0.94/0.96 and also adding a {{mapred}} implementation. Once that's
available, we should add an input format. A follow-on patch could work out
how to integrate this functionality into the StorageHandler, similar to how
HIVE-6473 integrates the HFileOutputFormat into existing table definitions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7225) Unclosed Statement's in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
steve, Oh updated HIVE-7225:
Attachment: HIVE-7225.1.patch
Unclosed Statement's in TxnHandler
--
Key: HIVE-7225
URL: https://issues.apache.org/jira/browse/HIVE-7225
Project: Hive
Issue Type: Bug
Reporter: Ted Yu
Assignee: steve, Oh
Attachments: HIVE-7225.1.patch
There're several methods in TxnHandler where Statement (local to the method)
is not closed upon return.
Here're a few examples:
In compact():
{code}
stmt.executeUpdate(s);
LOG.debug(Going to commit);
dbConn.commit();
{code}
In showCompact():
{code}
Statement stmt = dbConn.createStatement();
String s = select cq_database, cq_table, cq_partition, cq_state,
cq_type, cq_worker_id, +
cq_start, cq_run_as from COMPACTION_QUEUE;
LOG.debug(Going to execute query + s + );
ResultSet rs = stmt.executeQuery(s);
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7225) Unclosed Statement's in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
steve, Oh updated HIVE-7225:
Status: Patch Available (was: Open)
Unclosed Statement's in TxnHandler
--
Key: HIVE-7225
URL: https://issues.apache.org/jira/browse/HIVE-7225
Project: Hive
Issue Type: Bug
Reporter: Ted Yu
Assignee: steve, Oh
Attachments: HIVE-7225.1.patch
There're several methods in TxnHandler where Statement (local to the method)
is not closed upon return.
Here're a few examples:
In compact():
{code}
stmt.executeUpdate(s);
LOG.debug(Going to commit);
dbConn.commit();
{code}
In showCompact():
{code}
Statement stmt = dbConn.createStatement();
String s = select cq_database, cq_table, cq_partition, cq_state,
cq_type, cq_worker_id, +
cq_start, cq_run_as from COMPACTION_QUEUE;
LOG.debug(Going to execute query + s + );
ResultSet rs = stmt.executeQuery(s);
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Raw MapReduce code read RC file using HCatalog
ps: I use Hive 0.12 release and Hadoop 1.2.1 release.
On Jun 17, 2014, at 5:07 PM, Fastupload fastupl...@outlook.com wrote:
Hi,
When MapReduce code read RC file using HCatalog, and hive meta store in a
remote Oracle database. I write a demo code following by the wiki page,
https://cwiki.apache.org/confluence/display/Hive/HCatalog+InputOutput, and
package all dependence jar into one jar.
The job got the error while running. some line of failure stack is:
Caused by: org.datanucleus.exceptions.NucleusUserException: Persistence
process has been specified to use a ClassLoaderResolver of name datanucleus
yet this has not been found by the DataNucleus plugin mechanism. Please check
your CLASSPATH and plugin specification.
It seems that the HCatInputFormat class can not create a
JDOPersisitencemanagerFactory object for HiveMetaStoreClient object in
InitializeInput.java from line 101 to 106,
if (conf != null) {
hiveConf = HCatUtil.getHiveConf(conf);
} else {
hiveConf = new HiveConf(HCatInputFormat.class);
}
client = HCatUtil.getHiveClient(hiveConf);
The lines of code create a HiveMetaStoreClient object with job configuration
or HCatInputFormat properties. So I add code to add hive-site.xml file both
job configuration and HCatInputFormat properties. as,
// load hive meta store configuration file, both properties and
job config
Properties prop = new Properties();
FileInputStream confStream = new FileInputStream(args[4]);
prop.load(confStream);
conf.addResource(confStream);
HCatInputFormat.setInput(job, dbName,
tblName).setFilter(filter).setProperties(prop);
But the job still get the same error. any idea?
More error log and code please look at the two attachments.
error log.txt
code fragment.txt
Best Regards,
Link Qian
[jira] [Updated] (HIVE-7225) Unclosed Statement's in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-7225:
-
Status: Open (was: Patch Available)
Steve, thanks for the patch. A few comments:
I don't think there's a need to explicitly close the ResultSet. According to
the JavaDocs on Statement.close(), it closes any associated ResultSets. So
making this call seems like a waste of time.
If we're going to do this, it should be done in CompactionTxnHanlder as well,
since that subclasses this class and uses all of the same methods.
But that if leads to my last comment. Is this necessary at all? All of the
public methods close the db connection as soon as they are done. Is there any
value to closing the individual statements in that collection first?
Unclosed Statement's in TxnHandler
--
Key: HIVE-7225
URL: https://issues.apache.org/jira/browse/HIVE-7225
Project: Hive
Issue Type: Bug
Reporter: Ted Yu
Assignee: steve, Oh
Attachments: HIVE-7225.1.patch
There're several methods in TxnHandler where Statement (local to the method)
is not closed upon return.
Here're a few examples:
In compact():
{code}
stmt.executeUpdate(s);
LOG.debug(Going to commit);
dbConn.commit();
{code}
In showCompact():
{code}
Statement stmt = dbConn.createStatement();
String s = select cq_database, cq_table, cq_partition, cq_state,
cq_type, cq_worker_id, +
cq_start, cq_run_as from COMPACTION_QUEUE;
LOG.debug(Going to execute query + s + );
ResultSet rs = stmt.executeQuery(s);
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Question about org.apache.hive.service.cli.Column class
Hi I want to implement some compression algorithms on the data that is exchanged between clients and HS2. Just wanted to make sure that the Get function is called on the client side, Right? Cheers
[jira] [Commented] (HIVE-7219) Improve performance of serialization utils in ORC
[
https://issues.apache.org/jira/browse/HIVE-7219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034100#comment-14034100
]
Hive QA commented on HIVE-7219:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12650656/HIVE-7219.4.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 5653 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_columnar
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/491/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/491/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-491/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12650656
Improve performance of serialization utils in ORC
-
Key: HIVE-7219
URL: https://issues.apache.org/jira/browse/HIVE-7219
Project: Hive
Issue Type: Improvement
Components: File Formats
Affects Versions: 0.14.0
Reporter: Prasanth J
Assignee: Prasanth J
Attachments: HIVE-7219.1.patch, HIVE-7219.2.patch, HIVE-7219.3.patch,
HIVE-7219.4.patch, orc-read-perf-jmh-benchmark.png
ORC uses serialization utils heavily for reading and writing data. The
bitpacking and unpacking code in writeInts() and readInts() can be unrolled
for better performance. Also double reader/writer performance can be improved
by bulk reading/writing from/to byte array.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7219) Improve performance of serialization utils in ORC
[ https://issues.apache.org/jira/browse/HIVE-7219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-7219: - Resolution: Fixed Fix Version/s: 0.14.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks [~hagleitn] and [~gopalv] for the code review. Improve performance of serialization utils in ORC - Key: HIVE-7219 URL: https://issues.apache.org/jira/browse/HIVE-7219 Project: Hive Issue Type: Improvement Components: File Formats Affects Versions: 0.14.0 Reporter: Prasanth J Assignee: Prasanth J Fix For: 0.14.0 Attachments: HIVE-7219.1.patch, HIVE-7219.2.patch, HIVE-7219.3.patch, HIVE-7219.4.patch, orc-read-perf-jmh-benchmark.png ORC uses serialization utils heavily for reading and writing data. The bitpacking and unpacking code in writeInts() and readInts() can be unrolled for better performance. Also double reader/writer performance can be improved by bulk reading/writing from/to byte array. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7225) Unclosed Statement's in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034148#comment-14034148
]
Ted Yu commented on HIVE-7225:
--
Does this answer your question ?
http://stackoverflow.com/questions/2708689/impact-of-java-sql-connection-close-on-java-sql-statement-objects-and-the-like
Unclosed Statement's in TxnHandler
--
Key: HIVE-7225
URL: https://issues.apache.org/jira/browse/HIVE-7225
Project: Hive
Issue Type: Bug
Reporter: Ted Yu
Assignee: steve, Oh
Attachments: HIVE-7225.1.patch
There're several methods in TxnHandler where Statement (local to the method)
is not closed upon return.
Here're a few examples:
In compact():
{code}
stmt.executeUpdate(s);
LOG.debug(Going to commit);
dbConn.commit();
{code}
In showCompact():
{code}
Statement stmt = dbConn.createStatement();
String s = select cq_database, cq_table, cq_partition, cq_state,
cq_type, cq_worker_id, +
cq_start, cq_run_as from COMPACTION_QUEUE;
LOG.debug(Going to execute query + s + );
ResultSet rs = stmt.executeQuery(s);
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7225) Unclosed Statement's in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034192#comment-14034192
]
Alan Gates commented on HIVE-7225:
--
No. We aren't leaving these connections open for a long time. We're closing
them quite quickly at the end of each Thrift RPC. If we were leaving them open
for a long time then I would agree that it was important to close the
statements.
Unclosed Statement's in TxnHandler
--
Key: HIVE-7225
URL: https://issues.apache.org/jira/browse/HIVE-7225
Project: Hive
Issue Type: Bug
Reporter: Ted Yu
Assignee: steve, Oh
Attachments: HIVE-7225.1.patch
There're several methods in TxnHandler where Statement (local to the method)
is not closed upon return.
Here're a few examples:
In compact():
{code}
stmt.executeUpdate(s);
LOG.debug(Going to commit);
dbConn.commit();
{code}
In showCompact():
{code}
Statement stmt = dbConn.createStatement();
String s = select cq_database, cq_table, cq_partition, cq_state,
cq_type, cq_worker_id, +
cq_start, cq_run_as from COMPACTION_QUEUE;
LOG.debug(Going to execute query + s + );
ResultSet rs = stmt.executeQuery(s);
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HIVE-7246) Hive transaction manager hardwires bonecp as the JDBC pooling implementation
Alan Gates created HIVE-7246: Summary: Hive transaction manager hardwires bonecp as the JDBC pooling implementation Key: HIVE-7246 URL: https://issues.apache.org/jira/browse/HIVE-7246 Project: Hive Issue Type: Bug Components: Transactions Affects Versions: 0.13.0 Reporter: Alan Gates Assignee: Alan Gates Currently TxnManager hardwires BoneCP as the JDBC connection pooling implementation. Instead it should use the same connection pooling that the metastore does. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-7094:
-
Status: Open (was: Patch Available)
Looks like some tests failed. [~davidzchen], can you please take a look? Thanks.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7076) Plugin (exec hook) to log to application timeline data to Yarn
[ https://issues.apache.org/jira/browse/HIVE-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-7076: - Status: Open (was: Patch Available) Plugin (exec hook) to log to application timeline data to Yarn -- Key: HIVE-7076 URL: https://issues.apache.org/jira/browse/HIVE-7076 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7076.1.patch See: https://issues.apache.org/jira/browse/YARN-1530 This is a simple pre/post exec hook to log query + plan information to yarn. This information can be used to build tools and UIs to monitor, track, debug and tune Hive queries. Off by default, but can be enabled via: hive.exec.pre.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.post.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.failure.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7076) Plugin (exec hook) to log to application timeline data to Yarn
[ https://issues.apache.org/jira/browse/HIVE-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-7076: - Status: Patch Available (was: Open) Plugin (exec hook) to log to application timeline data to Yarn -- Key: HIVE-7076 URL: https://issues.apache.org/jira/browse/HIVE-7076 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7076.1.patch, HIVE-7076.2.patch See: https://issues.apache.org/jira/browse/YARN-1530 This is a simple pre/post exec hook to log query + plan information to yarn. This information can be used to build tools and UIs to monitor, track, debug and tune Hive queries. Off by default, but can be enabled via: hive.exec.pre.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.post.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.failure.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7076) Plugin (exec hook) to log to application timeline data to Yarn
[ https://issues.apache.org/jira/browse/HIVE-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-7076: - Attachment: HIVE-7076.2.patch Different route. Patch .2 simply excludes the ATSHook from compilation on hadoop-1 profile Plugin (exec hook) to log to application timeline data to Yarn -- Key: HIVE-7076 URL: https://issues.apache.org/jira/browse/HIVE-7076 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7076.1.patch, HIVE-7076.2.patch See: https://issues.apache.org/jira/browse/YARN-1530 This is a simple pre/post exec hook to log query + plan information to yarn. This information can be used to build tools and UIs to monitor, track, debug and tune Hive queries. Off by default, but can be enabled via: hive.exec.pre.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.post.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.failure.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7236) Tez progress monitor should indicate running/failed tasks
[
https://issues.apache.org/jira/browse/HIVE-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-7236:
--
Release Note: Log Tez running/failed tasks in TezJobMonitor
Status: Patch Available (was: In Progress)
Tez progress monitor should indicate running/failed tasks
-
Key: HIVE-7236
URL: https://issues.apache.org/jira/browse/HIVE-7236
Project: Hive
Issue Type: Bug
Components: Tez
Affects Versions: 0.14.0
Reporter: Gopal V
Assignee: Gopal V
Priority: Minor
Attachments: HIVE-7236.1.patch
Currently, the only logging in TezJobMonitor is for completed tasks.
This makes it hard to locate task stalls and task failures. Failure scenarios
are harder to debug, in particular when analyzing query runs on a cluster
with bad nodes.
Change the job monitor to log running failed tasks as follows.
{code}
Map 1: 0(+157,-1)/1755 Reducer 2: 0/1
Map 1: 0(+168,-1)/1755 Reducer 2: 0/1
Map 1: 0(+189,-1)/1755 Reducer 2: 0/1
Map 1: 0(+189,-1)/1755 Reducer 2: 0/1
{code}
That is 189 tasks running, 1 failure and 0 complete.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7119) Extended ACL's should be inherited if warehouse perm inheritance enabled
[ https://issues.apache.org/jira/browse/HIVE-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034293#comment-14034293 ] Gunther Hagleitner commented on HIVE-7119: -- After this patch itests cannot be compiled against hadoop-1 profile anymore. Could you take a look? $cd itests $mvn install -DskipTests -Phadoop-1 ... [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project hive-it-unit: Compilation failure: Compilation failure: [ERROR] /Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java:[31,39] cannot find symbol [ERROR] symbol : class AclStatus [ERROR] location: package org.apache.hadoop.fs.permission [ERROR] /Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[20,46] cannot find symbol [ERROR] symbol : class AclEntryScope [ERROR] location: package org.apache.hadoop.fs.permission [ERROR] /Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[20,1] static import only from classes and interfaces [ERROR] /Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[21,46] cannot find symbol [ERROR] symbol : class AclEntryType [ERROR] location: package org.apache.hadoop.fs.permission [ERROR] /Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[21,1] static import only from classes and interfaces [ERROR] /Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[22,46] cannot find symbol [ERROR] symbol : class AclEntryType ... Extended ACL's should be inherited if warehouse perm inheritance enabled Key: HIVE-7119 URL: https://issues.apache.org/jira/browse/HIVE-7119 Project: Hive Issue Type: Bug Reporter: Szehon Ho Assignee: Szehon Ho Labels: TODOC14 Fix For: 0.14.0 Attachments: HIVE-7119.2.patch, HIVE-7119.3.patch, HIVE-7119.4.patch, HIVE-7119.patch HDFS recently came out with support for extended ACL's, ie permission for specific group/user in addition to the general owner/group/other permission. Hive permission inheritance should also inherit those as well, if user has set them at any point in the warehouse directory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7105) Enable ReduceRecordProcessor to generate VectorizedRowBatches
[ https://issues.apache.org/jira/browse/HIVE-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034320#comment-14034320 ] Jitendra Nath Pandey commented on HIVE-7105: [~rusanu] Here is the RB link: https://reviews.apache.org/r/22540/ Enable ReduceRecordProcessor to generate VectorizedRowBatches - Key: HIVE-7105 URL: https://issues.apache.org/jira/browse/HIVE-7105 Project: Hive Issue Type: Bug Components: Tez, Vectorization Reporter: Rajesh Balamohan Assignee: Gopal V Fix For: 0.14.0 Attachments: HIVE-7105.1.patch, HIVE-7105.2.patch Currently, ReduceRecordProcessor sends one key,value pair at a time to its operator pipeline. It would be beneficial to send VectorizedRowBatch to downstream operators. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7232) ReduceSink is emitting NULL keys due to failed keyEval
[
https://issues.apache.org/jira/browse/HIVE-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034322#comment-14034322
]
Gopal V commented on HIVE-7232:
---
[~navis]: I found out that there are indeed o_orderkey entries which show up as
214800 in text, which lies outside the range of the TPC-H Identifier column
spec.
I will reload the data using bigint for o_orderkey soon.
But I still want to locate and confirm the different results between MR and Tez
here.
ReduceSink is emitting NULL keys due to failed keyEval
--
Key: HIVE-7232
URL: https://issues.apache.org/jira/browse/HIVE-7232
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.14.0
Reporter: Gopal V
Assignee: Navis
Attachments: HIVE-7232-extra-logging.patch, q5.explain.txt, q5.sql
After HIVE-4867 has been merged in, some queries have exhibited a very weird
skew towards NULL keys emitted from the ReduceSinkOperator.
Added extra logging to print expr.column() in ExprNodeColumnEvaluator in
reduce sink.
{code}
2014-06-14 00:37:19,186 INFO [TezChild]
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator:
numDistributionKeys = 1 {null -- ExprNodeColumnEvaluator(_col10)}
key_row={reducesinkkey0:442}
{code}
{code}
HiveKey firstKey = toHiveKey(cachedKeys[0], tag, null);
int distKeyLength = firstKey.getDistKeyLength();
if(distKeyLength = 1) {
StringBuffer x1 = new StringBuffer();
x1.append(numDistributionKeys = + numDistributionKeys + \n);
for (int i = 0; i numDistributionKeys; i++) {
x1.append(cachedKeys[0][i] + -- + keyEval[i] + \n);
}
x1.append(key_row=+ SerDeUtils.getJSONString(row,
keyObjectInspector));
LOG.info(GOPAL: + x1.toString());
}
{code}
The query is tpc-h query5, with extra NULL checks just to be sure.
{code}
ELECT n_name,
sum(l_extendedprice * (1 - l_discount)) AS revenue
FROM customer,
orders,
lineitem,
supplier,
nation,
region
WHERE c_custkey = o_custkey
AND l_orderkey = o_orderkey
AND l_suppkey = s_suppkey
AND c_nationkey = s_nationkey
AND s_nationkey = n_nationkey
AND n_regionkey = r_regionkey
AND r_name = 'ASIA'
AND o_orderdate = '1994-01-01'
AND o_orderdate '1995-01-01'
and l_orderkey is not null
and c_custkey is not null
and l_suppkey is not null
and c_nationkey is not null
and s_nationkey is not null
and n_regionkey is not null
GROUP BY n_name
ORDER BY revenue DESC;
{code}
The reducer which has the issue has the following plan
{code}
Reducer 3
Reduce Operator Tree:
Join Operator
condition map:
Inner Join 0 to 1
condition expressions:
0 {KEY.reducesinkkey0} {VALUE._col2}
1 {VALUE._col0} {KEY.reducesinkkey0} {VALUE._col3}
outputColumnNames: _col0, _col3, _col10, _col11, _col14
Statistics: Num rows: 18344 Data size: 95229140992 Basic
stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col10 (type: int)
sort order: +
Map-reduce partition columns: _col10 (type: int)
Statistics: Num rows: 18344 Data size: 95229140992
Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int), _col3 (type: int),
_col11 (type: int), _col14 (type: string)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7076) Plugin (exec hook) to log to application timeline data to Yarn
[ https://issues.apache.org/jira/browse/HIVE-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-7076: - Status: Open (was: Patch Available) Plugin (exec hook) to log to application timeline data to Yarn -- Key: HIVE-7076 URL: https://issues.apache.org/jira/browse/HIVE-7076 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7076.1.patch, HIVE-7076.2.patch See: https://issues.apache.org/jira/browse/YARN-1530 This is a simple pre/post exec hook to log query + plan information to yarn. This information can be used to build tools and UIs to monitor, track, debug and tune Hive queries. Off by default, but can be enabled via: hive.exec.pre.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.post.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.failure.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7076) Plugin (exec hook) to log to application timeline data to Yarn
[ https://issues.apache.org/jira/browse/HIVE-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-7076: - Status: Patch Available (was: Open) Plugin (exec hook) to log to application timeline data to Yarn -- Key: HIVE-7076 URL: https://issues.apache.org/jira/browse/HIVE-7076 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7076.1.patch, HIVE-7076.2.patch, HIVE-7076.3.patch See: https://issues.apache.org/jira/browse/YARN-1530 This is a simple pre/post exec hook to log query + plan information to yarn. This information can be used to build tools and UIs to monitor, track, debug and tune Hive queries. Off by default, but can be enabled via: hive.exec.pre.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.post.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.failure.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7076) Plugin (exec hook) to log to application timeline data to Yarn
[ https://issues.apache.org/jira/browse/HIVE-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034338#comment-14034338 ] Gunther Hagleitner commented on HIVE-7076: -- .3 addresses review comments. Plugin (exec hook) to log to application timeline data to Yarn -- Key: HIVE-7076 URL: https://issues.apache.org/jira/browse/HIVE-7076 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7076.1.patch, HIVE-7076.2.patch, HIVE-7076.3.patch See: https://issues.apache.org/jira/browse/YARN-1530 This is a simple pre/post exec hook to log query + plan information to yarn. This information can be used to build tools and UIs to monitor, track, debug and tune Hive queries. Off by default, but can be enabled via: hive.exec.pre.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.post.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.failure.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7076) Plugin (exec hook) to log to application timeline data to Yarn
[ https://issues.apache.org/jira/browse/HIVE-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-7076: - Attachment: HIVE-7076.3.patch Plugin (exec hook) to log to application timeline data to Yarn -- Key: HIVE-7076 URL: https://issues.apache.org/jira/browse/HIVE-7076 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7076.1.patch, HIVE-7076.2.patch, HIVE-7076.3.patch See: https://issues.apache.org/jira/browse/YARN-1530 This is a simple pre/post exec hook to log query + plan information to yarn. This information can be used to build tools and UIs to monitor, track, debug and tune Hive queries. Off by default, but can be enabled via: hive.exec.pre.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.post.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.failure.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7190) WebHCat launcher task failure can cause two concurent user jobs to run
[ https://issues.apache.org/jira/browse/HIVE-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-7190: Resolution: Fixed Fix Version/s: 0.14.0 Status: Resolved (was: Patch Available) Patch committed to trunk. Thanks for the contribution Ivan, thanks for the review Eugene! WebHCat launcher task failure can cause two concurent user jobs to run -- Key: HIVE-7190 URL: https://issues.apache.org/jira/browse/HIVE-7190 Project: Hive Issue Type: Bug Components: WebHCat Affects Versions: 0.13.0 Reporter: Ivan Mitic Assignee: Ivan Mitic Fix For: 0.14.0 Attachments: HIVE-7190.2.patch, HIVE-7190.3.patch, HIVE-7190.patch Templeton uses launcher jobs to launch the actual user jobs. Launcher jobs are 1-map jobs (a single task jobs) which kick off the actual user job and monitor it until it finishes. Given that the launcher is a task, like any other MR task, it has a retry policy in case it fails (due to a task crash, tasktracker/nodemanager crash, machine level outage, etc.). Further, when launcher task is retried, it will again launch the same user job, *however* the previous attempt user job is already running. What this means is that we can have two identical user jobs running in parallel. In case of MRv2, there will be an MRAppMaster and the launcher task, which are subject to failure. In case any of the two fails, another instance of a user job will be launched again in parallel. Above situation is already a bug. Now going further to RM HA, what RM does on failover/restart is that it kills all containers, and it restarts all applications. This means that if our customer had 10 jobs on the cluster (this is 10 launcher jobs and 10 user jobs), on RM failover, all 20 jobs will be restarted, and launcher jobs will queue user jobs again. There are two issues with this design: 1. There are *possible* chances for corruption of job outputs (it would be useful to analyze this scenario more and confirm this statement). 2. Cluster resources are spent on jobs redundantly To address the issue at least on Yarn (Hadoop 2.0) clusters, webhcat should do the same thing Oozie does in this scenario, and that is to tag all its child jobs with an id, and kill those jobs on task restart before they are kicked off again. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7225) Unclosed Statement's in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034400#comment-14034400
]
Ted Yu commented on HIVE-7225:
--
Hive depends on BoneCP 0.8.0
I cloned BoneCP locally and looked at their code.
{code}
public class ConnectionHandle implements Connection,Serializable{
{code}
In ConnectionHandle ctor, I see:
{code}
this.detectUnclosedStatements =
pool.getConfig().isDetectUnclosedStatements();
this.closeOpenStatements = pool.getConfig().isCloseOpenStatements();
{code}
In close() method:
{code}
if (this.closeOpenStatements){
for (EntryStatement, String statementEntry:
this.trackedStatement.entrySet()){
statementEntry.getKey().close();
if (this.detectUnclosedStatements){
logger.warn(String.format(UNCLOSED_LOG_ERROR_MESSAGE,
statementEntry.getValue()));
}
}
this.trackedStatement.clear();
}
{code}
In Hive codebase, I don't see call that turns on closeOpenStatements.
Unclosed Statement's in TxnHandler
--
Key: HIVE-7225
URL: https://issues.apache.org/jira/browse/HIVE-7225
Project: Hive
Issue Type: Bug
Reporter: Ted Yu
Assignee: steve, Oh
Attachments: HIVE-7225.1.patch
There're several methods in TxnHandler where Statement (local to the method)
is not closed upon return.
Here're a few examples:
In compact():
{code}
stmt.executeUpdate(s);
LOG.debug(Going to commit);
dbConn.commit();
{code}
In showCompact():
{code}
Statement stmt = dbConn.createStatement();
String s = select cq_database, cq_table, cq_partition, cq_state,
cq_type, cq_worker_id, +
cq_start, cq_run_as from COMPACTION_QUEUE;
LOG.debug(Going to execute query + s + );
ResultSet rs = stmt.executeQuery(s);
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7231) Improve ORC padding
[
https://issues.apache.org/jira/browse/HIVE-7231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034409#comment-14034409
]
Hive QA commented on HIVE-7231:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12650660/HIVE-7231.2.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 5536 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_columnar
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/492/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/492/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-492/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12650660
Improve ORC padding
---
Key: HIVE-7231
URL: https://issues.apache.org/jira/browse/HIVE-7231
Project: Hive
Issue Type: Improvement
Components: File Formats
Affects Versions: 0.14.0
Reporter: Prasanth J
Assignee: Prasanth J
Labels: orcfile
Attachments: HIVE-7231.1.patch, HIVE-7231.2.patch
Current ORC padding is not optimal because of fixed stripe sizes within
block. The padding overhead will be significant in some cases. Also padding
percentage relative to stripe size is not configurable.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034418#comment-14034418
]
David Chen commented on HIVE-7094:
--
The Hive CLI and QL test failures are unrelated to my patch.
The only one that is concerning to me is the TestOrcHCatLoader failure.
However, I am not hitting that failure when I run the tests on my machine, but
I have rebased my changes on the current trunk. Nonetheless, all of the
HCatalog Core tests pass.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Chen updated HIVE-7094:
-
Attachment: HIVE-7094.5.patch
Attaching a new patch rebased on trunk.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch,
HIVE-7094.5.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7159) For inner joins push a 'is not null predicate' to the join sources for every non nullSafe join condition
[ https://issues.apache.org/jira/browse/HIVE-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Butani updated HIVE-7159: Attachment: HIVE-7159.7.patch For inner joins push a 'is not null predicate' to the join sources for every non nullSafe join condition Key: HIVE-7159 URL: https://issues.apache.org/jira/browse/HIVE-7159 Project: Hive Issue Type: Bug Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-7159.1.patch, HIVE-7159.2.patch, HIVE-7159.3.patch, HIVE-7159.4.patch, HIVE-7159.5.patch, HIVE-7159.6.patch, HIVE-7159.7.patch A join B on A.x = B.y can be transformed to (A where x is not null) join (B where y is not null) on A.x = B.y Apart from avoiding shuffling null keyed rows it also avoids issues with reduce-side skew when there are a lot of null values in the data. Thanks to [~gopalv] for the analysis and coming up with the solution. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Chen updated HIVE-7094:
-
Status: Patch Available (was: Open)
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch,
HIVE-7094.5.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7242) alter table drop partition is acquiring the wrong type of lock
[
https://issues.apache.org/jira/browse/HIVE-7242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034420#comment-14034420
]
Hive QA commented on HIVE-7242:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12650664/HIVE-7242.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/493/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/493/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-493/
Messages:
{noformat}
This message was trimmed, see log for full details
Decision can match input such as LPAREN KW_CASE KW_ARRAY using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4:
Decision can match input such as LPAREN KW_CASE TinyintLiteral using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4:
Decision can match input such as LPAREN KW_CASE KW_STRUCT using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4:
Decision can match input such as LPAREN KW_CASE SmallintLiteral using
multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:115:5:
Decision can match input such as KW_CLUSTER KW_BY LPAREN using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:127:5:
Decision can match input such as KW_PARTITION KW_BY LPAREN using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:138:5:
Decision can match input such as KW_DISTRIBUTE KW_BY LPAREN using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:149:5:
Decision can match input such as KW_SORT KW_BY LPAREN using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:166:7:
Decision can match input such as STAR using multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:179:5:
Decision can match input such as KW_STRUCT using multiple alternatives: 4, 6
As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5:
Decision can match input such as KW_UNIONTYPE using multiple alternatives: 5,
6
As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5:
Decision can match input such as KW_ARRAY using multiple alternatives: 2, 6
As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:261:5:
Decision can match input such as KW_DATE StringLiteral using multiple
alternatives: 2, 3
As a result, alternative(s) 3 were disabled for that input
warning(200): IdentifiersParser.g:261:5:
Decision can match input such as KW_FALSE using multiple alternatives: 3, 8
As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5:
Decision can match input such as KW_TRUE using multiple alternatives: 3, 8
As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5:
Decision can match input such as KW_NULL using multiple alternatives: 1, 8
As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT
KW_OVERWRITE using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE
KW_BY using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_MAP LPAREN
using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT
KW_INTO using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL
KW_VIEW using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_GROUP
[jira] [Commented] (HIVE-7230) Add Eclipse formatter file for Hive coding conventions
[ https://issues.apache.org/jira/browse/HIVE-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034421#comment-14034421 ] Swarnim Kulkarni commented on HIVE-7230: Go for it! The only comment I had was it would probably be nice to pull down the style guide xml file locally so that our builds don't fail in case the remote server goes down. Add Eclipse formatter file for Hive coding conventions -- Key: HIVE-7230 URL: https://issues.apache.org/jira/browse/HIVE-7230 Project: Hive Issue Type: Improvement Reporter: David Chen Assignee: David Chen Attachments: HIVE-7230.1.patch Eclipse's formatter is a convenient way to clean up formatting for Java code. Currently, there is no Eclipse formatter file checked into Hive's codebase. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7159) For inner joins push a 'is not null predicate' to the join sources for every non nullSafe join condition
[ https://issues.apache.org/jira/browse/HIVE-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034426#comment-14034426 ] Harish Butani commented on HIVE-7159: - The diff in windowing queries is because of an issue with ColumnPruner for PTF and Script Operators. [~ashutoshc] can you please look at changes to ColumnPruner. For inner joins push a 'is not null predicate' to the join sources for every non nullSafe join condition Key: HIVE-7159 URL: https://issues.apache.org/jira/browse/HIVE-7159 Project: Hive Issue Type: Bug Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-7159.1.patch, HIVE-7159.2.patch, HIVE-7159.3.patch, HIVE-7159.4.patch, HIVE-7159.5.patch, HIVE-7159.6.patch, HIVE-7159.7.patch A join B on A.x = B.y can be transformed to (A where x is not null) join (B where y is not null) on A.x = B.y Apart from avoiding shuffling null keyed rows it also avoids issues with reduce-side skew when there are a lot of null values in the data. Thanks to [~gopalv] for the analysis and coming up with the solution. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7231) Improve ORC padding
[ https://issues.apache.org/jira/browse/HIVE-7231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034434#comment-14034434 ] Gopal V commented on HIVE-7231: --- The approach results in stray writes across the stripe boundaries. I think this approach needs to be revisited to disconnect the HDFS block size from the ORC stripe size. The stripe size needs to be a factor of the HDFS block size, but the fraction should not remain at 0.5x. Improve ORC padding --- Key: HIVE-7231 URL: https://issues.apache.org/jira/browse/HIVE-7231 Project: Hive Issue Type: Improvement Components: File Formats Affects Versions: 0.14.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile Attachments: HIVE-7231.1.patch, HIVE-7231.2.patch Current ORC padding is not optimal because of fixed stripe sizes within block. The padding overhead will be significant in some cases. Also padding percentage relative to stripe size is not configurable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034436#comment-14034436
]
Sushanth Sowmyan commented on HIVE-7094:
Some of the HCat tests are flaky when run on the ptest framework (they succeed
when run on a serial run) - that's on my tofix radar. But in the meanwhile,
I'll try running the HCat tests with David's patch to verify.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch,
HIVE-7094.5.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7230) Add Eclipse formatter file for Hive coding conventions
[ https://issues.apache.org/jira/browse/HIVE-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034454#comment-14034454 ] David Chen commented on HIVE-7230: -- I am going to try to point the Maven Eclipse plugin to the local eclipse-styles.xml that I checked in with this patch. The code snipped above seems to point to the Google Java coding style. Add Eclipse formatter file for Hive coding conventions -- Key: HIVE-7230 URL: https://issues.apache.org/jira/browse/HIVE-7230 Project: Hive Issue Type: Improvement Reporter: David Chen Assignee: David Chen Attachments: HIVE-7230.1.patch Eclipse's formatter is a convenient way to clean up formatting for Java code. Currently, there is no Eclipse formatter file checked into Hive's codebase. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7159) For inner joins push a 'is not null predicate' to the join sources for every non nullSafe join condition
[ https://issues.apache.org/jira/browse/HIVE-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034466#comment-14034466 ] Harish Butani commented on HIVE-7159: - The issue with join pushdown is that now genNotNullFilterForJoinSourcePlan is always working of the top QBJoinTree; in the original patch the joinTree was passed in, somewhere along the line this got refactored, looking into reverting. For inner joins push a 'is not null predicate' to the join sources for every non nullSafe join condition Key: HIVE-7159 URL: https://issues.apache.org/jira/browse/HIVE-7159 Project: Hive Issue Type: Bug Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-7159.1.patch, HIVE-7159.2.patch, HIVE-7159.3.patch, HIVE-7159.4.patch, HIVE-7159.5.patch, HIVE-7159.6.patch, HIVE-7159.7.patch A join B on A.x = B.y can be transformed to (A where x is not null) join (B where y is not null) on A.x = B.y Apart from avoiding shuffling null keyed rows it also avoids issues with reduce-side skew when there are a lot of null values in the data. Thanks to [~gopalv] for the analysis and coming up with the solution. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7185) KeyWrapperFactory#TextKeyWrapper#equals() extracts Text incorrectly when isCopy is false
[
https://issues.apache.org/jira/browse/HIVE-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034510#comment-14034510
]
Swarnim Kulkarni commented on HIVE-7185:
[~suyeon1222] Any test failures after you made this update? If not can you add
a test to show what was broken before this change and show how this change
fixes that?
KeyWrapperFactory#TextKeyWrapper#equals() extracts Text incorrectly when
isCopy is false
Key: HIVE-7185
URL: https://issues.apache.org/jira/browse/HIVE-7185
Project: Hive
Issue Type: Bug
Reporter: Ted Yu
Assignee: SUYEON LEE
Priority: Minor
Attachments: HIVE-7185.patch
{code}
} else {
t1 = soi_new.getPrimitiveWritableObject(key);
t2 = soi_copy.getPrimitiveWritableObject(obj);
{code}
t2 should be assigned soi_new.getPrimitiveWritableObject(obj)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034531#comment-14034531
]
Sushanth Sowmyan commented on HIVE-7094:
The only test that fails in my environment with this patch is
TestHCatPartitionPublish, and that's flaky in how it fails - it succeeds when
run by itself, as opposed to after other tests run, and that has nothing to do
with this patch. All other tests under hcatalog succeed.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch,
HIVE-7094.5.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HIVE-7247) Fix itests using hadoop-1 profile
Szehon Ho created HIVE-7247: --- Summary: Fix itests using hadoop-1 profile Key: HIVE-7247 URL: https://issues.apache.org/jira/browse/HIVE-7247 Project: Hive Issue Type: Bug Components: Testing Infrastructure Reporter: Szehon Ho Assignee: Szehon Ho -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 19830: HIVE-6694: Beeline should provide a way to execute shell command as Hive CLI does
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19830/#review46006 --- beeline/src/java/org/apache/hive/beeline/Commands.java https://reviews.apache.org/r/19830/#comment81167 Should we be returning a false at this point since the command failed? itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java https://reviews.apache.org/r/19830/#comment81168 Is it possible to test a failed command as well? - Swarnim Kulkarni On June 17, 2014, 1:49 a.m., Xuefu Zhang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19830/ --- (Updated June 17, 2014, 1:49 a.m.) Review request for hive. Bugs: HIVE-6694 https://issues.apache.org/jira/browse/HIVE-6694 Repository: hive-git Description --- Introduce the new requested feature. Diffs - beeline/src/java/org/apache/hive/beeline/BeeLine.java dcdd6f2 beeline/src/java/org/apache/hive/beeline/Commands.java 94f74d2 beeline/src/main/resources/BeeLine.properties 697c29a cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 3357414 itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java 4d15e3c itests/util/pom.xml b38c55b itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 2974327 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 32c2fb0 ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 2ce4dbd ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java d2e122d Diff: https://reviews.apache.org/r/19830/diff/ Testing --- New test cases are added. Thanks, Xuefu Zhang
[jira] [Commented] (HIVE-7119) Extended ACL's should be inherited if warehouse perm inheritance enabled
[ https://issues.apache.org/jira/browse/HIVE-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034537#comment-14034537 ] Szehon Ho commented on HIVE-7119: - Sorry about that, I'll take a look, made HIVE-7247 Extended ACL's should be inherited if warehouse perm inheritance enabled Key: HIVE-7119 URL: https://issues.apache.org/jira/browse/HIVE-7119 Project: Hive Issue Type: Bug Reporter: Szehon Ho Assignee: Szehon Ho Labels: TODOC14 Fix For: 0.14.0 Attachments: HIVE-7119.2.patch, HIVE-7119.3.patch, HIVE-7119.4.patch, HIVE-7119.patch HDFS recently came out with support for extended ACL's, ie permission for specific group/user in addition to the general owner/group/other permission. Hive permission inheritance should also inherit those as well, if user has set them at any point in the warehouse directory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034544#comment-14034544
]
David Chen commented on HIVE-7094:
--
Thanks for your help, [~sushanth]! I have a separate ticket open to track down
the flakiness in TestHCatPartitionPublish (HIVE-7110).
Is there any more feedback before this patch can be committed?
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch,
HIVE-7094.5.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6694) Beeline should provide a way to execute shell command as Hive CLI does
[ https://issues.apache.org/jira/browse/HIVE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034543#comment-14034543 ] Swarnim Kulkarni commented on HIVE-6694: Left few minor comments on RB. Beeline should provide a way to execute shell command as Hive CLI does -- Key: HIVE-6694 URL: https://issues.apache.org/jira/browse/HIVE-6694 Project: Hive Issue Type: Improvement Components: CLI, Clients Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Fix For: 0.14.0 Attachments: HIVE-6694.1.patch, HIVE-6694.1.patch, HIVE-6694.2.patch, HIVE-6694.3.patch, HIVE-6694.patch Hive CLI allows a user to execute a shell command using ! notation. For instance, !cat myfile.txt. Being able to execute shell command may be important for some users. As a replacement, however, Beeline provides no such capability, possibly because ! notation is reserved for SQLLine commands. It's possible to provide this using a slightly syntactic variation such as !sh cat myfilie.txt. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 22612: HIVE-7234: Handle nulls from decimal columns elegantly
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22612/#review46008 --- common/src/java/org/apache/hadoop/hive/common/type/HiveDecimal.java https://reviews.apache.org/r/22612/#comment81170 This doesn't seem specific to the HiveDecimal class. Why not simply make it private and move to the class where it is actually being used? itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestDecimal.java https://reviews.apache.org/r/22612/#comment81171 I think this class is nice. But is it possible to throw in a .q file as well with queries testing querying of decimals greater than 99 - Swarnim Kulkarni On June 16, 2014, 2:06 a.m., Ashish Singh wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22612/ --- (Updated June 16, 2014, 2:06 a.m.) Review request for hive, Szehon Ho and Xuefu Zhang. Bugs: HIVE-7234 https://issues.apache.org/jira/browse/HIVE-7234 Repository: hive-git Description --- HIVE-7234: Handle nulls from decimal columns elegantly Diffs - common/src/java/org/apache/hadoop/hive/common/type/HiveDecimal.java ad0901548217fbb828a01f8f5edda64581ac2c1e data/files/decimal_10_0.txt PRE-CREATION data/files/decimal_9_0.txt PRE-CREATION itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestDecimal.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyHiveDecimal.java 78cc3819c61f5a1bcb0cdd3425a0105416c26861 serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 5a4623729ec955bbe8fcf662503b42ff8735eaad Diff: https://reviews.apache.org/r/22612/diff/ Testing --- Added unit tests to test the scenario. Thanks, Ashish Singh
[jira] [Created] (HIVE-7248) UNION ALL in hive returns incorrect results on Hbase backed table
Mala Chikka Kempanna created HIVE-7248: -- Summary: UNION ALL in hive returns incorrect results on Hbase backed table Key: HIVE-7248 URL: https://issues.apache.org/jira/browse/HIVE-7248 Project: Hive Issue Type: Bug Affects Versions: 0.13.1, 0.13.0, 0.12.0 Reporter: Mala Chikka Kempanna The issue can be recreated with following steps 1) In hbase create 'TABLE_EMP','default' 2) On hive sudo -u hive hive CREATE EXTERNAL TABLE TABLE_EMP(FIRST_NAME string,LAST_NAME string,CDS_UPDATED_DATE string,CDS_PK string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES(hbase.columns.mapping = default:FIRST_NAME,default:LAST_NAME,default:CDS_UPDATED_DATE,:key, hbase.scan.cache = 500, hbase.scan.cacheblocks = false ) TBLPROPERTIES(hbase.table.name = TABLE_EMP,'serialization.null.format'=''); 3) On hbase insert the following data put 'TABLE_EMP', '1', 'default:FIRST_NAME', 'Srini' put 'TABLE_EMP', '1', 'default:LAST_NAME', 'P' put 'TABLE_EMP', '1', 'default:CDS_UPDATED_DATE', '2014-06-16 00:00:00' put 'TABLE_EMP', '2', 'default:FIRST_NAME', 'Aravind' put 'TABLE_EMP', '2', 'default:LAST_NAME', 'K' put 'TABLE_EMP', '2', 'default:CDS_UPDATED_DATE', '2014-06-16 00:00:00' 4) On hive execute the following query hive SELECT * FROM ( SELECT CDS_PK FROM TABLE_EMP WHERE CDS_PK = '0' AND CDS_PK = '9' AND CDS_UPDATED_DATE IS NOT NULL UNION ALL SELECT CDS_PK FROM TABLE_EMP WHERE CDS_PK = 'a' AND CDS_PK = 'z' AND CDS_UPDATED_DATE IS NOT NULL )t ; 5) Output of the query 1 1 2 2 6) Output of just SELECT CDS_PK FROM TABLE_EMP WHERE CDS_PK = '0' AND CDS_PK = '9' AND CDS_UPDATED_DATE IS NOT NULL is 1 2 7) Output of just SELECT CDS_PK FROM TABLE_EMP WHERE CDS_PK = 'a' AND CDS_PK = 'z' AND CDS_UPDATED_DATE IS NOT NULL Empty 8) UNION is used to combine the result from multiple SELECT statements into a single result set. Hive currently only supports UNION ALL (bag union), in which duplicates are not eliminated Accordingly above query should return output 1 2 instead it is giving wrong output 1 1 2 2 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7248) UNION ALL in hive returns incorrect results on Hbase backed table
[ https://issues.apache.org/jira/browse/HIVE-7248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034555#comment-14034555 ] Mala Chikka Kempanna commented on HIVE-7248: Current work-around for the problem is to use DISTINCT in one of the sub queries. UNION ALL in hive returns incorrect results on Hbase backed table - Key: HIVE-7248 URL: https://issues.apache.org/jira/browse/HIVE-7248 Project: Hive Issue Type: Bug Affects Versions: 0.12.0, 0.13.0, 0.13.1 Reporter: Mala Chikka Kempanna The issue can be recreated with following steps 1) In hbase create 'TABLE_EMP','default' 2) On hive sudo -u hive hive CREATE EXTERNAL TABLE TABLE_EMP(FIRST_NAME string,LAST_NAME string,CDS_UPDATED_DATE string,CDS_PK string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES(hbase.columns.mapping = default:FIRST_NAME,default:LAST_NAME,default:CDS_UPDATED_DATE,:key, hbase.scan.cache = 500, hbase.scan.cacheblocks = false ) TBLPROPERTIES(hbase.table.name = TABLE_EMP,'serialization.null.format'=''); 3) On hbase insert the following data put 'TABLE_EMP', '1', 'default:FIRST_NAME', 'Srini' put 'TABLE_EMP', '1', 'default:LAST_NAME', 'P' put 'TABLE_EMP', '1', 'default:CDS_UPDATED_DATE', '2014-06-16 00:00:00' put 'TABLE_EMP', '2', 'default:FIRST_NAME', 'Aravind' put 'TABLE_EMP', '2', 'default:LAST_NAME', 'K' put 'TABLE_EMP', '2', 'default:CDS_UPDATED_DATE', '2014-06-16 00:00:00' 4) On hive execute the following query hive SELECT * FROM ( SELECT CDS_PK FROM TABLE_EMP WHERE CDS_PK = '0' AND CDS_PK = '9' AND CDS_UPDATED_DATE IS NOT NULL UNION ALL SELECT CDS_PK FROM TABLE_EMP WHERE CDS_PK = 'a' AND CDS_PK = 'z' AND CDS_UPDATED_DATE IS NOT NULL )t ; 5) Output of the query 1 1 2 2 6) Output of just SELECT CDS_PK FROM TABLE_EMP WHERE CDS_PK = '0' AND CDS_PK = '9' AND CDS_UPDATED_DATE IS NOT NULL is 1 2 7) Output of just SELECT CDS_PK FROM TABLE_EMP WHERE CDS_PK = 'a' AND CDS_PK = 'z' AND CDS_UPDATED_DATE IS NOT NULL Empty 8) UNION is used to combine the result from multiple SELECT statements into a single result set. Hive currently only supports UNION ALL (bag union), in which duplicates are not eliminated Accordingly above query should return output 1 2 instead it is giving wrong output 1 1 2 2 -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 22612: HIVE-7234: Handle nulls from decimal columns elegantly
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22612/#review46009 --- serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyHiveDecimal.java https://reviews.apache.org/r/22612/#comment81172 Since it's a public method, some javadoc on this method would be nice. - Swarnim Kulkarni On June 16, 2014, 2:06 a.m., Ashish Singh wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22612/ --- (Updated June 16, 2014, 2:06 a.m.) Review request for hive, Szehon Ho and Xuefu Zhang. Bugs: HIVE-7234 https://issues.apache.org/jira/browse/HIVE-7234 Repository: hive-git Description --- HIVE-7234: Handle nulls from decimal columns elegantly Diffs - common/src/java/org/apache/hadoop/hive/common/type/HiveDecimal.java ad0901548217fbb828a01f8f5edda64581ac2c1e data/files/decimal_10_0.txt PRE-CREATION data/files/decimal_9_0.txt PRE-CREATION itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestDecimal.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyHiveDecimal.java 78cc3819c61f5a1bcb0cdd3425a0105416c26861 serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 5a4623729ec955bbe8fcf662503b42ff8735eaad Diff: https://reviews.apache.org/r/22612/diff/ Testing --- Added unit tests to test the scenario. Thanks, Ashish Singh
[jira] [Commented] (HIVE-7234) Select on decimal column throws NPE
[ https://issues.apache.org/jira/browse/HIVE-7234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034563#comment-14034563 ] Swarnim Kulkarni commented on HIVE-7234: Left some minor comments on RB. Select on decimal column throws NPE --- Key: HIVE-7234 URL: https://issues.apache.org/jira/browse/HIVE-7234 Project: Hive Issue Type: Bug Reporter: Ashish Kumar Singh Assignee: Ashish Kumar Singh Attachments: HIVE-7234.patch Select on decimal column throws NPE for values greater than maximum permissible value (99) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7248) UNION ALL in hive returns incorrect results on Hbase backed table
[ https://issues.apache.org/jira/browse/HIVE-7248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swarnim Kulkarni updated HIVE-7248: --- Component/s: HBase Handler UNION ALL in hive returns incorrect results on Hbase backed table - Key: HIVE-7248 URL: https://issues.apache.org/jira/browse/HIVE-7248 Project: Hive Issue Type: Bug Components: HBase Handler Affects Versions: 0.12.0, 0.13.0, 0.13.1 Reporter: Mala Chikka Kempanna The issue can be recreated with following steps 1) In hbase create 'TABLE_EMP','default' 2) On hive sudo -u hive hive CREATE EXTERNAL TABLE TABLE_EMP(FIRST_NAME string,LAST_NAME string,CDS_UPDATED_DATE string,CDS_PK string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES(hbase.columns.mapping = default:FIRST_NAME,default:LAST_NAME,default:CDS_UPDATED_DATE,:key, hbase.scan.cache = 500, hbase.scan.cacheblocks = false ) TBLPROPERTIES(hbase.table.name = TABLE_EMP,'serialization.null.format'=''); 3) On hbase insert the following data put 'TABLE_EMP', '1', 'default:FIRST_NAME', 'Srini' put 'TABLE_EMP', '1', 'default:LAST_NAME', 'P' put 'TABLE_EMP', '1', 'default:CDS_UPDATED_DATE', '2014-06-16 00:00:00' put 'TABLE_EMP', '2', 'default:FIRST_NAME', 'Aravind' put 'TABLE_EMP', '2', 'default:LAST_NAME', 'K' put 'TABLE_EMP', '2', 'default:CDS_UPDATED_DATE', '2014-06-16 00:00:00' 4) On hive execute the following query hive SELECT * FROM ( SELECT CDS_PK FROM TABLE_EMP WHERE CDS_PK = '0' AND CDS_PK = '9' AND CDS_UPDATED_DATE IS NOT NULL UNION ALL SELECT CDS_PK FROM TABLE_EMP WHERE CDS_PK = 'a' AND CDS_PK = 'z' AND CDS_UPDATED_DATE IS NOT NULL )t ; 5) Output of the query 1 1 2 2 6) Output of just SELECT CDS_PK FROM TABLE_EMP WHERE CDS_PK = '0' AND CDS_PK = '9' AND CDS_UPDATED_DATE IS NOT NULL is 1 2 7) Output of just SELECT CDS_PK FROM TABLE_EMP WHERE CDS_PK = 'a' AND CDS_PK = 'z' AND CDS_UPDATED_DATE IS NOT NULL Empty 8) UNION is used to combine the result from multiple SELECT statements into a single result set. Hive currently only supports UNION ALL (bag union), in which duplicates are not eliminated Accordingly above query should return output 1 2 instead it is giving wrong output 1 1 2 2 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7230) Add Eclipse formatter file for Hive coding conventions
[ https://issues.apache.org/jira/browse/HIVE-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034566#comment-14034566 ] David Chen commented on HIVE-7230: -- [~swarnim] I was not able to generate the formatter settings in the .settings/org.eclipse.jdt.core.prefs files even with using code snippet you provided. Were you able to use that plugin to generate the correct preference settings? I am not that familiar with Maven. I think if getting Maven to generate the correct project files becomes more involved than expected, can we go ahead and merge this patch? This patch was for just adding the formatter file to the repository. Add Eclipse formatter file for Hive coding conventions -- Key: HIVE-7230 URL: https://issues.apache.org/jira/browse/HIVE-7230 Project: Hive Issue Type: Improvement Reporter: David Chen Assignee: David Chen Attachments: HIVE-7230.1.patch Eclipse's formatter is a convenient way to clean up formatting for Java code. Currently, there is no Eclipse formatter file checked into Hive's codebase. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7240) Add classifier for avro-mapred jar
[ https://issues.apache.org/jira/browse/HIVE-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034569#comment-14034569 ] Gunther Hagleitner commented on HIVE-7240: -- +1 Add classifier for avro-mapred jar -- Key: HIVE-7240 URL: https://issues.apache.org/jira/browse/HIVE-7240 Project: Hive Issue Type: Task Components: File Formats Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-7240.1.patch, HIVE-7240.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7230) Add Eclipse formatter file for Hive coding conventions
[ https://issues.apache.org/jira/browse/HIVE-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034576#comment-14034576 ] Swarnim Kulkarni commented on HIVE-7230: My motivation behind using maven was that when someone builds a project and imports it into eclipse, the formatter should get imported as well. If someone has to manually import it, it kind of again leaves an open gap for a badly formatted file to get merged in. I'll give this a shot and get back to you. Add Eclipse formatter file for Hive coding conventions -- Key: HIVE-7230 URL: https://issues.apache.org/jira/browse/HIVE-7230 Project: Hive Issue Type: Improvement Reporter: David Chen Assignee: David Chen Attachments: HIVE-7230.1.patch Eclipse's formatter is a convenient way to clean up formatting for Java code. Currently, there is no Eclipse formatter file checked into Hive's codebase. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7230) Add Eclipse formatter file for Hive coding conventions
[ https://issues.apache.org/jira/browse/HIVE-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034581#comment-14034581 ] David Chen commented on HIVE-7230: -- Got it. Thanks for your help! Add Eclipse formatter file for Hive coding conventions -- Key: HIVE-7230 URL: https://issues.apache.org/jira/browse/HIVE-7230 Project: Hive Issue Type: Improvement Reporter: David Chen Assignee: David Chen Attachments: HIVE-7230.1.patch Eclipse's formatter is a convenient way to clean up formatting for Java code. Currently, there is no Eclipse formatter file checked into Hive's codebase. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7159) For inner joins push a 'is not null predicate' to the join sources for every non nullSafe join condition
[ https://issues.apache.org/jira/browse/HIVE-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Butani updated HIVE-7159: Attachment: HIVE-7159.8.patch For inner joins push a 'is not null predicate' to the join sources for every non nullSafe join condition Key: HIVE-7159 URL: https://issues.apache.org/jira/browse/HIVE-7159 Project: Hive Issue Type: Bug Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-7159.1.patch, HIVE-7159.2.patch, HIVE-7159.3.patch, HIVE-7159.4.patch, HIVE-7159.5.patch, HIVE-7159.6.patch, HIVE-7159.7.patch, HIVE-7159.8.patch A join B on A.x = B.y can be transformed to (A where x is not null) join (B where y is not null) on A.x = B.y Apart from avoiding shuffling null keyed rows it also avoids issues with reduce-side skew when there are a lot of null values in the data. Thanks to [~gopalv] for the analysis and coming up with the solution. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7159) For inner joins push a 'is not null predicate' to the join sources for every non nullSafe join condition
[ https://issues.apache.org/jira/browse/HIVE-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034587#comment-14034587 ] Harish Butani commented on HIVE-7159: - [~hagleitn] issues a) and b) were because the check was being made against the wrong QBJoinTree. patch.8 fixes this; I removed the null and empty checks on the nullSafes error. For inner joins push a 'is not null predicate' to the join sources for every non nullSafe join condition Key: HIVE-7159 URL: https://issues.apache.org/jira/browse/HIVE-7159 Project: Hive Issue Type: Bug Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-7159.1.patch, HIVE-7159.2.patch, HIVE-7159.3.patch, HIVE-7159.4.patch, HIVE-7159.5.patch, HIVE-7159.6.patch, HIVE-7159.7.patch, HIVE-7159.8.patch A join B on A.x = B.y can be transformed to (A where x is not null) join (B where y is not null) on A.x = B.y Apart from avoiding shuffling null keyed rows it also avoids issues with reduce-side skew when there are a lot of null values in the data. Thanks to [~gopalv] for the analysis and coming up with the solution. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7159) For inner joins push a 'is not null predicate' to the join sources for every non nullSafe join condition
[ https://issues.apache.org/jira/browse/HIVE-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034620#comment-14034620 ] Eugene Koifman commented on HIVE-7159: -- FWIW, you can do the same with outer joins on inner side R left outer join S on R.r=S.s is the same as R LOJ (select * from S where s is not null) as S on R.r=S.s and symmetrically for ROJ. For inner joins push a 'is not null predicate' to the join sources for every non nullSafe join condition Key: HIVE-7159 URL: https://issues.apache.org/jira/browse/HIVE-7159 Project: Hive Issue Type: Bug Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-7159.1.patch, HIVE-7159.2.patch, HIVE-7159.3.patch, HIVE-7159.4.patch, HIVE-7159.5.patch, HIVE-7159.6.patch, HIVE-7159.7.patch, HIVE-7159.8.patch A join B on A.x = B.y can be transformed to (A where x is not null) join (B where y is not null) on A.x = B.y Apart from avoiding shuffling null keyed rows it also avoids issues with reduce-side skew when there are a lot of null values in the data. Thanks to [~gopalv] for the analysis and coming up with the solution. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7230) Add Eclipse formatter file for Hive coding conventions
[
https://issues.apache.org/jira/browse/HIVE-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034628#comment-14034628
]
Swarnim Kulkarni commented on HIVE-7230:
[~davidzchen] What command were you trying to run to generate the files? I used
the same plugin and used the following command:
{noformat}
mvn clean eclipse:eclipse -pl hbase-handler -P hadoop-1
{noformat}
and my .settings/org.eclipse.jdt.core.prefs was generated[1]
[1] http://pastebin.com/zupmuMrf
Add Eclipse formatter file for Hive coding conventions
--
Key: HIVE-7230
URL: https://issues.apache.org/jira/browse/HIVE-7230
Project: Hive
Issue Type: Improvement
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7230.1.patch
Eclipse's formatter is a convenient way to clean up formatting for Java code.
Currently, there is no Eclipse formatter file checked into Hive's codebase.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HIVE-7249) HiveTxnManager.closeTxnManger() throws if called after commitTxn()
Eugene Koifman created HIVE-7249: Summary: HiveTxnManager.closeTxnManger() throws if called after commitTxn() Key: HIVE-7249 URL: https://issues.apache.org/jira/browse/HIVE-7249 Project: Hive Issue Type: Bug Components: Locking Affects Versions: 0.13.1 Reporter: Eugene Koifman Assignee: Alan Gates I openTxn() and acquireLocks() for a query that looks like INSERT INTO T PARTITION(p) SELECT * FROM T. Then I call commitTxn(). Then I call closeTxnManger() I get an exception saying lock not found (the only lock in this txn). So it seems TxnMgr doesn't know that commit released the locks. Here is the stack trace and some log output which maybe useful: 2014-06-17 15:54:40,771 DEBUG mapreduce.TransactionContext (TransactionContext.java:onCommitJob(128)) - onCommitJob(job_local557130041_0001). this=46719652 2014-06-17 15:54:40,771 DEBUG lockmgr.DbTxnManager (DbTxnManager.java:commitTxn(205)) - Committing txn 1 2014-06-17 15:54:40,771 DEBUG txn.TxnHandler (TxnHandler.java:getDbTime(872)) - Going to execute query values current_timestamp 2014-06-17 15:54:40,772 DEBUG txn.TxnHandler (TxnHandler.java:heartbeatTxn(1423)) - Going to execute query select txn_state from TXNS where txn_id = 1 for\ update 2014-06-17 15:54:40,773 DEBUG txn.TxnHandler (TxnHandler.java:heartbeatTxn(1438)) - Going to execute update update TXNS set txn_last_heartbeat = 140304568\ 0772 where txn_id = 1 2014-06-17 15:54:40,778 DEBUG txn.TxnHandler (TxnHandler.java:heartbeatTxn(1440)) - Going to commit 2014-06-17 15:54:40,779 DEBUG txn.TxnHandler (TxnHandler.java:commitTxn(344)) - Going to execute insert insert into COMPLETED_TXN_COMPONENTS select tc_txn\ id, tc_database, tc_table, tc_partition from TXN_COMPONENTS where tc_txnid = 1 2014-06-17 15:54:40,784 DEBUG txn.TxnHandler (TxnHandler.java:commitTxn(352)) - Going to execute update delete from TXN_COMPONENTS where tc_txnid = 1 2014-06-17 15:54:40,788 DEBUG txn.TxnHandler (TxnHandler.java:commitTxn(356)) - Going to execute update delete from HIVE_LOCKS where hl_txnid = 1 2014-06-17 15:54:40,791 DEBUG txn.TxnHandler (TxnHandler.java:commitTxn(359)) - Going to execute update delete from TXNS where txn_id = 1 2014-06-17 15:54:40,794 DEBUG txn.TxnHandler (TxnHandler.java:commitTxn(361)) - Going to commit 2014-06-17 15:54:40,795 WARN mapreduce.TransactionContext (TransactionContext.java:cleanup(317)) - cleanupJob(JobID=job_local557130041_0001)this=46719652 2014-06-17 15:54:40,795 DEBUG lockmgr.DbLockManager (DbLockManager.java:unlock(109)) - Unlocking id:1 2014-06-17 15:54:40,796 DEBUG txn.TxnHandler (TxnHandler.java:getDbTime(872)) - Going to execute query values current_timestamp 2014-06-17 15:54:40,796 DEBUG txn.TxnHandler (TxnHandler.java:heartbeatLock(1402)) - Going to execute update update HIVE_LOCKS set hl_last_heartbeat = 140\ 3045680796 where hl_lock_ext_id = 1 2014-06-17 15:54:40,800 DEBUG txn.TxnHandler (TxnHandler.java:heartbeatLock(1405)) - Going to rollback 2014-06-17 15:54:40,804 ERROR metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(143)) - NoSuchLockException(message:No such lock: 1) at org.apache.hadoop.hive.metastore.txn.TxnHandler.heartbeatLock(TxnHandler.java:1407) at org.apache.hadoop.hive.metastore.txn.TxnHandler.unlock(TxnHandler.java:477) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.unlock(HiveMetaStore.java:4817) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105) at com.sun.proxy.$Proxy14.unlock(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.unlock(HiveMetaStoreClient.java:1598) at org.apache.hadoop.hive.ql.lockmgr.DbLockManager.unlock(DbLockManager.java:110) at org.apache.hadoop.hive.ql.lockmgr.DbLockManager.close(DbLockManager.java:162) at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.destruct(DbTxnManager.java:300) at org.apache.hadoop.hive.ql.lockmgr.HiveTxnManagerImpl.closeTxnManager(HiveTxnManagerImpl.java:39) at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.closeTxnManager(DbTxnManager.java:43) at org.apache.hive.hcatalog.mapreduce.TransactionContext.cleanup(TransactionContext.java:327) at org.apache.hive.hcatalog.mapreduce.TransactionContext.onCommitJob(TransactionContext.java:142) at org.apache.hive.hcatalog.mapreduce.OutputCommitterContainer.commitJob(OutputCommitterContainer.java:61) at
[jira] [Commented] (HIVE-7230) Add Eclipse formatter file for Hive coding conventions
[
https://issues.apache.org/jira/browse/HIVE-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034633#comment-14034633
]
David Chen commented on HIVE-7230:
--
I see. I ran the following command from the Hive Developer FAQ
(https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ#HiveDeveloperFAQ-HowdoIimportintoeclipse?):
{code}
mvn eclipse:eclipse -DdownloadSources -DdownloadJavadocs -Phadoop-2
{code}
Do I need to specify {{-pl module}} in order for the settings to be generated?
Add Eclipse formatter file for Hive coding conventions
--
Key: HIVE-7230
URL: https://issues.apache.org/jira/browse/HIVE-7230
Project: Hive
Issue Type: Improvement
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7230.1.patch
Eclipse's formatter is a convenient way to clean up formatting for Java code.
Currently, there is no Eclipse formatter file checked into Hive's codebase.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7249) HiveTxnManager.closeTxnManger() throws if called after commitTxn()
[
https://issues.apache.org/jira/browse/HIVE-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-7249:
-
Description:
I openTxn() and acquireLocks() for a query that looks like INSERT INTO T
PARTITION(p) SELECT * FROM T.
Then I call commitTxn(). Then I call closeTxnManger() I get an exception
saying lock not found (the only lock in this txn). So it seems TxnMgr doesn't
know that commit released the locks.
Here is the stack trace and some log output which maybe useful:
{noformat}
2014-06-17 15:54:40,771 DEBUG mapreduce.TransactionContext
(TransactionContext.java:onCommitJob(128)) -
onCommitJob(job_local557130041_0001). this=46719652
2014-06-17 15:54:40,771 DEBUG lockmgr.DbTxnManager
(DbTxnManager.java:commitTxn(205)) - Committing txn 1
2014-06-17 15:54:40,771 DEBUG txn.TxnHandler (TxnHandler.java:getDbTime(872)) -
Going to execute query values current_timestamp
2014-06-17 15:54:40,772 DEBUG txn.TxnHandler
(TxnHandler.java:heartbeatTxn(1423)) - Going to execute query select txn_state
from TXNS where txn_id = 1 for\
update
2014-06-17 15:54:40,773 DEBUG txn.TxnHandler
(TxnHandler.java:heartbeatTxn(1438)) - Going to execute update update TXNS set
txn_last_heartbeat = 140304568\
0772 where txn_id = 1
2014-06-17 15:54:40,778 DEBUG txn.TxnHandler
(TxnHandler.java:heartbeatTxn(1440)) - Going to commit
2014-06-17 15:54:40,779 DEBUG txn.TxnHandler (TxnHandler.java:commitTxn(344)) -
Going to execute insert insert into COMPLETED_TXN_COMPONENTS select tc_txn\
id, tc_database, tc_table, tc_partition from TXN_COMPONENTS where tc_txnid = 1
2014-06-17 15:54:40,784 DEBUG txn.TxnHandler (TxnHandler.java:commitTxn(352)) -
Going to execute update delete from TXN_COMPONENTS where tc_txnid = 1
2014-06-17 15:54:40,788 DEBUG txn.TxnHandler (TxnHandler.java:commitTxn(356)) -
Going to execute update delete from HIVE_LOCKS where hl_txnid = 1
2014-06-17 15:54:40,791 DEBUG txn.TxnHandler (TxnHandler.java:commitTxn(359)) -
Going to execute update delete from TXNS where txn_id = 1
2014-06-17 15:54:40,794 DEBUG txn.TxnHandler (TxnHandler.java:commitTxn(361)) -
Going to commit
2014-06-17 15:54:40,795 WARN mapreduce.TransactionContext
(TransactionContext.java:cleanup(317)) -
cleanupJob(JobID=job_local557130041_0001)this=46719652
2014-06-17 15:54:40,795 DEBUG lockmgr.DbLockManager
(DbLockManager.java:unlock(109)) - Unlocking id:1
2014-06-17 15:54:40,796 DEBUG txn.TxnHandler (TxnHandler.java:getDbTime(872)) -
Going to execute query values current_timestamp
2014-06-17 15:54:40,796 DEBUG txn.TxnHandler
(TxnHandler.java:heartbeatLock(1402)) - Going to execute update update
HIVE_LOCKS set hl_last_heartbeat = 140\
3045680796 where hl_lock_ext_id = 1
2014-06-17 15:54:40,800 DEBUG txn.TxnHandler
(TxnHandler.java:heartbeatLock(1405)) - Going to rollback
2014-06-17 15:54:40,804 ERROR metastore.RetryingHMSHandler
(RetryingHMSHandler.java:invoke(143)) - NoSuchLockException(message:No such
lock: 1)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.heartbeatLock(TxnHandler.java:1407)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.unlock(TxnHandler.java:477)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.unlock(HiveMetaStore.java:4817)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at com.sun.proxy.$Proxy14.unlock(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.unlock(HiveMetaStoreClient.java:1598)
at
org.apache.hadoop.hive.ql.lockmgr.DbLockManager.unlock(DbLockManager.java:110)
at
org.apache.hadoop.hive.ql.lockmgr.DbLockManager.close(DbLockManager.java:162)
at
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.destruct(DbTxnManager.java:300)
at
org.apache.hadoop.hive.ql.lockmgr.HiveTxnManagerImpl.closeTxnManager(HiveTxnManagerImpl.java:39)
at
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.closeTxnManager(DbTxnManager.java:43)
at
org.apache.hive.hcatalog.mapreduce.TransactionContext.cleanup(TransactionContext.java:327)
at
org.apache.hive.hcatalog.mapreduce.TransactionContext.onCommitJob(TransactionContext.java:142)
at
org.apache.hive.hcatalog.mapreduce.OutputCommitterContainer.commitJob(OutputCommitterContainer.java:61)
at
org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.commitJob(FileOutputCommitterContainer.java:251)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:537)
2014-06-17 15:54:40,804 ERROR lockmgr.DbLockManager
(DbLockManager.java:unlock(114)) -
[jira] [Commented] (HIVE-7076) Plugin (exec hook) to log to application timeline data to Yarn
[ https://issues.apache.org/jira/browse/HIVE-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034646#comment-14034646 ] Thejas M Nair commented on HIVE-7076: - +1 Plugin (exec hook) to log to application timeline data to Yarn -- Key: HIVE-7076 URL: https://issues.apache.org/jira/browse/HIVE-7076 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-7076.1.patch, HIVE-7076.2.patch, HIVE-7076.3.patch See: https://issues.apache.org/jira/browse/YARN-1530 This is a simple pre/post exec hook to log query + plan information to yarn. This information can be used to build tools and UIs to monitor, track, debug and tune Hive queries. Off by default, but can be enabled via: hive.exec.pre.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.post.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook hive.exec.failure.hooks=ql.src.java.org.apache.hadoop.hive.ql.hooks.ATSHook -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7247) Fix itests using hadoop-1 profile
[ https://issues.apache.org/jira/browse/HIVE-7247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034645#comment-14034645 ] Szehon Ho commented on HIVE-7247: - The offending test is heavily using the HDFS ACL API, which is hadoop-2 only, so it won't be easy to fix. All I can think of is to create a new itests sub-proj that gets compiled and run only in hadoop-2 profile. It might be useful, as there could be more hadoop-2 features we want to test in the future. [~hagleitn], [~brocknoland] [~ashutoshc], any thoughts on that? Fix itests using hadoop-1 profile -- Key: HIVE-7247 URL: https://issues.apache.org/jira/browse/HIVE-7247 Project: Hive Issue Type: Bug Components: Testing Infrastructure Reporter: Szehon Ho Assignee: Szehon Ho -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7247) Fix itests using hadoop-1 profile
[
https://issues.apache.org/jira/browse/HIVE-7247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-7247:
Description:
Currently building itests using -Phadoop-1 profile results in following failure:
{noformat}
$cd itests
$mvn install -DskipTests -Phadoop-1
...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project hive-it-unit: Compilation failure: Compilation
failure:
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java:[31,39]
cannot find symbol
[ERROR] symbol : class AclStatus
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[20,46]
cannot find symbol
[ERROR] symbol : class AclEntryScope
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[20,1]
static import only from classes and interfaces
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[21,46]
cannot find symbol
[ERROR] symbol : class AclEntryType
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[21,1]
static import only from classes and interfaces
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[22,46]
cannot find symbol
[ERROR] symbol : class AclEntryType
{noformat}
Fix itests using hadoop-1 profile
--
Key: HIVE-7247
URL: https://issues.apache.org/jira/browse/HIVE-7247
Project: Hive
Issue Type: Bug
Components: Testing Infrastructure
Reporter: Szehon Ho
Assignee: Szehon Ho
Currently building itests using -Phadoop-1 profile results in following
failure:
{noformat}
$cd itests
$mvn install -DskipTests -Phadoop-1
...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project hive-it-unit: Compilation failure:
Compilation failure:
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java:[31,39]
cannot find symbol
[ERROR] symbol : class AclStatus
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[20,46]
cannot find symbol
[ERROR] symbol : class AclEntryScope
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[20,1]
static import only from classes and interfaces
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[21,46]
cannot find symbol
[ERROR] symbol : class AclEntryType
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[21,1]
static import only from classes and interfaces
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[22,46]
cannot find symbol
[ERROR] symbol : class AclEntryType
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7241) Wrong lock acquired for alter table rename partition
[
https://issues.apache.org/jira/browse/HIVE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034648#comment-14034648
]
Hive QA commented on HIVE-7241:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12650679/HIVE-7241.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/495/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/495/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-495/
Messages:
{noformat}
This message was trimmed, see log for full details
Decision can match input such as LPAREN KW_CASE KW_ARRAY using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4:
Decision can match input such as LPAREN KW_CASE TinyintLiteral using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4:
Decision can match input such as LPAREN KW_CASE KW_STRUCT using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4:
Decision can match input such as LPAREN KW_CASE SmallintLiteral using
multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:115:5:
Decision can match input such as KW_CLUSTER KW_BY LPAREN using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:127:5:
Decision can match input such as KW_PARTITION KW_BY LPAREN using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:138:5:
Decision can match input such as KW_DISTRIBUTE KW_BY LPAREN using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:149:5:
Decision can match input such as KW_SORT KW_BY LPAREN using multiple
alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:166:7:
Decision can match input such as STAR using multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:179:5:
Decision can match input such as KW_STRUCT using multiple alternatives: 4, 6
As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5:
Decision can match input such as KW_UNIONTYPE using multiple alternatives: 5,
6
As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5:
Decision can match input such as KW_ARRAY using multiple alternatives: 2, 6
As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:261:5:
Decision can match input such as KW_DATE StringLiteral using multiple
alternatives: 2, 3
As a result, alternative(s) 3 were disabled for that input
warning(200): IdentifiersParser.g:261:5:
Decision can match input such as KW_FALSE using multiple alternatives: 3, 8
As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5:
Decision can match input such as KW_TRUE using multiple alternatives: 3, 8
As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5:
Decision can match input such as KW_NULL using multiple alternatives: 1, 8
As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT
KW_OVERWRITE using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE
KW_BY using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_MAP LPAREN
using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT
KW_INTO using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL
KW_VIEW using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_GROUP
[jira] [Commented] (HIVE-7232) ReduceSink is emitting NULL keys due to failed keyEval
[
https://issues.apache.org/jira/browse/HIVE-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034675#comment-14034675
]
Navis commented on HIVE-7232:
-
Looks like something is wrong in broadcast join. I'll look into this.
ReduceSink is emitting NULL keys due to failed keyEval
--
Key: HIVE-7232
URL: https://issues.apache.org/jira/browse/HIVE-7232
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.14.0
Reporter: Gopal V
Assignee: Navis
Attachments: HIVE-7232-extra-logging.patch, q5.explain.txt, q5.sql
After HIVE-4867 has been merged in, some queries have exhibited a very weird
skew towards NULL keys emitted from the ReduceSinkOperator.
Added extra logging to print expr.column() in ExprNodeColumnEvaluator in
reduce sink.
{code}
2014-06-14 00:37:19,186 INFO [TezChild]
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator:
numDistributionKeys = 1 {null -- ExprNodeColumnEvaluator(_col10)}
key_row={reducesinkkey0:442}
{code}
{code}
HiveKey firstKey = toHiveKey(cachedKeys[0], tag, null);
int distKeyLength = firstKey.getDistKeyLength();
if(distKeyLength = 1) {
StringBuffer x1 = new StringBuffer();
x1.append(numDistributionKeys = + numDistributionKeys + \n);
for (int i = 0; i numDistributionKeys; i++) {
x1.append(cachedKeys[0][i] + -- + keyEval[i] + \n);
}
x1.append(key_row=+ SerDeUtils.getJSONString(row,
keyObjectInspector));
LOG.info(GOPAL: + x1.toString());
}
{code}
The query is tpc-h query5, with extra NULL checks just to be sure.
{code}
ELECT n_name,
sum(l_extendedprice * (1 - l_discount)) AS revenue
FROM customer,
orders,
lineitem,
supplier,
nation,
region
WHERE c_custkey = o_custkey
AND l_orderkey = o_orderkey
AND l_suppkey = s_suppkey
AND c_nationkey = s_nationkey
AND s_nationkey = n_nationkey
AND n_regionkey = r_regionkey
AND r_name = 'ASIA'
AND o_orderdate = '1994-01-01'
AND o_orderdate '1995-01-01'
and l_orderkey is not null
and c_custkey is not null
and l_suppkey is not null
and c_nationkey is not null
and s_nationkey is not null
and n_regionkey is not null
GROUP BY n_name
ORDER BY revenue DESC;
{code}
The reducer which has the issue has the following plan
{code}
Reducer 3
Reduce Operator Tree:
Join Operator
condition map:
Inner Join 0 to 1
condition expressions:
0 {KEY.reducesinkkey0} {VALUE._col2}
1 {VALUE._col0} {KEY.reducesinkkey0} {VALUE._col3}
outputColumnNames: _col0, _col3, _col10, _col11, _col14
Statistics: Num rows: 18344 Data size: 95229140992 Basic
stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col10 (type: int)
sort order: +
Map-reduce partition columns: _col10 (type: int)
Statistics: Num rows: 18344 Data size: 95229140992
Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int), _col3 (type: int),
_col11 (type: int), _col14 (type: string)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7240) Add classifier for avro-mapred jar
[ https://issues.apache.org/jira/browse/HIVE-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-7240: --- Resolution: Fixed Fix Version/s: 0.14.0 Status: Resolved (was: Patch Available) Committed to trunk. Add classifier for avro-mapred jar -- Key: HIVE-7240 URL: https://issues.apache.org/jira/browse/HIVE-7240 Project: Hive Issue Type: Task Components: File Formats Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.14.0 Attachments: HIVE-7240.1.patch, HIVE-7240.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7210) NPE with No plan file found when running Driver instances on multiple threads
[
https://issues.apache.org/jira/browse/HIVE-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034687#comment-14034687
]
Jason Dere commented on HIVE-7210:
--
I've seen several of these tests failing on previous tests. Ran MiniTez
dynpart_sort_optimization.q, TestHCatLoader#testReadDataPrimitiveTypes,
TestHiveServer2 locally and these pass for me. Will commit this patch shortly.
NPE with No plan file found when running Driver instances on multiple
threads
---
Key: HIVE-7210
URL: https://issues.apache.org/jira/browse/HIVE-7210
Project: Hive
Issue Type: Bug
Reporter: Jason Dere
Assignee: Gunther Hagleitner
Attachments: HIVE-7210.1.patch, HIVE-7210.2.patch
Informatica has a multithreaded application running multiple instances of
CLIDriver. When running concurrent queries they sometimes hit the following
error:
{noformat}
2014-05-30 10:24:59 pool-10-thread-1 INFO: Hadoop_Native_Log :INFO
org.apache.hadoop.hive.ql.exec.Utilities: No plan file found:
hdfs://ICRHHW21NODE1:8020/tmp/hive-qamercury/hive_2014-05-30_10-24-57_346_890014621821056491-2/-mr-10002/6169987c-3263-4737-b5cb-38daab882afb/map.xml
2014-05-30 10:24:59 pool-10-thread-1 INFO: Hadoop_Native_Log :INFO
org.apache.hadoop.mapreduce.JobSubmitter: Cleaning up the staging area
/tmp/hadoop-yarn/staging/qamercury/.staging/job_1401360353644_0078
2014-05-30 10:24:59 pool-10-thread-1 INFO: Hadoop_Native_Log :ERROR
org.apache.hadoop.hive.ql.exec.Task: Job Submission failed with exception
'java.lang.NullPointerException(null)'
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:271)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:520)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:512)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at
org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
at
org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:420)
at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at
org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
at
org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
com.informatica.platform.dtm.executor.hive.impl.AbstractHiveDriverBaseImpl.run(AbstractHiveDriverBaseImpl.java:86)
at
com.informatica.platform.dtm.executor.hive.MHiveDriver.executeQuery(MHiveDriver.java:126)
at
com.informatica.platform.dtm.executor.hive.task.impl.HiveTaskHandlerImpl.executeQuery(HiveTaskHandlerImpl.java:358)
at
com.informatica.platform.dtm.executor.hive.task.impl.HiveTaskHandlerImpl.executeScript(HiveTaskHandlerImpl.java:247)
at
com.informatica.platform.dtm.executor.hive.task.impl.HiveTaskHandlerImpl.executeMainScript(HiveTaskHandlerImpl.java:194)
at
[jira] [Commented] (HIVE-7190) WebHCat launcher task failure can cause two concurent user jobs to run
[ https://issues.apache.org/jira/browse/HIVE-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034698#comment-14034698 ] Ivan Mitic commented on HIVE-7190: -- Thanks Thejas and Eugene for the review and commit! WebHCat launcher task failure can cause two concurent user jobs to run -- Key: HIVE-7190 URL: https://issues.apache.org/jira/browse/HIVE-7190 Project: Hive Issue Type: Bug Components: WebHCat Affects Versions: 0.13.0 Reporter: Ivan Mitic Assignee: Ivan Mitic Fix For: 0.14.0 Attachments: HIVE-7190.2.patch, HIVE-7190.3.patch, HIVE-7190.patch Templeton uses launcher jobs to launch the actual user jobs. Launcher jobs are 1-map jobs (a single task jobs) which kick off the actual user job and monitor it until it finishes. Given that the launcher is a task, like any other MR task, it has a retry policy in case it fails (due to a task crash, tasktracker/nodemanager crash, machine level outage, etc.). Further, when launcher task is retried, it will again launch the same user job, *however* the previous attempt user job is already running. What this means is that we can have two identical user jobs running in parallel. In case of MRv2, there will be an MRAppMaster and the launcher task, which are subject to failure. In case any of the two fails, another instance of a user job will be launched again in parallel. Above situation is already a bug. Now going further to RM HA, what RM does on failover/restart is that it kills all containers, and it restarts all applications. This means that if our customer had 10 jobs on the cluster (this is 10 launcher jobs and 10 user jobs), on RM failover, all 20 jobs will be restarted, and launcher jobs will queue user jobs again. There are two issues with this design: 1. There are *possible* chances for corruption of job outputs (it would be useful to analyze this scenario more and confirm this statement). 2. Cluster resources are spent on jobs redundantly To address the issue at least on Yarn (Hadoop 2.0) clusters, webhcat should do the same thing Oozie does in this scenario, and that is to tag all its child jobs with an id, and kill those jobs on task restart before they are kicked off again. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-7250) Adaptive compression buffer size for wide tables in ORC
Prasanth J created HIVE-7250: Summary: Adaptive compression buffer size for wide tables in ORC Key: HIVE-7250 URL: https://issues.apache.org/jira/browse/HIVE-7250 Project: Hive Issue Type: Improvement Components: File Formats Affects Versions: 0.14.0 Reporter: Prasanth J Assignee: Prasanth J If the input table is wide (in the order of 1000s), ORC compression buffer size overhead becomes significant causing OOM issues. To overcome this issue, buffer size should be adaptively chosen based on the available memory and the number of columns. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HIVE-7210) NPE with No plan file found when running Driver instances on multiple threads
[
https://issues.apache.org/jira/browse/HIVE-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere reassigned HIVE-7210:
Assignee: Jason Dere (was: Gunther Hagleitner)
NPE with No plan file found when running Driver instances on multiple
threads
---
Key: HIVE-7210
URL: https://issues.apache.org/jira/browse/HIVE-7210
Project: Hive
Issue Type: Bug
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-7210.1.patch, HIVE-7210.2.patch
Informatica has a multithreaded application running multiple instances of
CLIDriver. When running concurrent queries they sometimes hit the following
error:
{noformat}
2014-05-30 10:24:59 pool-10-thread-1 INFO: Hadoop_Native_Log :INFO
org.apache.hadoop.hive.ql.exec.Utilities: No plan file found:
hdfs://ICRHHW21NODE1:8020/tmp/hive-qamercury/hive_2014-05-30_10-24-57_346_890014621821056491-2/-mr-10002/6169987c-3263-4737-b5cb-38daab882afb/map.xml
2014-05-30 10:24:59 pool-10-thread-1 INFO: Hadoop_Native_Log :INFO
org.apache.hadoop.mapreduce.JobSubmitter: Cleaning up the staging area
/tmp/hadoop-yarn/staging/qamercury/.staging/job_1401360353644_0078
2014-05-30 10:24:59 pool-10-thread-1 INFO: Hadoop_Native_Log :ERROR
org.apache.hadoop.hive.ql.exec.Task: Job Submission failed with exception
'java.lang.NullPointerException(null)'
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:271)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:520)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:512)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at
org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
at
org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:420)
at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at
org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
at
org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
com.informatica.platform.dtm.executor.hive.impl.AbstractHiveDriverBaseImpl.run(AbstractHiveDriverBaseImpl.java:86)
at
com.informatica.platform.dtm.executor.hive.MHiveDriver.executeQuery(MHiveDriver.java:126)
at
com.informatica.platform.dtm.executor.hive.task.impl.HiveTaskHandlerImpl.executeQuery(HiveTaskHandlerImpl.java:358)
at
com.informatica.platform.dtm.executor.hive.task.impl.HiveTaskHandlerImpl.executeScript(HiveTaskHandlerImpl.java:247)
at
com.informatica.platform.dtm.executor.hive.task.impl.HiveTaskHandlerImpl.executeMainScript(HiveTaskHandlerImpl.java:194)
at
com.informatica.platform.ldtm.executor.common.workflow.taskhandler.impl.BaseTaskHandlerImpl.run(BaseTaskHandlerImpl.java:126)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
Review Request 22711: HIVE-7247 - Fix itests using hadoop-1 profile
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22711/ --- Review request for hive. Bugs: HIVE-7247 https://issues.apache.org/jira/browse/HIVE-7247 Repository: hive-git Description --- Create a new project 'hive-unit-hadoop2' for unit-tests that need to rely on hadoop-2 API's. Diffs - itests/hive-unit-hadoop2/pom.xml PRE-CREATION itests/hive-unit-hadoop2/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java PRE-CREATION itests/hive-unit/pom.xml 0eb784b itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java 154cec5 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java 44a33d8 itests/pom.xml 11c6282 Diff: https://reviews.apache.org/r/22711/diff/ Testing --- Thanks, Szehon Ho
[jira] [Updated] (HIVE-7210) NPE with No plan file found when running Driver instances on multiple threads
[
https://issues.apache.org/jira/browse/HIVE-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-7210:
-
Resolution: Fixed
Fix Version/s: 0.14.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks Gunther/Vikram for the review.
NPE with No plan file found when running Driver instances on multiple
threads
---
Key: HIVE-7210
URL: https://issues.apache.org/jira/browse/HIVE-7210
Project: Hive
Issue Type: Bug
Reporter: Jason Dere
Assignee: Jason Dere
Fix For: 0.14.0
Attachments: HIVE-7210.1.patch, HIVE-7210.2.patch
Informatica has a multithreaded application running multiple instances of
CLIDriver. When running concurrent queries they sometimes hit the following
error:
{noformat}
2014-05-30 10:24:59 pool-10-thread-1 INFO: Hadoop_Native_Log :INFO
org.apache.hadoop.hive.ql.exec.Utilities: No plan file found:
hdfs://ICRHHW21NODE1:8020/tmp/hive-qamercury/hive_2014-05-30_10-24-57_346_890014621821056491-2/-mr-10002/6169987c-3263-4737-b5cb-38daab882afb/map.xml
2014-05-30 10:24:59 pool-10-thread-1 INFO: Hadoop_Native_Log :INFO
org.apache.hadoop.mapreduce.JobSubmitter: Cleaning up the staging area
/tmp/hadoop-yarn/staging/qamercury/.staging/job_1401360353644_0078
2014-05-30 10:24:59 pool-10-thread-1 INFO: Hadoop_Native_Log :ERROR
org.apache.hadoop.hive.ql.exec.Task: Job Submission failed with exception
'java.lang.NullPointerException(null)'
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:271)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:520)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:512)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at
org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
at
org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:420)
at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at
org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
at
org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
com.informatica.platform.dtm.executor.hive.impl.AbstractHiveDriverBaseImpl.run(AbstractHiveDriverBaseImpl.java:86)
at
com.informatica.platform.dtm.executor.hive.MHiveDriver.executeQuery(MHiveDriver.java:126)
at
com.informatica.platform.dtm.executor.hive.task.impl.HiveTaskHandlerImpl.executeQuery(HiveTaskHandlerImpl.java:358)
at
com.informatica.platform.dtm.executor.hive.task.impl.HiveTaskHandlerImpl.executeScript(HiveTaskHandlerImpl.java:247)
at
com.informatica.platform.dtm.executor.hive.task.impl.HiveTaskHandlerImpl.executeMainScript(HiveTaskHandlerImpl.java:194)
at
com.informatica.platform.ldtm.executor.common.workflow.taskhandler.impl.BaseTaskHandlerImpl.run(BaseTaskHandlerImpl.java:126)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[jira] [Updated] (HIVE-7247) Fix itests using hadoop-1 profile
[
https://issues.apache.org/jira/browse/HIVE-7247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-7247:
Attachment: HIVE-7247.patch
Fix itests using hadoop-1 profile
--
Key: HIVE-7247
URL: https://issues.apache.org/jira/browse/HIVE-7247
Project: Hive
Issue Type: Bug
Components: Testing Infrastructure
Reporter: Szehon Ho
Assignee: Szehon Ho
Attachments: HIVE-7247.patch
Currently building itests using -Phadoop-1 profile results in following
failure:
{noformat}
$cd itests
$mvn install -DskipTests -Phadoop-1
...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project hive-it-unit: Compilation failure:
Compilation failure:
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java:[31,39]
cannot find symbol
[ERROR] symbol : class AclStatus
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[20,46]
cannot find symbol
[ERROR] symbol : class AclEntryScope
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[20,1]
static import only from classes and interfaces
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[21,46]
cannot find symbol
[ERROR] symbol : class AclEntryType
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[21,1]
static import only from classes and interfaces
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[22,46]
cannot find symbol
[ERROR] symbol : class AclEntryType
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7247) Fix itests using hadoop-1 profile
[
https://issues.apache.org/jira/browse/HIVE-7247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-7247:
Status: Patch Available (was: Open)
Fix itests using hadoop-1 profile
--
Key: HIVE-7247
URL: https://issues.apache.org/jira/browse/HIVE-7247
Project: Hive
Issue Type: Bug
Components: Testing Infrastructure
Reporter: Szehon Ho
Assignee: Szehon Ho
Attachments: HIVE-7247.patch, HIVE-7247.patch
Currently building itests using -Phadoop-1 profile results in following
failure:
{noformat}
$cd itests
$mvn install -DskipTests -Phadoop-1
...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project hive-it-unit: Compilation failure:
Compilation failure:
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java:[31,39]
cannot find symbol
[ERROR] symbol : class AclStatus
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[20,46]
cannot find symbol
[ERROR] symbol : class AclEntryScope
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[20,1]
static import only from classes and interfaces
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[21,46]
cannot find symbol
[ERROR] symbol : class AclEntryType
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[21,1]
static import only from classes and interfaces
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[22,46]
cannot find symbol
[ERROR] symbol : class AclEntryType
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7247) Fix itests using hadoop-1 profile
[
https://issues.apache.org/jira/browse/HIVE-7247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-7247:
Attachment: HIVE-7247.patch
Submitting a fix, based on that.
Fix itests using hadoop-1 profile
--
Key: HIVE-7247
URL: https://issues.apache.org/jira/browse/HIVE-7247
Project: Hive
Issue Type: Bug
Components: Testing Infrastructure
Reporter: Szehon Ho
Assignee: Szehon Ho
Attachments: HIVE-7247.patch, HIVE-7247.patch
Currently building itests using -Phadoop-1 profile results in following
failure:
{noformat}
$cd itests
$mvn install -DskipTests -Phadoop-1
...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project hive-it-unit: Compilation failure:
Compilation failure:
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java:[31,39]
cannot find symbol
[ERROR] symbol : class AclStatus
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[20,46]
cannot find symbol
[ERROR] symbol : class AclEntryScope
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[20,1]
static import only from classes and interfaces
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[21,46]
cannot find symbol
[ERROR] symbol : class AclEntryType
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[21,1]
static import only from classes and interfaces
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[22,46]
cannot find symbol
[ERROR] symbol : class AclEntryType
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7247) Fix itests using hadoop-1 profile
[
https://issues.apache.org/jira/browse/HIVE-7247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-7247:
Attachment: (was: HIVE-7247.patch)
Fix itests using hadoop-1 profile
--
Key: HIVE-7247
URL: https://issues.apache.org/jira/browse/HIVE-7247
Project: Hive
Issue Type: Bug
Components: Testing Infrastructure
Reporter: Szehon Ho
Assignee: Szehon Ho
Attachments: HIVE-7247.patch
Currently building itests using -Phadoop-1 profile results in following
failure:
{noformat}
$cd itests
$mvn install -DskipTests -Phadoop-1
...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project hive-it-unit: Compilation failure:
Compilation failure:
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java:[31,39]
cannot find symbol
[ERROR] symbol : class AclStatus
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[20,46]
cannot find symbol
[ERROR] symbol : class AclEntryScope
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[20,1]
static import only from classes and interfaces
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[21,46]
cannot find symbol
[ERROR] symbol : class AclEntryType
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[21,1]
static import only from classes and interfaces
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[22,46]
cannot find symbol
[ERROR] symbol : class AclEntryType
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7250) Adaptive compression buffer size for wide tables in ORC
[ https://issues.apache.org/jira/browse/HIVE-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-7250: - Attachment: HIVE-7250.1.patch Adaptive compression buffer size for wide tables in ORC --- Key: HIVE-7250 URL: https://issues.apache.org/jira/browse/HIVE-7250 Project: Hive Issue Type: Improvement Components: File Formats Affects Versions: 0.14.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile Attachments: HIVE-7250.1.patch If the input table is wide (in the order of 1000s), ORC compression buffer size overhead becomes significant causing OOM issues. To overcome this issue, buffer size should be adaptively chosen based on the available memory and the number of columns. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7232) ReduceSink is emitting NULL keys due to failed keyEval
[
https://issues.apache.org/jira/browse/HIVE-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034734#comment-14034734
]
Navis commented on HIVE-7232:
-
I've reproduced the problem. It occurs on mapjoin + vetorization combination.
ReduceSink is emitting NULL keys due to failed keyEval
--
Key: HIVE-7232
URL: https://issues.apache.org/jira/browse/HIVE-7232
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.14.0
Reporter: Gopal V
Assignee: Navis
Attachments: HIVE-7232-extra-logging.patch, q5.explain.txt, q5.sql
After HIVE-4867 has been merged in, some queries have exhibited a very weird
skew towards NULL keys emitted from the ReduceSinkOperator.
Added extra logging to print expr.column() in ExprNodeColumnEvaluator in
reduce sink.
{code}
2014-06-14 00:37:19,186 INFO [TezChild]
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator:
numDistributionKeys = 1 {null -- ExprNodeColumnEvaluator(_col10)}
key_row={reducesinkkey0:442}
{code}
{code}
HiveKey firstKey = toHiveKey(cachedKeys[0], tag, null);
int distKeyLength = firstKey.getDistKeyLength();
if(distKeyLength = 1) {
StringBuffer x1 = new StringBuffer();
x1.append(numDistributionKeys = + numDistributionKeys + \n);
for (int i = 0; i numDistributionKeys; i++) {
x1.append(cachedKeys[0][i] + -- + keyEval[i] + \n);
}
x1.append(key_row=+ SerDeUtils.getJSONString(row,
keyObjectInspector));
LOG.info(GOPAL: + x1.toString());
}
{code}
The query is tpc-h query5, with extra NULL checks just to be sure.
{code}
ELECT n_name,
sum(l_extendedprice * (1 - l_discount)) AS revenue
FROM customer,
orders,
lineitem,
supplier,
nation,
region
WHERE c_custkey = o_custkey
AND l_orderkey = o_orderkey
AND l_suppkey = s_suppkey
AND c_nationkey = s_nationkey
AND s_nationkey = n_nationkey
AND n_regionkey = r_regionkey
AND r_name = 'ASIA'
AND o_orderdate = '1994-01-01'
AND o_orderdate '1995-01-01'
and l_orderkey is not null
and c_custkey is not null
and l_suppkey is not null
and c_nationkey is not null
and s_nationkey is not null
and n_regionkey is not null
GROUP BY n_name
ORDER BY revenue DESC;
{code}
The reducer which has the issue has the following plan
{code}
Reducer 3
Reduce Operator Tree:
Join Operator
condition map:
Inner Join 0 to 1
condition expressions:
0 {KEY.reducesinkkey0} {VALUE._col2}
1 {VALUE._col0} {KEY.reducesinkkey0} {VALUE._col3}
outputColumnNames: _col0, _col3, _col10, _col11, _col14
Statistics: Num rows: 18344 Data size: 95229140992 Basic
stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col10 (type: int)
sort order: +
Map-reduce partition columns: _col10 (type: int)
Statistics: Num rows: 18344 Data size: 95229140992
Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int), _col3 (type: int),
_col11 (type: int), _col14 (type: string)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7230) Add Eclipse formatter file for Hive coding conventions
[ https://issues.apache.org/jira/browse/HIVE-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Chen updated HIVE-7230: - Attachment: HIVE-7230.2.patch Add Eclipse formatter file for Hive coding conventions -- Key: HIVE-7230 URL: https://issues.apache.org/jira/browse/HIVE-7230 Project: Hive Issue Type: Improvement Reporter: David Chen Assignee: David Chen Attachments: HIVE-7230.1.patch, HIVE-7230.2.patch Eclipse's formatter is a convenient way to clean up formatting for Java code. Currently, there is no Eclipse formatter file checked into Hive's codebase. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7230) Add Eclipse formatter file for Hive coding conventions
[ https://issues.apache.org/jira/browse/HIVE-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034747#comment-14034747 ] David Chen commented on HIVE-7230: -- [~swarnim] I tried following your steps but was still not able to generate my .settings/org.eclipse.jdt.core.prefs that included the formatter settings. It only included the following: http://pastebin.com/NpkFpwit In any case, I have posted a new patch with my changes to the root pom.xml. Add Eclipse formatter file for Hive coding conventions -- Key: HIVE-7230 URL: https://issues.apache.org/jira/browse/HIVE-7230 Project: Hive Issue Type: Improvement Reporter: David Chen Assignee: David Chen Attachments: HIVE-7230.1.patch, HIVE-7230.2.patch Eclipse's formatter is a convenient way to clean up formatting for Java code. Currently, there is no Eclipse formatter file checked into Hive's codebase. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 22590: HIVE-7230: Add Eclipse formatter file and apply Maven Eclipse plugin in root pom.xml.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22590/ --- (Updated June 18, 2014, 3:01 a.m.) Review request for hive. Changes --- Apply Maven Eclipse plugin in root pom.xml. Summary (updated) - HIVE-7230: Add Eclipse formatter file and apply Maven Eclipse plugin in root pom.xml. Bugs: HIVE-7230 https://issues.apache.org/jira/browse/HIVE-7230 Repository: hive-git Description (updated) --- HIVE-7230: Add Eclipse formatter file and apply Maven Eclipse plugin in root pom.xml. Diffs (updated) - eclipse-styles.xml PRE-CREATION pom.xml 2b91846c2b9511022367ceb41a216c603c8ac281 testutils/ptest2/pom.xml 18cd68074a226cbc38e3981886e871d5b99de756 Diff: https://reviews.apache.org/r/22590/diff/ Testing --- Manual Thanks, David Chen
[jira] [Commented] (HIVE-6561) Beeline should accept -i option to Initializing a SQL file
[
https://issues.apache.org/jira/browse/HIVE-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034750#comment-14034750
]
Hive QA commented on HIVE-6561:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12650709/HIVE-6561.4.patch.txt
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 5654 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_columnar
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/496/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/496/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-496/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12650709
Beeline should accept -i option to Initializing a SQL file
--
Key: HIVE-6561
URL: https://issues.apache.org/jira/browse/HIVE-6561
Project: Hive
Issue Type: Improvement
Affects Versions: 0.10.0, 0.11.0, 0.12.0
Reporter: Xuefu Zhang
Assignee: Navis
Attachments: HIVE-6561.1.patch.txt, HIVE-6561.2.patch.txt,
HIVE-6561.3.patch.txt, HIVE-6561.4.patch.txt
Hive CLI has -i option. From Hive CLI help:
{code}
...
-i filenameInitialization SQL file
...
{code}
However, Beeline has no such option:
{code}
xzhang@xzlt:~/apa/hive3$
./packaging/target/apache-hive-0.14.0-SNAPSHOT-bin/apache-hive-0.14.0-SNAPSHOT-bin/bin/beeline
-u jdbc:hive2:// -i hive.rc
...
Connected to: Apache Hive (version 0.14.0-SNAPSHOT)
Driver: Hive JDBC (version 0.14.0-SNAPSHOT)
Transaction isolation: TRANSACTION_REPEATABLE_READ
-i (No such file or directory)
Property url is required
Beeline version 0.14.0-SNAPSHOT by Apache Hive
...
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6694) Beeline should provide a way to execute shell command as Hive CLI does
[
https://issues.apache.org/jira/browse/HIVE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034753#comment-14034753
]
Hive QA commented on HIVE-6694:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12650713/HIVE-6694.3.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/497/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/497/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-497/
Messages:
{noformat}
This message was trimmed, see log for full details
As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5:
Decision can match input such as KW_TRUE using multiple alternatives: 3, 8
As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5:
Decision can match input such as KW_NULL using multiple alternatives: 1, 8
As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT
KW_OVERWRITE using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE
KW_BY using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_MAP LPAREN
using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT
KW_INTO using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL
KW_VIEW using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_GROUP
KW_BY using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as KW_BETWEEN KW_MAP LPAREN using multiple
alternatives: 8, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_ORDER
KW_BY using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_SORT KW_BY
using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_UNION
KW_ALL using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5:
Decision can match input such as {KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER
KW_BY using multiple alternatives: 2, 9
As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:518:5:
Decision can match input such as {AMPERSAND..BITWISEXOR, DIV..DIVIDE,
EQUAL..EQUAL_NS, GREATERTHAN..GREATERTHANOREQUALTO, KW_AND, KW_ARRAY,
KW_BETWEEN..KW_BOOLEAN, KW_CASE, KW_DOUBLE, KW_FLOAT, KW_IF, KW_IN, KW_INT,
KW_LIKE, KW_MAP, KW_NOT, KW_OR, KW_REGEXP, KW_RLIKE, KW_SMALLINT,
KW_STRING..KW_STRUCT, KW_TINYINT, KW_UNIONTYPE, KW_WHEN,
LESSTHAN..LESSTHANOREQUALTO, MINUS..NOTEQUAL, PLUS, STAR, TILDE} using
multiple alternatives: 1, 3
As a result, alternative(s) 3 were disabled for that input
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-exec ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-exec
---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-exec ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-exec ---
[INFO] Compiling 1698 source files to
/data/hive-ptest/working/apache-svn-trunk-source/ql/target/classes
[INFO] -
[WARNING] COMPILATION WARNING :
[INFO] -
[WARNING]
[jira] [Commented] (HIVE-7247) Fix itests using hadoop-1 profile
[
https://issues.apache.org/jira/browse/HIVE-7247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034778#comment-14034778
]
Brock Noland commented on HIVE-7247:
+1
Fix itests using hadoop-1 profile
--
Key: HIVE-7247
URL: https://issues.apache.org/jira/browse/HIVE-7247
Project: Hive
Issue Type: Bug
Components: Testing Infrastructure
Reporter: Szehon Ho
Assignee: Szehon Ho
Attachments: HIVE-7247.patch
Currently building itests using -Phadoop-1 profile results in following
failure:
{noformat}
$cd itests
$mvn install -DskipTests -Phadoop-1
...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project hive-it-unit: Compilation failure:
Compilation failure:
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java:[31,39]
cannot find symbol
[ERROR] symbol : class AclStatus
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[20,46]
cannot find symbol
[ERROR] symbol : class AclEntryScope
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[20,1]
static import only from classes and interfaces
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[21,46]
cannot find symbol
[ERROR] symbol : class AclEntryType
[ERROR] location: package org.apache.hadoop.fs.permission
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[21,1]
static import only from classes and interfaces
[ERROR]
/Users/ghagleitner/Projects/hive-test-trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestExtendedAcls.java:[22,46]
cannot find symbol
[ERROR] symbol : class AclEntryType
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6694) Beeline should provide a way to execute shell command as Hive CLI does
[ https://issues.apache.org/jira/browse/HIVE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-6694: -- Attachment: HIVE-6694.4.patch Patch #3 missing new files. Patch #4 fixes that. Beeline should provide a way to execute shell command as Hive CLI does -- Key: HIVE-6694 URL: https://issues.apache.org/jira/browse/HIVE-6694 Project: Hive Issue Type: Improvement Components: CLI, Clients Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Fix For: 0.14.0 Attachments: HIVE-6694.1.patch, HIVE-6694.1.patch, HIVE-6694.2.patch, HIVE-6694.3.patch, HIVE-6694.4.patch, HIVE-6694.patch Hive CLI allows a user to execute a shell command using ! notation. For instance, !cat myfile.txt. Being able to execute shell command may be important for some users. As a replacement, however, Beeline provides no such capability, possibly because ! notation is reserved for SQLLine commands. It's possible to provide this using a slightly syntactic variation such as !sh cat myfilie.txt. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 19830: HIVE-6694: Beeline should provide a way to execute shell command as Hive CLI does
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19830/#review46038 --- itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java https://reviews.apache.org/r/19830/#comment81204 Okay. Will create a followup JIRA for this. - Xuefu Zhang On June 17, 2014, 1:49 a.m., Xuefu Zhang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19830/ --- (Updated June 17, 2014, 1:49 a.m.) Review request for hive. Bugs: HIVE-6694 https://issues.apache.org/jira/browse/HIVE-6694 Repository: hive-git Description --- Introduce the new requested feature. Diffs - beeline/src/java/org/apache/hive/beeline/BeeLine.java dcdd6f2 beeline/src/java/org/apache/hive/beeline/Commands.java 94f74d2 beeline/src/main/resources/BeeLine.properties 697c29a cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 3357414 itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java 4d15e3c itests/util/pom.xml b38c55b itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 2974327 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 32c2fb0 ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 2ce4dbd ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java d2e122d Diff: https://reviews.apache.org/r/19830/diff/ Testing --- New test cases are added. Thanks, Xuefu Zhang
[jira] [Updated] (HIVE-6694) Beeline should provide a way to execute shell command as Hive CLI does
[ https://issues.apache.org/jira/browse/HIVE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-6694: -- Attachment: (was: HIVE-6694.4.patch) Beeline should provide a way to execute shell command as Hive CLI does -- Key: HIVE-6694 URL: https://issues.apache.org/jira/browse/HIVE-6694 Project: Hive Issue Type: Improvement Components: CLI, Clients Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Fix For: 0.14.0 Attachments: HIVE-6694.1.patch, HIVE-6694.1.patch, HIVE-6694.2.patch, HIVE-6694.3.patch, HIVE-6694.4.patch, HIVE-6694.patch Hive CLI allows a user to execute a shell command using ! notation. For instance, !cat myfile.txt. Being able to execute shell command may be important for some users. As a replacement, however, Beeline provides no such capability, possibly because ! notation is reserved for SQLLine commands. It's possible to provide this using a slightly syntactic variation such as !sh cat myfilie.txt. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6694) Beeline should provide a way to execute shell command as Hive CLI does
[ https://issues.apache.org/jira/browse/HIVE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-6694: -- Attachment: HIVE-6694.4.patch Beeline should provide a way to execute shell command as Hive CLI does -- Key: HIVE-6694 URL: https://issues.apache.org/jira/browse/HIVE-6694 Project: Hive Issue Type: Improvement Components: CLI, Clients Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Fix For: 0.14.0 Attachments: HIVE-6694.1.patch, HIVE-6694.1.patch, HIVE-6694.2.patch, HIVE-6694.3.patch, HIVE-6694.4.patch, HIVE-6694.patch Hive CLI allows a user to execute a shell command using ! notation. For instance, !cat myfile.txt. Being able to execute shell command may be important for some users. As a replacement, however, Beeline provides no such capability, possibly because ! notation is reserved for SQLLine commands. It's possible to provide this using a slightly syntactic variation such as !sh cat myfilie.txt. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 19830: HIVE-6694: Beeline should provide a way to execute shell command as Hive CLI does
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/19830/ --- (Updated June 18, 2014, 5:34 a.m.) Review request for hive. Bugs: HIVE-6694 https://issues.apache.org/jira/browse/HIVE-6694 Repository: hive-git Description --- Introduce the new requested feature. Diffs (updated) - beeline/src/java/org/apache/hive/beeline/BeeLine.java dcdd6f2 beeline/src/java/org/apache/hive/beeline/Commands.java 94f74d2 beeline/src/main/resources/BeeLine.properties 697c29a cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 3357414 common/src/java/org/apache/hadoop/hive/common/cli/ShellCmdExecutor.java PRE-CREATION common/src/java/org/apache/hive/common/util/StreamPrinter.java PRE-CREATION itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java 4d15e3c itests/util/pom.xml b38c55b itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 2974327 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5e5cf97 ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 2ce4dbd ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java d2e122d Diff: https://reviews.apache.org/r/19830/diff/ Testing --- New test cases are added. Thanks, Xuefu Zhang
