[jira] [Updated] (HIVE-9896) \N un-recognized in AVRO format Hive tables
[ https://issues.apache.org/jira/browse/HIVE-9896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Sundararajan Devaki updated HIVE-9896: - Description: We Sqooped (1.4.5) data from many RDBMS into HDFS in text format with options --null-non-string '\\N' --null-string '\\N'. When we load these into Hive tables in text format the \N is properly recognized as NULL and we are able to use SQL clauses such as IS NULL and IS NOT NULL against columns. However, when we convert the text files into AVRO (1.7.6) with SNAPPY compression and try to query using the above SQL clauses, the query does not return results as expected. Further, we have to use column_name = '\\N' or column_name '\\N' as a workaround. was: We Sqooped data from many RDBMS into HDFS in text format with options --null-non-string '\\N' --null-string '\\N'. When we load these into Hive tables in text format the \N is properly recognized as NULL and we are able to use SQL clauses such as IS NULL and IS NOT NULL against columns. However, when we convert the text files into AVRO (1.7.6) with SNAPPY compression and try to query using the above SQL clauses, the query does not return results as expected. Further, we have to use column_name = '\\N' or column_name '\\N' as a workaround. \N un-recognized in AVRO format Hive tables --- Key: HIVE-9896 URL: https://issues.apache.org/jira/browse/HIVE-9896 Project: Hive Issue Type: Bug Components: Database/Schema, File Formats, Hive Affects Versions: 0.13.0 Environment: CDH5.2.1, RHEL6.5, Java 7 Reporter: Madhan Sundararajan Devaki We Sqooped (1.4.5) data from many RDBMS into HDFS in text format with options --null-non-string '\\N' --null-string '\\N'. When we load these into Hive tables in text format the \N is properly recognized as NULL and we are able to use SQL clauses such as IS NULL and IS NOT NULL against columns. However, when we convert the text files into AVRO (1.7.6) with SNAPPY compression and try to query using the above SQL clauses, the query does not return results as expected. Further, we have to use column_name = '\\N' or column_name '\\N' as a workaround. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9870) Add JvmPauseMonitor threads to HMS and HS2 daemons
[ https://issues.apache.org/jira/browse/HIVE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HIVE-9870: -- Attachment: HIVE-9870.patch Sorry, missed the new try-catch block change in the previous attachment. Should work this time. Add JvmPauseMonitor threads to HMS and HS2 daemons -- Key: HIVE-9870 URL: https://issues.apache.org/jira/browse/HIVE-9870 Project: Hive Issue Type: Improvement Components: HiveServer2, Metastore Affects Versions: 1.1 Reporter: Harsh J Assignee: Harsh J Priority: Minor Attachments: HIVE-9870.patch, HIVE-9870.patch, HIVE-9870.patch The hadoop-common carries in it a nifty thread that prints GC or non-GC pauses within the JVM if it exceeds a specific threshold. This has been immeasurably useful in supporting several clusters, in identifying GC or other form of process pauses to be the root cause of some event being investigated. The HMS and HS2 daemons are good targets for running similar threads within it. It can be loaded in an if-available style. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9855) Runtime skew join doesn't work when skewed data only exists in big table
[
https://issues.apache.org/jira/browse/HIVE-9855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352955#comment-14352955
]
Rui Li commented on HIVE-9855:
--
Merged into spark.
Runtime skew join doesn't work when skewed data only exists in big table
Key: HIVE-9855
URL: https://issues.apache.org/jira/browse/HIVE-9855
Project: Hive
Issue Type: Bug
Reporter: Rui Li
Assignee: Rui Li
Fix For: 1.2.0
Attachments: HIVE-9855.1.patch
To reproduce, enable runtime skew join and then join two tables that skewed
data only exists in one of them. The task will fail with the following
exception:
{noformat}
Error: java.lang.RuntimeException: Hive Runtime Error while closing
operators: java.io.IOException: Unable to rename output to: hdfs://..
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9889) Merge trunk to Spark branch 3/6/2015 [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353056#comment-14353056 ] Sergio Peña commented on HIVE-9889: --- I'll take a look. Merge trunk to Spark branch 3/6/2015 [Spark Branch] --- Key: HIVE-9889 URL: https://issues.apache.org/jira/browse/HIVE-9889 Project: Hive Issue Type: Task Components: Spark Reporter: Xuefu Zhang Assignee: Xuefu Zhang Attachments: HIVE-9889.1-spark.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9894) Use new parquet Types API builder to construct DATE data type
[ https://issues.apache.org/jira/browse/HIVE-9894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353055#comment-14353055 ] Sergio Peña commented on HIVE-9894: --- +1 Use new parquet Types API builder to construct DATE data type - Key: HIVE-9894 URL: https://issues.apache.org/jira/browse/HIVE-9894 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen Attachments: HIVE-9894.patch The DATE type was implemented in HIVE-8119. And new parquet Types API builder was used in HIVE-9657 for all data types. But DATE is missed. We should also use new Types API for DATE type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9659) 'Error while trying to create table container' occurs during hive query case execution when hive.optimize.skewjoin set to 'true' [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-9659:
-
Attachment: HIVE-9659.3-spark.patch
Address RB comments
'Error while trying to create table container' occurs during hive query case
execution when hive.optimize.skewjoin set to 'true' [Spark Branch]
---
Key: HIVE-9659
URL: https://issues.apache.org/jira/browse/HIVE-9659
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Xin Hao
Assignee: Rui Li
Attachments: HIVE-9659.1-spark.patch, HIVE-9659.2-spark.patch,
HIVE-9659.3-spark.patch
We found that 'Error while trying to create table container' occurs during
Big-Bench Q12 case execution when hive.optimize.skewjoin set to 'true'.
If hive.optimize.skewjoin set to 'false', the case could pass.
How to reproduce:
1. set hive.optimize.skewjoin=true;
2. Run BigBench case Q12 and it will fail.
Check the executor log (e.g. /usr/lib/spark/work/app-/2/stderr) and you
will found error 'Error while trying to create table container' in the log
and also a NullPointerException near the end of the log.
(a) Detail error message for 'Error while trying to create table container':
{noformat}
15/02/12 01:29:49 ERROR SparkMapRecordHandler: Error processing row:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Error while trying to
create table container
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Error while trying to
create table container
at
org.apache.hadoop.hive.ql.exec.spark.HashTableLoader.load(HashTableLoader.java:118)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:193)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:219)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1051)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:486)
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:141)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:47)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:98)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
at
org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:217)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:65)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error while
trying to create table container
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainerSerDe.load(MapJoinTableContainerSerDe.java:158)
at
org.apache.hadoop.hive.ql.exec.spark.HashTableLoader.load(HashTableLoader.java:115)
... 21 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error, not a
directory:
hdfs://bhx1:8020/tmp/hive/root/d22ef465-bff5-4edb-a822-0a9f1c25b66c/hive_2015-02-12_01-28-10_008_6897031694580088767-1/-mr-10009/HashTable-Stage-6/MapJoin-mapfile01--.hashtable
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainerSerDe.load(MapJoinTableContainerSerDe.java:106)
... 22 more
15/02/12 01:29:49 INFO SparkRecordHandler: maximum memory = 40939028480
15/02/12 01:29:49 INFO PerfLogger: PERFLOG method=SparkInitializeOperators
from=org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler
{noformat}
(b) Detail error message for
[jira] [Updated] (HIVE-9601) New Beeline queries will hang If Beeline terminates in-properly [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-9601: -- Attachment: HIVE-9601.1-spark.patch Re-attach the patch to trigger QA test. New Beeline queries will hang If Beeline terminates in-properly [Spark Branch] -- Key: HIVE-9601 URL: https://issues.apache.org/jira/browse/HIVE-9601 Project: Hive Issue Type: Bug Components: Spark Reporter: Szehon Ho Assignee: Jimmy Xiang Fix For: spark-branch Attachments: HIVE-9601.1-spark.patch, HIVE-9601.1-spark.patch User session's Spark application seems to stay around if beeline is not quit properly (!quit) because the user is not disconnected. If Beeline is started, it will create a new Spark application which will hang waiting for the first one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9897) Issue a warning when using an existing table/view name as an alias in a with statement.
[
https://issues.apache.org/jira/browse/HIVE-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mario Konschake updated HIVE-9897:
--
Description:
Consider the following query:
{code:sql}
WITH
table_a AS (
SELECT
'johndoe' AS name
FROM
my_table
)
SELECT
DISTINCT name
FROM
table_a;
{code}
Observation:
If a table or a view with name `table_a` exists it is used instead of the one
defined in the WITH statement.
Expectation:
As the expectation is ambiguous (using the alias in the WITH statement vs.
using the existing table) issuing a warning when using a existing name in a
WITH statement is recommended.
was:
Consider the following query:
{code:sql}
WITH
table_a AS (
SELECT
'johndoe' AS name
FROM
my_table
)
SELECT
DISTINCT name
FROM
table_a;
{code:sql}
Observation:
If a table or a view with name `table_a` exists it is used instead of the one
defined in the WITH statement.
Expectation:
As the expectation is ambiguous (using the alias in the WITH statement vs.
using the existing table) issuing a warning when using a existing name in a
WITH statement is recommended.
Issue a warning when using an existing table/view name as an alias in a with
statement.
Key: HIVE-9897
URL: https://issues.apache.org/jira/browse/HIVE-9897
Project: Hive
Issue Type: Improvement
Components: Hive
Affects Versions: 0.13.1
Environment: cdh5.3.0
Reporter: Mario Konschake
Priority: Minor
Consider the following query:
{code:sql}
WITH
table_a AS (
SELECT
'johndoe' AS name
FROM
my_table
)
SELECT
DISTINCT name
FROM
table_a;
{code}
Observation:
If a table or a view with name `table_a` exists it is used instead of the one
defined in the WITH statement.
Expectation:
As the expectation is ambiguous (using the alias in the WITH statement vs.
using the existing table) issuing a warning when using a existing name in a
WITH statement is recommended.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9897) Issue a warning when using an existing table/view name as an alias in a with statement.
[
https://issues.apache.org/jira/browse/HIVE-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mario Konschake updated HIVE-9897:
--
Description:
Consider the following query:
{code:sql}
WITH
table_a AS (
SELECT
'johndoe' AS name
FROM
my_table
)
SELECT
DISTINCT name
FROM
table_a;
{code:sql}
Observation:
If a table or a view with name `table_a` exists it is used instead of the one
defined in the WITH statement.
Expectation:
As the expectation is ambiguous (using the alias in the WITH statement vs.
using the existing table) issuing a warning when using a existing name in a
WITH statement is recommended.
was:
Consider the following query:
WITH
table_a AS (
SELECT
'johndoe' AS name
FROM
my_table
)
SELECT
DISTINCT name
FROM
table_a;
Observation:
If a table or a view with name `table_a` exists it is used instead of the one
defined in the WITH statement.
Expectation:
As the expectation is ambiguous (using the alias in the WITH statement vs.
using the existing table) issuing a warning when using a existing name in a
WITH statement is recommended.
Issue a warning when using an existing table/view name as an alias in a with
statement.
Key: HIVE-9897
URL: https://issues.apache.org/jira/browse/HIVE-9897
Project: Hive
Issue Type: Improvement
Components: Hive
Affects Versions: 0.13.1
Environment: cdh5.3.0
Reporter: Mario Konschake
Priority: Minor
Consider the following query:
{code:sql}
WITH
table_a AS (
SELECT
'johndoe' AS name
FROM
my_table
)
SELECT
DISTINCT name
FROM
table_a;
{code:sql}
Observation:
If a table or a view with name `table_a` exists it is used instead of the one
defined in the WITH statement.
Expectation:
As the expectation is ambiguous (using the alias in the WITH statement vs.
using the existing table) issuing a warning when using a existing name in a
WITH statement is recommended.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9808) LLAP: Push work into daemons instead of the current pull
[ https://issues.apache.org/jira/browse/HIVE-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-9808: - Attachment: HIVE-9808.2.txt Rebased patch. Will commit shortly; this one was painful to rebase. There's some UGI / closeAllForFileSystem changes which will need to be worked on in a follow up. LLAP: Push work into daemons instead of the current pull Key: HIVE-9808 URL: https://issues.apache.org/jira/browse/HIVE-9808 Project: Hive Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: llap Attachments: HIVE-9808.1.txt, HIVE-9808.2.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9808) LLAP: Push work into daemons instead of the current pull
[ https://issues.apache.org/jira/browse/HIVE-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth resolved HIVE-9808. -- Resolution: Fixed LLAP: Push work into daemons instead of the current pull Key: HIVE-9808 URL: https://issues.apache.org/jira/browse/HIVE-9808 Project: Hive Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: llap Attachments: HIVE-9808.1.txt, HIVE-9808.2.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9897) Issue a warning when using an existing table/view name as an alias in a with statement.
[
https://issues.apache.org/jira/browse/HIVE-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353208#comment-14353208
]
RAUNAK JHAWAR commented on HIVE-9897:
-
I did a quick check on Hive on HDP:
Assumption users is an existing table in Hive (database: default)
{code:sql}
with users as (select * from products limit 10)select * from users limit 10;
{code}
The effective output is same as that of doing a {code:sql} select * from users
limit 10{code}
The products data is not shown as the output neither any warning is issued that
there already is an object by name users
Issue a warning when using an existing table/view name as an alias in a with
statement.
Key: HIVE-9897
URL: https://issues.apache.org/jira/browse/HIVE-9897
Project: Hive
Issue Type: Improvement
Components: Hive
Affects Versions: 0.13.1
Environment: cdh5.3.0
Reporter: Mario Konschake
Priority: Minor
Consider the following query:
WITH
table_a AS (
SELECT
'johndoe' AS name
FROM
my_table
)
SELECT
DISTINCT name
FROM
table_a;
Observation:
If a table or a view with name `table_a` exists it is used instead of the one
defined in the WITH statement.
Expectation:
As the expectation is ambiguous (using the alias in the WITH statement vs.
using the existing table) issuing a warning when using a existing name in a
WITH statement is recommended.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9537) string expressions on a fixed length character do not preserve trailing spaces
[
https://issues.apache.org/jira/browse/HIVE-9537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353274#comment-14353274
]
Aihua Xu commented on HIVE-9537:
HIVE-9899 doc bug is opened to keep track of the doc change. I'm closing this
out. Let me know if you have other concerns.
string expressions on a fixed length character do not preserve trailing spaces
--
Key: HIVE-9537
URL: https://issues.apache.org/jira/browse/HIVE-9537
Project: Hive
Issue Type: Bug
Components: SQL
Reporter: N Campbell
Assignee: Aihua Xu
When a string expression such as upper or lower is applied to a fixed length
column the trailing spaces of the fixed length character are not preserved.
{code:sql}
CREATE TABLE if not exists TCHAR (
RNUM int,
CCHAR char(32)
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE;
{code}
{{cchar}} as a {{char(32)}}.
{code:sql}
select cchar, concat(cchar, cchar), concat(lower(cchar), cchar),
concat(upper(cchar), cchar)
from tchar;
{code}
0|\N
1|
2|
3|BB
4|EE
5|FF
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9775) LLAP: Add a MiniLLAPCluster for tests
[ https://issues.apache.org/jira/browse/HIVE-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-9775: - Attachment: HIVE-9775.2.patch Re-based patch. LLAP: Add a MiniLLAPCluster for tests - Key: HIVE-9775 URL: https://issues.apache.org/jira/browse/HIVE-9775 Project: Hive Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: llap Attachments: HIVE-9775.1.patch, HIVE-9775.2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9273) Add option to fire metastore event on insert
[ https://issues.apache.org/jira/browse/HIVE-9273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-9273: --- Issue Type: Sub-task (was: New Feature) Parent: HIVE-7973 Add option to fire metastore event on insert Key: HIVE-9273 URL: https://issues.apache.org/jira/browse/HIVE-9273 Project: Hive Issue Type: Sub-task Reporter: Alan Gates Assignee: Alan Gates Labels: TODOC1.2 Fix For: 1.2.0 Attachments: HIVE-9273.2.patch, HIVE-9273.patch HIVE-9271 adds the ability for the client to request firing metastore events. This can be used in the MoveTask to fire events when an insert is done that does not add partitions to a table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9550) ObjectStore.getNextNotification() can return events inside NotificationEventResponse as null which conflicts with its thrift required tag
[
https://issues.apache.org/jira/browse/HIVE-9550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-9550:
---
Issue Type: Sub-task (was: Bug)
Parent: HIVE-7973
ObjectStore.getNextNotification() can return events inside
NotificationEventResponse as null which conflicts with its thrift required
tag
---
Key: HIVE-9550
URL: https://issues.apache.org/jira/browse/HIVE-9550
Project: Hive
Issue Type: Sub-task
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan
Fix For: 1.2.0
Attachments: HIVE-9550.patch
Per hive_metastore.thrift, the events list inside NotificationEventResponse
is a required field that cannot be null.
{code}
struct NotificationEventResponse {
1: required listNotificationEvent events,
}
{code}
However, per ObjectStore.java, this events field can be uninitialized if the
events retrieved from the metastore is empty instead of null:
{code}
NotificationEventResponse result = new NotificationEventResponse();
int maxEvents = rqst.getMaxEvents() 0 ? rqst.getMaxEvents() :
Integer.MAX_VALUE;
int numEvents = 0;
while (i.hasNext() numEvents++ maxEvents) {
result.addToEvents(translateDbToThrift(i.next()));
}
return result;
{code}
The fix is simple enough - we need to call result.setEvents(new
ArrayListNotificationEvent()) before we begin the iteration to do
result.addToEvents(...).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9658) Reduce parquet memory use by bypassing java primitive objects on ETypeConverter
[ https://issues.apache.org/jira/browse/HIVE-9658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-9658: -- Attachment: HIVE-9658.2.patch Reduce parquet memory use by bypassing java primitive objects on ETypeConverter --- Key: HIVE-9658 URL: https://issues.apache.org/jira/browse/HIVE-9658 Project: Hive Issue Type: Sub-task Reporter: Sergio Peña Assignee: Sergio Peña Attachments: HIVE-9658.1.patch, HIVE-9658.2.patch The ETypeConverter class passes Writable objects to the collection converters in order to be read later by the map/reduce functions. These objects are all wrapped in a unique ArrayWritable object. We can save some memory by returning the java primitive objects instead in order to prevent memory allocation. The only writable object needed by map/reduce is ArrayWritable. If we create another writable class where to store primitive objects (Object), then we can stop using all primitive wirtables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9577) Fire insert event on HCatalog appends
[ https://issues.apache.org/jira/browse/HIVE-9577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-9577: --- Issue Type: Sub-task (was: Bug) Parent: HIVE-7973 Fire insert event on HCatalog appends - Key: HIVE-9577 URL: https://issues.apache.org/jira/browse/HIVE-9577 Project: Hive Issue Type: Sub-task Components: HCatalog Affects Versions: 1.2.0 Reporter: Sushanth Sowmyan Assignee: Alan Gates HIVE-9271 added the ability to fire a listener event on the metastore to notify on inserts, and HIVE-9273 added this ability to register data appends in hive to trigger this event. HCatalog appends should also trigger this event. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9601) New Beeline queries will hang If Beeline terminates in-properly [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-9601: -- Attachment: HIVE-9601.2-spark.patch Attached v2 that fixed unit test failures. The patch works only for beeline client using binary Thrift transport. New Beeline queries will hang If Beeline terminates in-properly [Spark Branch] -- Key: HIVE-9601 URL: https://issues.apache.org/jira/browse/HIVE-9601 Project: Hive Issue Type: Bug Components: Spark Reporter: Szehon Ho Assignee: Jimmy Xiang Fix For: spark-branch Attachments: HIVE-9601.1-spark.patch, HIVE-9601.1-spark.patch, HIVE-9601.2-spark.patch User session's Spark application seems to stay around if beeline is not quit properly (!quit) because the user is not disconnected. If Beeline is started, it will create a new Spark application which will hang waiting for the first one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9882) Add jar/file doesn't work with yarn-cluster mode [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353633#comment-14353633
]
Hive QA commented on HIVE-9882:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12703340/HIVE-9882.1-spark.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/771/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/771/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-771/
Messages:
{noformat}
This message was trimmed, see log for full details
Reverted
'hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java'
Reverted
'hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java'
Reverted
'metastore/src/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java'
Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java'
Reverted
'metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java'
Reverted
'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java'
Reverted
'metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java'
Reverted
'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java'
Reverted 'bin/hive'
Reverted 'data/files/parquet_types.txt'
Reverted
'itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java'
Reverted
'itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java'
Reverted 'LICENSE'
Reverted
'hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java'
Reverted
'hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatSplit.java'
Reverted
'hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java'
Reverted
'hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/MessageFactory.java'
Reverted
'hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/JobState.java'
Reverted
'hcatalog/webhcat/java-client/src/main/java/org/apache/hive/hcatalog/api/HCatClientHMSImpl.java'
Reverted 'hcatalog/webhcat/java-client/pom.xml'
Reverted
'accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/PrimitiveComparisonFilter.java'
Reverted 'accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/Utils.java'
Reverted
'accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/AccumuloSerDeParameters.java'
Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java'
Reverted 'common/src/java/org/apache/hadoop/hive/common/JavaUtils.java'
Reverted
'service/src/java/org/apache/hive/service/server/ThreadFactoryWithGarbageCleanup.java'
Reverted
'serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerializer.java'
Reverted 'serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerializer.java'
Reverted 'beeline/src/test/org/apache/hive/beeline/TestBeelineArgParsing.java'
Reverted 'beeline/src/java/org/apache/hive/beeline/BeeLine.java'
Reverted
'beeline/src/java/org/apache/hive/beeline/DeprecatedSeparatedValuesOutputFormat.java'
Reverted 'beeline/src/java/org/apache/hive/beeline/DatabaseConnection.java'
Reverted 'beeline/src/java/org/apache/hive/beeline/Commands.java'
Reverted 'beeline/src/java/org/apache/hive/beeline/ClassNameCompleter.java'
Reverted
'beeline/src/java/org/apache/hive/beeline/SeparatedValuesOutputFormat.java'
Reverted 'beeline/src/main/resources/BeeLine.properties'
Reverted 'ql/src/test/results/clientnegative/udf_add_months_error_2.q.out'
Reverted 'ql/src/test/results/clientnegative/udf_next_day_error_1.q.out'
Reverted 'ql/src/test/results/clientnegative/parquet_date.q.out'
Reverted 'ql/src/test/results/clientnegative/udf_next_day_error_2.q.out'
Reverted 'ql/src/test/results/clientnegative/udf_last_day_error_1.q.out'
Reverted 'ql/src/test/results/clientnegative/udf_last_day_error_2.q.out'
Reverted 'ql/src/test/results/clientnegative/invalid_arithmetic_type.q.out'
Reverted 'ql/src/test/results/clientnegative/udf_add_months_error_1.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join27.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join13.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join4.q.out'
Reverted 'ql/src/test/results/clientpositive/ql_rewrite_gbtoidx_cbo_1.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd_join_filter.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out'
Reverted 'ql/src/test/results/clientpositive/subquery_in_having.q.out'
Reverted
'ql/src/test/results/clientpositive/tez/dynamic_partition_pruning.q.out'
Reverted
[jira] [Updated] (HIVE-4256) JDBC2 HiveConnection does not use the specified database
[ https://issues.apache.org/jira/browse/HIVE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick White updated HIVE-4256: - Affects Version/s: 0.12.0 JDBC2 HiveConnection does not use the specified database Key: HIVE-4256 URL: https://issues.apache.org/jira/browse/HIVE-4256 Project: Hive Issue Type: Bug Components: HiveServer2, JDBC Affects Versions: 0.11.0, 0.12.0 Reporter: Chris Drome Assignee: Anandha L Ranganathan Fix For: 0.13.0 Attachments: HIVE-4256.1.patch, HIVE-4256.2.patch, HIVE-4256.3.patch, HIVE-4256.4.patch, HIVE-4256.5.patch, HIVE-4256.6.patch, HIVE-4256.patch HiveConnection ignores the database specified in the connection string when configuring the connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6617) Reduce ambiguity in grammar
[ https://issues.apache.org/jira/browse/HIVE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-6617: -- Attachment: HIVE-6617.24.patch Reduce ambiguity in grammar --- Key: HIVE-6617 URL: https://issues.apache.org/jira/browse/HIVE-6617 Project: Hive Issue Type: Task Reporter: Ashutosh Chauhan Assignee: Pengcheng Xiong Attachments: HIVE-6617.01.patch, HIVE-6617.02.patch, HIVE-6617.03.patch, HIVE-6617.04.patch, HIVE-6617.05.patch, HIVE-6617.06.patch, HIVE-6617.07.patch, HIVE-6617.08.patch, HIVE-6617.09.patch, HIVE-6617.10.patch, HIVE-6617.11.patch, HIVE-6617.12.patch, HIVE-6617.13.patch, HIVE-6617.14.patch, HIVE-6617.15.patch, HIVE-6617.16.patch, HIVE-6617.17.patch, HIVE-6617.18.patch, HIVE-6617.19.patch, HIVE-6617.20.patch, HIVE-6617.21.patch, HIVE-6617.22.patch, HIVE-6617.23.patch, HIVE-6617.24.patch, parser.png CLEAR LIBRARY CACHE As of today, antlr reports 214 warnings. Need to bring down this number, ideally to 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9271) Add ability for client to request metastore to fire an event
[ https://issues.apache.org/jira/browse/HIVE-9271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-9271: --- Issue Type: Sub-task (was: New Feature) Parent: HIVE-7973 Add ability for client to request metastore to fire an event Key: HIVE-9271 URL: https://issues.apache.org/jira/browse/HIVE-9271 Project: Hive Issue Type: Sub-task Components: Metastore Reporter: Alan Gates Assignee: Alan Gates Fix For: 1.2.0 Attachments: HIVE-9271.patch Currently all events in Hive are fired by the metastore. However, there are events that only the client fully understands, such as DML operations. There should be a way for the client to request the metastore to fire a particular event. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9175) Add alters to list of events handled by NotificationListener
[ https://issues.apache.org/jira/browse/HIVE-9175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-9175: --- Issue Type: Sub-task (was: New Feature) Parent: HIVE-7973 Add alters to list of events handled by NotificationListener Key: HIVE-9175 URL: https://issues.apache.org/jira/browse/HIVE-9175 Project: Hive Issue Type: Sub-task Components: HCatalog Reporter: Alan Gates Assignee: Alan Gates Labels: TODOC15 Fix For: 1.1.0 Attachments: HIVE-9175.patch HCatalog currently doesn't implement onAlterTable and onAlterPartition. It should. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9658) Reduce parquet memory use by bypassing java primitive objects on ETypeConverter
[ https://issues.apache.org/jira/browse/HIVE-9658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353576#comment-14353576 ] Sergio Peña commented on HIVE-9658: --- Latest changes on 'trunk' need to be merged on 'parquet' branch so that this patch can be tested. Reduce parquet memory use by bypassing java primitive objects on ETypeConverter --- Key: HIVE-9658 URL: https://issues.apache.org/jira/browse/HIVE-9658 Project: Hive Issue Type: Sub-task Reporter: Sergio Peña Assignee: Sergio Peña Attachments: HIVE-9658.1.patch, HIVE-9658.2.patch The ETypeConverter class passes Writable objects to the collection converters in order to be read later by the map/reduce functions. These objects are all wrapped in a unique ArrayWritable object. We can save some memory by returning the java primitive objects instead in order to prevent memory allocation. The only writable object needed by map/reduce is ArrayWritable. If we create another writable class where to store primitive objects (Object), then we can stop using all primitive wirtables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9889) Merge trunk to Spark branch 3/6/2015 [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-9889: -- Attachment: HIVE-9889.2-spark.patch Regenerate the patch since some patches were merged individually. Merge trunk to Spark branch 3/6/2015 [Spark Branch] --- Key: HIVE-9889 URL: https://issues.apache.org/jira/browse/HIVE-9889 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Xuefu Zhang Assignee: Xuefu Zhang Attachments: HIVE-9889.1-spark.patch, HIVE-9889.2-spark.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-1161) Hive Replication
[ https://issues.apache.org/jira/browse/HIVE-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353377#comment-14353377 ] SHAILESH PILARE commented on HIVE-1161: --- Ahutosh, Currently i am working on best feasible approach, out of Two available approaches a) Use Hive Import/Export along with Distcp with some background auditing in default DB . b) Using Distcp for data and Database replication for metastore transfer . After successful completion of POC i would update the design doc . Hive Replication Key: HIVE-1161 URL: https://issues.apache.org/jira/browse/HIVE-1161 Project: Hive Issue Type: New Feature Components: Contrib Reporter: Edward Capriolo Assignee: SHAILESH PILARE Priority: Minor Users may want to replicate data between two distinct hadoop clusters or two hive warehouses on the same cluster. Users may want to replicate entire catalogs or possibly on a table by table basis. Should this process be batch driven or a be a full time running application? What are some practical requirements, what are the limitations? Comments? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9902) Map join small table files need more replications [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HIVE-9902:
--
Attachment: HIVE-9902.1-spark.patch
Map join small table files need more replications [Spark Branch]
Key: HIVE-9902
URL: https://issues.apache.org/jira/browse/HIVE-9902
Project: Hive
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: spark-branch
Attachments: HIVE-9902.1-spark.patch
We have
{noformat}
replication = (short) Math.min(MIN_REPLICATION, numOfPartitions);
{noformat}
It should be
{noformat}
replication = (short) Math.max(MIN_REPLICATION, numOfPartitions);
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[ https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-9664: - Assignee: Anant Nag Hive add jar command should be able to download and add jars from a repository Key: HIVE-9664 URL: https://issues.apache.org/jira/browse/HIVE-9664 Project: Hive Issue Type: Improvement Affects Versions: 0.14.0 Reporter: Anant Nag Assignee: Anant Nag Labels: hive, patch Attachments: HIVE-9664.patch Currently Hive's add jar command takes a local path to the dependency jar. This clutters the local file-system as users may forget to remove this jar later It would be nice if Hive supported a Gradle like notation to download the jar from a repository. Example: add jar org:module:version It should also be backward compatible and should take jar from the local file-system as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9851) org.apache.hadoop.hive.serde2.avro.AvroSerializer should use org.apache.avro.generic.GenericData.Array when serializing a list
[ https://issues.apache.org/jira/browse/HIVE-9851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-9851: - Fix Version/s: 1.2.0 org.apache.hadoop.hive.serde2.avro.AvroSerializer should use org.apache.avro.generic.GenericData.Array when serializing a list -- Key: HIVE-9851 URL: https://issues.apache.org/jira/browse/HIVE-9851 Project: Hive Issue Type: Bug Components: Hive, Serializers/Deserializers Reporter: Ratandeep Ratti Fix For: 1.2.0 Attachments: HIVE-9851.patch Currently AvroSerializer uses java.util.ArrayList for serializing a list in Hive. This causes problems when we need to convert the avro object into some other representation say a tuple in Pig. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9851) org.apache.hadoop.hive.serde2.avro.AvroSerializer should use org.apache.avro.generic.GenericData.Array when serializing a list
[ https://issues.apache.org/jira/browse/HIVE-9851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-9851: - Assignee: Ratandeep Ratti org.apache.hadoop.hive.serde2.avro.AvroSerializer should use org.apache.avro.generic.GenericData.Array when serializing a list -- Key: HIVE-9851 URL: https://issues.apache.org/jira/browse/HIVE-9851 Project: Hive Issue Type: Bug Components: Hive, Serializers/Deserializers Reporter: Ratandeep Ratti Assignee: Ratandeep Ratti Fix For: 1.2.0 Attachments: HIVE-9851.patch Currently AvroSerializer uses java.util.ArrayList for serializing a list in Hive. This causes problems when we need to convert the avro object into some other representation say a tuple in Pig. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9871) Print spark job id in history file [spark branch]
[ https://issues.apache.org/jira/browse/HIVE-9871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353468#comment-14353468 ] Xuefu Zhang commented on HIVE-9871: --- [~chinnalalam], thanks for working on this. Patch looks good, but I'm wondering if you can come up with a better name for the private method added. Something like recordJobId() or addToHistory(), etc. Print spark job id in history file [spark branch] - Key: HIVE-9871 URL: https://issues.apache.org/jira/browse/HIVE-9871 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-9871.1-spark.patch Maintain the spark job id in history file for the corresponding queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9658) Reduce parquet memory use by bypassing java primitive objects on ETypeConverter
[
https://issues.apache.org/jira/browse/HIVE-9658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353565#comment-14353565
]
Hive QA commented on HIVE-9658:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12703476/HIVE-9658.2.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2979/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2979/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2979/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-2979/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted
'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java'
Reverted 'service/src/java/org/apache/hive/service/server/HiveServer2.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20S/target
shims/0.23/target shims/aggregator/target shims/common/target
shims/scheduler/target packaging/target hbase-handler/target testutils/target
jdbc/target metastore/target
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java.orig
itests/target itests/thirdparty itests/hcatalog-unit/target
itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target
itests/hive-minikdc/target itests/hive-jmh/target itests/hive-unit/target
itests/custom-serde/target itests/util/target itests/qtest-spark/target
hcatalog/target hcatalog/core/target hcatalog/streaming/target
hcatalog/server-extensions/target hcatalog/webhcat/svr/target
hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target
accumulo-handler/target hwi/target common/target common/src/gen
spark-client/target service/target contrib/target serde/target beeline/target
odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1665350.
At revision 1665350.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12703476 - PreCommit-HIVE-TRUNK-Build
Reduce parquet memory use by bypassing java primitive objects on
ETypeConverter
---
Key: HIVE-9658
URL: https://issues.apache.org/jira/browse/HIVE-9658
Project: Hive
Issue Type: Sub-task
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-9658.1.patch, HIVE-9658.2.patch
The ETypeConverter class passes Writable objects to the collection converters
in order to be read later by the map/reduce functions. These objects are all
wrapped in a unique ArrayWritable object.
We can save some memory by returning the java primitive objects instead in
order to prevent memory allocation. The only writable object needed by
map/reduce is ArrayWritable. If we create another writable
[jira] [Updated] (HIVE-9892) various MSSQL upgrade scripts don't work
[ https://issues.apache.org/jira/browse/HIVE-9892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-9892: --- Attachment: HIVE-9892.ADDENDUM.patch [~thejas] can you take a look at addendum? The pre-create script now conflicts with some other part of upgrade script various MSSQL upgrade scripts don't work Key: HIVE-9892 URL: https://issues.apache.org/jira/browse/HIVE-9892 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.13.0, 0.14.0, 1.0.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 1.1 Attachments: HIVE-9892.ADDENDUM.patch, HIVE-9892.patch * Issue with GO statement when run through schematool - it results in syntax error. * the create if not exists logic for PART_COL_STATS wasn't working NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9903) Update calcite version
[ https://issues.apache.org/jira/browse/HIVE-9903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-9903: --- Attachment: HIVE-9903.patch Update calcite version -- Key: HIVE-9903 URL: https://issues.apache.org/jira/browse/HIVE-9903 Project: Hive Issue Type: Task Components: CBO, Logical Optimizer Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-9903.patch Calcite-1.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9852) virtual column “INPUT__FILE__NAME” does not exclude files before map
[ https://issues.apache.org/jira/browse/HIVE-9852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353912#comment-14353912 ] Nickolay Ponomarev commented on HIVE-9852: -- This appears to be a duplicate of HIVE-1662. virtual column “INPUT__FILE__NAME” does not exclude files before map Key: HIVE-9852 URL: https://issues.apache.org/jira/browse/HIVE-9852 Project: Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.13.1 Reporter: Zhang Jingpeng virtual column “INPUT__FILE__NAME” does not exclude files before map . If I have three files I only want it to deal one file of three files, Then the map count will be 3 and will read all three files , so It's only tell what's file the line in and then filter it but not filter before map input path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9569) Enable more unit tests for UNION ALL [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao updated HIVE-9569: --- Attachment: HIVE-9569.5.patch Forgot about this one... Attaching a new patch. Enable more unit tests for UNION ALL [Spark Branch] --- Key: HIVE-9569 URL: https://issues.apache.org/jira/browse/HIVE-9569 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chao Assignee: Chao Attachments: HIVE-9569.1-spark.patch, HIVE-9569.1.patch, HIVE-9569.2.patch, HIVE-9569.3.patch, HIVE-9569.4.patch, HIVE-9569.5.patch Currently, we only enabled a subset of all the union tests. We should try to enable the rest, and see if there's any issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6617) Reduce ambiguity in grammar
[
https://issues.apache.org/jira/browse/HIVE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353868#comment-14353868
]
Hive QA commented on HIVE-6617:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12703503/HIVE-6617.24.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 7761 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_innerjoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_innerjoin
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2980/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2980/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2980/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12703503 - PreCommit-HIVE-TRUNK-Build
Reduce ambiguity in grammar
---
Key: HIVE-6617
URL: https://issues.apache.org/jira/browse/HIVE-6617
Project: Hive
Issue Type: Task
Reporter: Ashutosh Chauhan
Assignee: Pengcheng Xiong
Attachments: HIVE-6617.01.patch, HIVE-6617.02.patch,
HIVE-6617.03.patch, HIVE-6617.04.patch, HIVE-6617.05.patch,
HIVE-6617.06.patch, HIVE-6617.07.patch, HIVE-6617.08.patch,
HIVE-6617.09.patch, HIVE-6617.10.patch, HIVE-6617.11.patch,
HIVE-6617.12.patch, HIVE-6617.13.patch, HIVE-6617.14.patch,
HIVE-6617.15.patch, HIVE-6617.16.patch, HIVE-6617.17.patch,
HIVE-6617.18.patch, HIVE-6617.19.patch, HIVE-6617.20.patch,
HIVE-6617.21.patch, HIVE-6617.22.patch, HIVE-6617.23.patch,
HIVE-6617.24.patch, parser.png
CLEAR LIBRARY CACHE
As of today, antlr reports 214 warnings. Need to bring down this number,
ideally to 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353949#comment-14353949 ] Thiruvel Thirumoolan commented on HIVE-9582: Thanks Sushanth. Thejas looked at the patch and gave a first level review. I have addressed his comments and uploaded the patch. HCatalog should use IMetaStoreClient interface -- Key: HIVE-9582 URL: https://issues.apache.org/jira/browse/HIVE-9582 Project: Hive Issue Type: Sub-task Components: HCatalog, Metastore Affects Versions: 0.14.0, 0.13.1 Reporter: Thiruvel Thirumoolan Assignee: Thiruvel Thirumoolan Labels: hcatalog, metastore, rolling_upgrade Fix For: 0.14.1 Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, HIVE-9582.4.patch, HIVE-9583.1.patch Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. Hence during a failure, the client retries and possibly succeeds. But HCatalog has long been using HiveMetaStoreClient directly and hence failures are costly, especially if they are during the commit stage of a job. Its also not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9678) create timediff UDF
[
https://issues.apache.org/jira/browse/HIVE-9678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov updated HIVE-9678:
--
Attachment: HIVE-9678.4.patch
patch #4 again
create timediff UDF
---
Key: HIVE-9678
URL: https://issues.apache.org/jira/browse/HIVE-9678
Project: Hive
Issue Type: Improvement
Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Attachments: HIVE-9678.1.patch, HIVE-9678.2.patch, HIVE-9678.3.patch,
HIVE-9678.4.patch, HIVE-9678.4.patch
MySQL has very useful function timediff. We should have it in Hive
{code}
select timediff('2015-02-12 05:09:07.140', '2015-02-12 01:18:20');
OK
03:50:47.140
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-6617) Reduce ambiguity in grammar
[ https://issues.apache.org/jira/browse/HIVE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-6617: -- Attachment: HIVE-6617.25.patch update golden file for keyword inner Reduce ambiguity in grammar --- Key: HIVE-6617 URL: https://issues.apache.org/jira/browse/HIVE-6617 Project: Hive Issue Type: Task Reporter: Ashutosh Chauhan Assignee: Pengcheng Xiong Attachments: HIVE-6617.01.patch, HIVE-6617.02.patch, HIVE-6617.03.patch, HIVE-6617.04.patch, HIVE-6617.05.patch, HIVE-6617.06.patch, HIVE-6617.07.patch, HIVE-6617.08.patch, HIVE-6617.09.patch, HIVE-6617.10.patch, HIVE-6617.11.patch, HIVE-6617.12.patch, HIVE-6617.13.patch, HIVE-6617.14.patch, HIVE-6617.15.patch, HIVE-6617.16.patch, HIVE-6617.17.patch, HIVE-6617.18.patch, HIVE-6617.19.patch, HIVE-6617.20.patch, HIVE-6617.21.patch, HIVE-6617.22.patch, HIVE-6617.23.patch, HIVE-6617.24.patch, HIVE-6617.25.patch, parser.png CLEAR LIBRARY CACHE As of today, antlr reports 214 warnings. Need to bring down this number, ideally to 0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9858) Create cbrt (cube root) UDF
[
https://issues.apache.org/jira/browse/HIVE-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov updated HIVE-9858:
--
Attachment: HIVE-9858.1.patch
patch #1 again
Create cbrt (cube root) UDF
---
Key: HIVE-9858
URL: https://issues.apache.org/jira/browse/HIVE-9858
Project: Hive
Issue Type: Improvement
Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Attachments: HIVE-9858.1.patch, HIVE-9858.1.patch
returns the cube root of a double value
cbrt(double a) : double
For example:
{code}
select cbrt(87860583272930481.0);
OK
444561.0
{code}
I noticed that Math.pow(a, 1.0/3.0) and hive power UDF return
444560.965 for the example above.
However Math.cbrt returns 444561.0
This is why we should have hive cbrt function in hive
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9902) Map join small table files need more replications [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353873#comment-14353873
]
Chao commented on HIVE-9902:
+1
Map join small table files need more replications [Spark Branch]
Key: HIVE-9902
URL: https://issues.apache.org/jira/browse/HIVE-9902
Project: Hive
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: spark-branch
Attachments: HIVE-9902.1-spark.patch
We have
{noformat}
replication = (short) Math.min(MIN_REPLICATION, numOfPartitions);
{noformat}
It should be
{noformat}
replication = (short) Math.max(MIN_REPLICATION, numOfPartitions);
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9892) various MSSQL upgrade scripts don't work
[ https://issues.apache.org/jira/browse/HIVE-9892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353900#comment-14353900 ] Thejas M Nair commented on HIVE-9892: - +1 various MSSQL upgrade scripts don't work Key: HIVE-9892 URL: https://issues.apache.org/jira/browse/HIVE-9892 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.13.0, 0.14.0, 1.0.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 1.2.0 Attachments: HIVE-9892.ADDENDUM.patch, HIVE-9892.patch * Issue with GO statement when run through schematool - it results in syntax error. * the create if not exists logic for PART_COL_STATS wasn't working NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9659) 'Error while trying to create table container' occurs during hive query case execution when hive.optimize.skewjoin set to 'true' [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354331#comment-14354331
]
Xuefu Zhang commented on HIVE-9659:
---
[~ruili], let's create a JIRA for MR and move on. We enable the test only for
Spark.
'Error while trying to create table container' occurs during hive query case
execution when hive.optimize.skewjoin set to 'true' [Spark Branch]
---
Key: HIVE-9659
URL: https://issues.apache.org/jira/browse/HIVE-9659
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Xin Hao
Assignee: Rui Li
Attachments: HIVE-9659.1-spark.patch, HIVE-9659.2-spark.patch,
HIVE-9659.3-spark.patch
We found that 'Error while trying to create table container' occurs during
Big-Bench Q12 case execution when hive.optimize.skewjoin set to 'true'.
If hive.optimize.skewjoin set to 'false', the case could pass.
How to reproduce:
1. set hive.optimize.skewjoin=true;
2. Run BigBench case Q12 and it will fail.
Check the executor log (e.g. /usr/lib/spark/work/app-/2/stderr) and you
will found error 'Error while trying to create table container' in the log
and also a NullPointerException near the end of the log.
(a) Detail error message for 'Error while trying to create table container':
{noformat}
15/02/12 01:29:49 ERROR SparkMapRecordHandler: Error processing row:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Error while trying to
create table container
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Error while trying to
create table container
at
org.apache.hadoop.hive.ql.exec.spark.HashTableLoader.load(HashTableLoader.java:118)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:193)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:219)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1051)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:486)
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:141)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:47)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:98)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
at
org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:217)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:65)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error while
trying to create table container
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainerSerDe.load(MapJoinTableContainerSerDe.java:158)
at
org.apache.hadoop.hive.ql.exec.spark.HashTableLoader.load(HashTableLoader.java:115)
... 21 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error, not a
directory:
hdfs://bhx1:8020/tmp/hive/root/d22ef465-bff5-4edb-a822-0a9f1c25b66c/hive_2015-02-12_01-28-10_008_6897031694580088767-1/-mr-10009/HashTable-Stage-6/MapJoin-mapfile01--.hashtable
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainerSerDe.load(MapJoinTableContainerSerDe.java:106)
... 22 more
15/02/12 01:29:49 INFO SparkRecordHandler: maximum memory = 40939028480
15/02/12 01:29:49 INFO PerfLogger: PERFLOG method=SparkInitializeOperators
[jira] [Updated] (HIVE-9904) Perform stats aggregation in HBase co-processor [hbase-metastore branch]
[ https://issues.apache.org/jira/browse/HIVE-9904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-9904: --- Component/s: Metastore Perform stats aggregation in HBase co-processor [hbase-metastore branch] - Key: HIVE-9904 URL: https://issues.apache.org/jira/browse/HIVE-9904 Project: Hive Issue Type: Sub-task Components: Metastore Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6617) Reduce ambiguity in grammar
[
https://issues.apache.org/jira/browse/HIVE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354387#comment-14354387
]
Hive QA commented on HIVE-6617:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12703548/HIVE-6617.25.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 7762 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2985/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2985/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2985/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12703548 - PreCommit-HIVE-TRUNK-Build
Reduce ambiguity in grammar
---
Key: HIVE-6617
URL: https://issues.apache.org/jira/browse/HIVE-6617
Project: Hive
Issue Type: Task
Reporter: Ashutosh Chauhan
Assignee: Pengcheng Xiong
Attachments: HIVE-6617.01.patch, HIVE-6617.02.patch,
HIVE-6617.03.patch, HIVE-6617.04.patch, HIVE-6617.05.patch,
HIVE-6617.06.patch, HIVE-6617.07.patch, HIVE-6617.08.patch,
HIVE-6617.09.patch, HIVE-6617.10.patch, HIVE-6617.11.patch,
HIVE-6617.12.patch, HIVE-6617.13.patch, HIVE-6617.14.patch,
HIVE-6617.15.patch, HIVE-6617.16.patch, HIVE-6617.17.patch,
HIVE-6617.18.patch, HIVE-6617.19.patch, HIVE-6617.20.patch,
HIVE-6617.21.patch, HIVE-6617.22.patch, HIVE-6617.23.patch,
HIVE-6617.24.patch, HIVE-6617.25.patch, parser.png
CLEAR LIBRARY CACHE
As of today, antlr reports 214 warnings. Need to bring down this number,
ideally to 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9569) Enable more unit tests for UNION ALL [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354339#comment-14354339 ] Xuefu Zhang commented on HIVE-9569: --- +1 Enable more unit tests for UNION ALL [Spark Branch] --- Key: HIVE-9569 URL: https://issues.apache.org/jira/browse/HIVE-9569 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Chao Assignee: Chao Attachments: HIVE-9569.1-spark.patch, HIVE-9569.1.patch, HIVE-9569.2.patch, HIVE-9569.3.patch, HIVE-9569.4.patch, HIVE-9569.5.patch Currently, we only enabled a subset of all the union tests. We should try to enable the rest, and see if there's any issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9882) Add jar/file doesn't work with yarn-cluster mode [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354246#comment-14354246
]
Xiaomin Zhang commented on HIVE-9882:
-
It now passed the test, good job!
Add jar/file doesn't work with yarn-cluster mode [Spark Branch]
---
Key: HIVE-9882
URL: https://issues.apache.org/jira/browse/HIVE-9882
Project: Hive
Issue Type: Sub-task
Components: Hive, spark-branch
Affects Versions: spark-branch
Reporter: Xiaomin Zhang
Assignee: Rui Li
Fix For: spark-branch
Attachments: HIVE-9882.1-spark.patch, HIVE-9882.1-spark.patch
It seems current fix for HIVE-9425 only uploads the Jar/Files to HDFS,
however, they are not accessible by the Driver/Executor.
I found below in the AM log:
{noformat}
15/02/26 15:10:36 INFO Configuration.deprecation: mapred.min.split.size is
deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
15/02/26 15:10:36 INFO client.SparkClientUtilities: Added
jar[file:/data/hadoop-devel/data/nm/usercache/user/appcache/application_1424933948132_0002/container_1424933948132_0002_01_01/hdfs:/localhost:8020/tmp/hive/user/47040bca-1da4-49b6-b2c7-69be9bc92855/hive-exec-1.2.0-SNAPSHOT.jar]
to classpath.
15/02/26 15:10:36 INFO client.SparkClientUtilities: Added
jar[file:/data/hadoop-devel/data/nm/usercache/user/appcache/application_1424933948132_0002/container_1424933948132_0002_01_01/hdfs:/localhost:8020/tmp/hive/user/47040bca-1da4-49b6-b2c7-69be9bc92855/opennlp-maxent-3.0.3.jar]
to classpath.
15/02/26 15:10:36 INFO client.SparkClientUtilities: Added
jar[file:/data/hadoop-devel/data/nm/usercache/user/appcache/application_1424933948132_0002/container_1424933948132_0002_01_01/hdfs:/localhost:8020/tmp/hive/user/47040bca-1da4-49b6-b2c7-69be9bc92855/bigbenchqueriesmr.jar]
to classpath.
15/02/26 15:10:36 INFO client.SparkClientUtilities: Added
jar[file:/data/hadoop-devel/data/nm/usercache/user/appcache/application_1424933948132_0002/container_1424933948132_0002_01_01/hdfs:/localhost:8020/tmp/hive/user/47040bca-1da4-49b6-b2c7-69be9bc92855/opennlp-tools-1.5.3.jar]
to classpath.
15/02/26 15:10:36 INFO client.SparkClientUtilities: Added
jar[file:/data/hadoop-devel/data/nm/usercache/user/appcache/application_1424933948132_0002/container_1424933948132_0002_01_01/hdfs:/localhost:8020/tmp/hive/user/47040bca-1da4-49b6-b2c7-69be9bc92855/jcl-over-slf4j-1.7.5.jar]
to classpath.
15/02/26 15:10:36 INFO client.RemoteDriver: Failed to run job
6886df05-f430-456c-a0ff-c7621db712d6
org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find
class: de.bankmark.bigbench.queries.q10.SentimentUDF
{noformat}
As above shows, the file path which was attempted to add to Classpath is
invalid, so actually all uploaded Jars/Files are still not available for
Driver/Executor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9659) 'Error while trying to create table container' occurs during hive query case execution when hive.optimize.skewjoin set to 'true' [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354321#comment-14354321
]
Rui Li commented on HIVE-9659:
--
I tried to add golden file for MR for the added test. However MR doesn't
produce a correct result for the new test (I used trunk to test MR). I think
there's some issue with MR when map join and skew join are on. If I turn off
skew join, MR produces the same result as spark does here.
[~xuefuz] - what's your opinion about this?
'Error while trying to create table container' occurs during hive query case
execution when hive.optimize.skewjoin set to 'true' [Spark Branch]
---
Key: HIVE-9659
URL: https://issues.apache.org/jira/browse/HIVE-9659
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Xin Hao
Assignee: Rui Li
Attachments: HIVE-9659.1-spark.patch, HIVE-9659.2-spark.patch,
HIVE-9659.3-spark.patch
We found that 'Error while trying to create table container' occurs during
Big-Bench Q12 case execution when hive.optimize.skewjoin set to 'true'.
If hive.optimize.skewjoin set to 'false', the case could pass.
How to reproduce:
1. set hive.optimize.skewjoin=true;
2. Run BigBench case Q12 and it will fail.
Check the executor log (e.g. /usr/lib/spark/work/app-/2/stderr) and you
will found error 'Error while trying to create table container' in the log
and also a NullPointerException near the end of the log.
(a) Detail error message for 'Error while trying to create table container':
{noformat}
15/02/12 01:29:49 ERROR SparkMapRecordHandler: Error processing row:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Error while trying to
create table container
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Error while trying to
create table container
at
org.apache.hadoop.hive.ql.exec.spark.HashTableLoader.load(HashTableLoader.java:118)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:193)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:219)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1051)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:486)
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:141)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:47)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:98)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
at
org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:217)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:65)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error while
trying to create table container
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainerSerDe.load(MapJoinTableContainerSerDe.java:158)
at
org.apache.hadoop.hive.ql.exec.spark.HashTableLoader.load(HashTableLoader.java:115)
... 21 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error, not a
directory:
hdfs://bhx1:8020/tmp/hive/root/d22ef465-bff5-4edb-a822-0a9f1c25b66c/hive_2015-02-12_01-28-10_008_6897031694580088767-1/-mr-10009/HashTable-Stage-6/MapJoin-mapfile01--.hashtable
at
[jira] [Commented] (HIVE-9659) 'Error while trying to create table container' occurs during hive query case execution when hive.optimize.skewjoin set to 'true' [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354342#comment-14354342
]
Rui Li commented on HIVE-9659:
--
Is there a way to enable the test only for spark? Seems I add the test and MR
picks it up automatically.
'Error while trying to create table container' occurs during hive query case
execution when hive.optimize.skewjoin set to 'true' [Spark Branch]
---
Key: HIVE-9659
URL: https://issues.apache.org/jira/browse/HIVE-9659
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Xin Hao
Assignee: Rui Li
Attachments: HIVE-9659.1-spark.patch, HIVE-9659.2-spark.patch,
HIVE-9659.3-spark.patch
We found that 'Error while trying to create table container' occurs during
Big-Bench Q12 case execution when hive.optimize.skewjoin set to 'true'.
If hive.optimize.skewjoin set to 'false', the case could pass.
How to reproduce:
1. set hive.optimize.skewjoin=true;
2. Run BigBench case Q12 and it will fail.
Check the executor log (e.g. /usr/lib/spark/work/app-/2/stderr) and you
will found error 'Error while trying to create table container' in the log
and also a NullPointerException near the end of the log.
(a) Detail error message for 'Error while trying to create table container':
{noformat}
15/02/12 01:29:49 ERROR SparkMapRecordHandler: Error processing row:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Error while trying to
create table container
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Error while trying to
create table container
at
org.apache.hadoop.hive.ql.exec.spark.HashTableLoader.load(HashTableLoader.java:118)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:193)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:219)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1051)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:486)
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:141)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:47)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:98)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
at
org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:217)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:65)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error while
trying to create table container
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainerSerDe.load(MapJoinTableContainerSerDe.java:158)
at
org.apache.hadoop.hive.ql.exec.spark.HashTableLoader.load(HashTableLoader.java:115)
... 21 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error, not a
directory:
hdfs://bhx1:8020/tmp/hive/root/d22ef465-bff5-4edb-a822-0a9f1c25b66c/hive_2015-02-12_01-28-10_008_6897031694580088767-1/-mr-10009/HashTable-Stage-6/MapJoin-mapfile01--.hashtable
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainerSerDe.load(MapJoinTableContainerSerDe.java:106)
... 22 more
15/02/12 01:29:49 INFO SparkRecordHandler: maximum memory = 40939028480
15/02/12 01:29:49 INFO PerfLogger: PERFLOG method=SparkInitializeOperators
[jira] [Commented] (HIVE-9601) New Beeline queries will hang If Beeline terminates in-properly [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354020#comment-14354020
]
Hive QA commented on HIVE-9601:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12703494/HIVE-9601.2-spark.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 7567 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/775/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/775/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-775/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12703494 - PreCommit-HIVE-SPARK-Build
New Beeline queries will hang If Beeline terminates in-properly [Spark Branch]
--
Key: HIVE-9601
URL: https://issues.apache.org/jira/browse/HIVE-9601
Project: Hive
Issue Type: Bug
Components: Spark
Reporter: Szehon Ho
Assignee: Jimmy Xiang
Fix For: spark-branch
Attachments: HIVE-9601.1-spark.patch, HIVE-9601.1-spark.patch,
HIVE-9601.2-spark.patch
User session's Spark application seems to stay around if beeline is not quit
properly (!quit) because the user is not disconnected.
If Beeline is started, it will create a new Spark application which will hang
waiting for the first one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9858) Create cbrt (cube root) UDF
[
https://issues.apache.org/jira/browse/HIVE-9858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov updated HIVE-9858:
--
Attachment: HIVE-9858.2.patch
patch #2 - fixed show_functions.q.out
Create cbrt (cube root) UDF
---
Key: HIVE-9858
URL: https://issues.apache.org/jira/browse/HIVE-9858
Project: Hive
Issue Type: Improvement
Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Attachments: HIVE-9858.1.patch, HIVE-9858.1.patch, HIVE-9858.2.patch
returns the cube root of a double value
cbrt(double a) : double
For example:
{code}
select cbrt(87860583272930481.0);
OK
444561.0
{code}
I noticed that Math.pow(a, 1.0/3.0) and hive power UDF return
444560.965 for the example above.
However Math.cbrt returns 444561.0
This is why we should have hive cbrt function in hive
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9569) Enable more unit tests for UNION ALL [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354209#comment-14354209
]
Hive QA commented on HIVE-9569:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12703545/HIVE-9569.5.patch
{color:green}SUCCESS:{color} +1 7635 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2983/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2983/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2983/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12703545 - PreCommit-HIVE-TRUNK-Build
Enable more unit tests for UNION ALL [Spark Branch]
---
Key: HIVE-9569
URL: https://issues.apache.org/jira/browse/HIVE-9569
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Chao
Assignee: Chao
Attachments: HIVE-9569.1-spark.patch, HIVE-9569.1.patch,
HIVE-9569.2.patch, HIVE-9569.3.patch, HIVE-9569.4.patch, HIVE-9569.5.patch
Currently, we only enabled a subset of all the union tests. We should try to
enable the rest, and see if there's any issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9788) Make double quote optional in tsv/csv/dsv output
[
https://issues.apache.org/jira/browse/HIVE-9788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-9788:
-
Labels: TODOC1.2 (was: )
Make double quote optional in tsv/csv/dsv output
Key: HIVE-9788
URL: https://issues.apache.org/jira/browse/HIVE-9788
Project: Hive
Issue Type: Improvement
Reporter: Brock Noland
Assignee: Ferdinand Xu
Labels: TODOC1.2
Fix For: 1.2.0
Attachments: HIVE-9788.1.patch, HIVE-9788.2.patch, HIVE-9788.patch
Similar to HIVE-7390 some customers would like the double quotes to be
optional. So if the data is {{A}} then the output from beeline should be
{{A}} which is the same as the Hive CLI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9788) Make double quote optional in tsv/csv/dsv output
[
https://issues.apache.org/jira/browse/HIVE-9788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352690#comment-14352690
]
Lefty Leverenz commented on HIVE-9788:
--
Doc note: The release note needs to be updated now that disableQuotingForSV is
a system property.
(I don't know how users set the system properties, but the release note was
written for patch 2 which had --disableQuotingForSV in BeeLine.properties and
it's gone from patch 3, which has disable.quoting.for.sv in
SeparatedValuesOutputFormat.java instead.)
Make double quote optional in tsv/csv/dsv output
Key: HIVE-9788
URL: https://issues.apache.org/jira/browse/HIVE-9788
Project: Hive
Issue Type: Improvement
Reporter: Brock Noland
Assignee: Ferdinand Xu
Labels: TODOC1.2
Fix For: 1.2.0
Attachments: HIVE-9788.1.patch, HIVE-9788.2.patch, HIVE-9788.patch
Similar to HIVE-7390 some customers would like the double quotes to be
optional. So if the data is {{A}} then the output from beeline should be
{{A}} which is the same as the Hive CLI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9588) Reimplement HCatClientHMSImpl.dropPartitions() with HMSC.dropPartitions()
[
https://issues.apache.org/jira/browse/HIVE-9588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-9588:
-
Labels: TODOC1.2 (was: )
Reimplement HCatClientHMSImpl.dropPartitions() with HMSC.dropPartitions()
-
Key: HIVE-9588
URL: https://issues.apache.org/jira/browse/HIVE-9588
Project: Hive
Issue Type: Bug
Components: HCatalog, Metastore, Thrift API
Affects Versions: 0.14.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Labels: TODOC1.2
Fix For: 1.2.0
Attachments: HIVE-9588.1.patch, HIVE-9588.2.patch, HIVE-9588.3.patch,
HIVE-9588.4.patch
{{HCatClientHMSImpl.dropPartitions()}} currently has an embarrassingly
inefficient implementation. The partial partition-spec is converted into a
filter-string. The partitions are fetched from the server, and then dropped
one by one.
Here's a reimplementation that uses the {{ExprNode}}-based
{{HiveMetaStoreClient.dropPartitions()}}. It cuts out the excessive
back-and-forth between the HMS and the client-side. It also reduces the
memory footprint (from loading all the partitions that are to be dropped).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7100) Users of hive should be able to specify skipTrash when dropping tables.
[ https://issues.apache.org/jira/browse/HIVE-7100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352730#comment-14352730 ] Lefty Leverenz commented on HIVE-7100: -- Doc note: The documentation has been revised (please review). * [LanguageManual DDL -- Drop Table | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-DropTable] Users of hive should be able to specify skipTrash when dropping tables. --- Key: HIVE-7100 URL: https://issues.apache.org/jira/browse/HIVE-7100 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.13.0 Reporter: Ravi Prakash Assignee: david serafini Fix For: 0.14.0 Attachments: HIVE-7100.1.patch, HIVE-7100.10.patch, HIVE-7100.11.patch, HIVE-7100.2.patch, HIVE-7100.3.patch, HIVE-7100.4.patch, HIVE-7100.5.patch, HIVE-7100.8.patch, HIVE-7100.9.patch, HIVE-7100.patch Users of our clusters are often running up against their quota limits because of Hive tables. When they drop tables, they have to then manually delete the files from HDFS using skipTrash. This is cumbersome and unnecessary. We should enable users to skipTrash directly when dropping tables. We should also be able to provide this functionality without polluting SQL syntax. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9118) Support auto-purge for tables, when dropping tables/partitions.
[
https://issues.apache.org/jira/browse/HIVE-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352734#comment-14352734
]
Lefty Leverenz commented on HIVE-9118:
--
Doc note: This is documented in the TBLPROPERTIES bullet item after the CREATE
TABLE syntax in the DDL doc (please review).
* [LanguageManual DDL -- Create Table |
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTable]
Support auto-purge for tables, when dropping tables/partitions.
---
Key: HIVE-9118
URL: https://issues.apache.org/jira/browse/HIVE-9118
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 1.0.0, 1.1
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan
Fix For: 1.2.0
Attachments: HIVE-9118.1.patch, HIVE-9118.2.patch, HIVE-9118.3.patch
HIVE-7100 introduced a way to skip the trash directory, when deleting
table-data, while dropping tables.
In HIVE-9083/HIVE-9086, I extended this to work when partitions are dropped.
Here, I propose a table-parameter ({{auto.purge}}) to set up tables to
skip-trash when table/partition data is deleted, without needing to say
PURGE on the Hive CLI. Apropos, on {{dropTable()}} and {{dropPartition()}},
table data is deleted directly (and not moved to trash) if the following hold
true:
# The table is MANAGED.
# The {{deleteData}} parameter to the {{HMSC.drop*()}} methods is true.
# Either PURGE is explicitly specified on the command-line (or rather,
{{ifPurge}} is set in the environment context, OR
# TBLPROPERTIES contains {{auto.purge=true}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9086) Add language support to PURGE data while dropping partitions.
[ https://issues.apache.org/jira/browse/HIVE-9086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352746#comment-14352746 ] Lefty Leverenz commented on HIVE-9086: -- Doc note: The documentation has been revised (please review). * [LanguageManual DDL -- Drop Partitions | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-DropPartitions] Add language support to PURGE data while dropping partitions. - Key: HIVE-9086 URL: https://issues.apache.org/jira/browse/HIVE-9086 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.15.0 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Fix For: 1.2.0 Attachments: HIVE-9086.1.patch HIVE-9083 adds metastore-support to skip-trash while dropping partitions. This patch includes language support to do the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
