date:20190614

[jira] [Created] (HIVE-21878) Metric for AM to show whether it is currently running a DAG

2019-06-14 Thread Jason Dere (JIRA)

Jason Dere created HIVE-21878:
-

 Summary: Metric for AM to show whether it is currently running a 
DAG
 Key: HIVE-21878
 URL: https://issues.apache.org/jira/browse/HIVE-21878
 Project: Hive
  Issue Type: Bug
  Components: Tez
Reporter: Jason Dere
Assignee: Jason Dere
 Attachments: HIVE-21878.1.patch

Add a basic gauge metric to indicate whether a Tez AM is currently running a 
DAG for a Hive query.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21877) Change HCatTableInfo to not be transient in PartInfo

2019-06-14 Thread Ankit Jhalaria (JIRA)

Ankit Jhalaria created HIVE-21877:
-

 Summary: Change HCatTableInfo to not be transient in PartInfo
 Key: HIVE-21877
 URL: https://issues.apache.org/jira/browse/HIVE-21877
 Project: Hive
  Issue Type: New Feature
Reporter: Ankit Jhalaria






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21876) Documentation : collect_list documentation does not mention it silently ignores nulls

2019-06-14 Thread Guillaume Xavier Taillon (JIRA)

Guillaume Xavier Taillon created HIVE-21876:
---

 Summary: Documentation : collect_list documentation does not 
mention it silently ignores nulls
 Key: HIVE-21876
 URL: https://issues.apache.org/jira/browse/HIVE-21876
 Project: Hive
  Issue Type: Bug
  Components: Documentation
 Environment: select version()

_c0▼
2.1.0.2.6.5.1050-37 rb49a1217fdf463c41661ec6512348ff9ef4706a9
Reporter: Guillaume Xavier Taillon


Documentation at 

[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-Built-inAggregateFunctions(UDAF)]

For collect_list is currently only 
|array|collect_list(col)|Returns a list of objects with duplicates. (As of Hive 
0.13.0.)|

However it should mention that nulls are not captured, at least on Hive 2.1

select collect_list(v) from (
select 1 as v
union all select null as v
union all select 3 as v
) t

Result

_c0

[1,3]

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21875) Implement drop partition related methods on temporary tables

2019-06-14 Thread Laszlo Pinter (JIRA)

Laszlo Pinter created HIVE-21875:


 Summary: Implement drop partition related methods on temporary 
tables
 Key: HIVE-21875
 URL: https://issues.apache.org/jira/browse/HIVE-21875
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter


IMetaStoreClient exposes the following methods related to dropping partitions:
{code:java}
boolean dropPartition(String db_name, String tbl_name, List part_vals, 
boolean deleteData);
boolean dropPartition(String catName, String db_name, String tbl_name, 
List part_vals, boolean deleteData);
boolean dropPartition(String db_name, String tbl_name, List part_vals, 
PartitionDropOptions options);
boolean dropPartition(String catName, String db_name, String tbl_name, 
List part_vals, PartitionDropOptions options);
List dropPartitions(String dbName, String tblName, 
List> partExprs, boolean deleteData, boolean 
ifExists);
List dropPartitions(String catName, String dbName, String tblName, 
List> partExprs, boolean deleteData, boolean 
ifExists);
List dropPartitions(String dbName, String tblName, 
List> partExprs, boolean deleteData, boolean 
ifExists, boolean needResults);
List dropPartitions(String catName, String dbName, String tblName, 
List> partExprs, boolean deleteData, boolean 
ifExists, boolean needResults);
List dropPartitions(String dbName, String tblName, 
List> partExprs, PartitionDropOptions options);
List dropPartitions(String catName, String dbName, String tblName, 
List> partExprs, PartitionDropOptions options);
boolean dropPartition(String db_name, String tbl_name, String name, boolean 
deleteData);
boolean dropPartition(String catName, String db_name, String tbl_name, String 
name, boolean deleteData){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21874) Implement add partitions related methods on temporary table

2019-06-14 Thread Laszlo Pinter (JIRA)

Laszlo Pinter created HIVE-21874:


 Summary: Implement add partitions related methods on temporary 
table
 Key: HIVE-21874
 URL: https://issues.apache.org/jira/browse/HIVE-21874
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter


IMetaStoreClient exposes the following add partition related methods:
{code:java}
Partition add_partition(Partition partition);
int add_partitions(List partitions);
int add_partitions_pspec(PartitionSpecProxy partitionSpec);
List add_partitions(List partitions, boolean ifNotExists, 
boolean needResults);
{code}
These methods should be implemented in order to handle addition of partitions 
to temporary tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 70841: HIVE-21576: Introduce CAST...FORMAT and limited list of SQL:2016 datetime formats

2019-06-14 Thread Karen Coppage via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70841/
---

(Updated June 14, 2019, 8:30 a.m.)


Review request for hive and Gabor Kaszab.


Changes
---

Patch 14


Bugs: HIVE-21576
https://issues.apache.org/jira/browse/HIVE-21576


Repository: hive-git


Description
---

Timestamp and date handling and formatting are currently implemented in Hive 
using (sometimes very specific) Java SimpleDateFormat patterns with both 
SimpleDateFormat and java.time.DateTimeFormatter, however, these patterns are 
not what most standard SQL systems use. For example see Vertica, Netezza, 
Oracle, and PostgreSQL.

**Cast...Format**

SQL:2016 introduced the FORMAT clause for CAST which is the standard way to do 
string <-> datetime conversions

For example:

CAST( AS  [FORMAT ])
CAST( AS  [FORMAT ])
cast(dt as string format 'DD-MM-')
cast('01-05-2017' as date format 'DD-MM-')
Stuff like this wouldn't need to happen.

**New SQL:2016 Patterns**

Some conflicting examples:

SimpleDateTime: 'MMM dd,  HH:mm:ss'
SQL:2016: 'mon dd,  hh24:mi:ss'

SimpleDateTime: '-MM-dd HH:mm:ss'
SQL:2016: '-mm-dd hh24:mi:ss'

For the full list of patterns, see subsection "Proposal for Impala’s datetime 
patterns" in this doc: 
https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit

**Continued usage of SimpleDateFormat patterns**

[Update] This feature will NOT be behind a flag in order to keep things simple 
for users. Existing Hive functions that accept SimpleDateFormat patterns as 
input will continue to do so. Please let me know if you disagree with this 
decision. These are the functions (afaik) affected:

from_unixtime(bigint unixtime[, string format])
unix_timestamp(string date, string pattern)
to_unix_timestamp(date[, pattern])
add_months(string start_date, int num_months, output_date_format)
date_format(date/timestamp/string ts, string fmt)
This description is a heavily edited description of IMPALA-4018.


Diffs (updated)
-

  
common/src/java/org/apache/hadoop/hive/common/format/datetime/HiveDateTimeFormatter.java
 PRE-CREATION 
  
common/src/java/org/apache/hadoop/hive/common/format/datetime/HiveJavaDateTimeFormatter.java
 PRE-CREATION 
  
common/src/java/org/apache/hadoop/hive/common/format/datetime/HiveSimpleDateFormatter.java
 PRE-CREATION 
  
common/src/java/org/apache/hadoop/hive/common/format/datetime/HiveSqlDateTimeFormatter.java
 PRE-CREATION 
  
common/src/java/org/apache/hadoop/hive/common/format/datetime/package-info.java 
PRE-CREATION 
  
common/src/test/org/apache/hadoop/hive/common/format/datetime/TestHiveJavaDateTimeFormatter.java
 PRE-CREATION 
  
common/src/test/org/apache/hadoop/hive/common/format/datetime/TestHiveSimpleDateFormatter.java
 PRE-CREATION 
  
common/src/test/org/apache/hadoop/hive/common/format/datetime/TestHiveSqlDateTimeFormatter.java
 PRE-CREATION 
  
common/src/test/org/apache/hadoop/hive/common/format/datetime/package-info.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d08b05fb68 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 58fe0cd32e 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCastFormat.java 
PRE-CREATION 
  
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFCastFormat.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/cast_datetime_with_sql_2016_format.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/cast_datetime_with_sql_2016_format.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 374e9c4fce 


Diff: https://reviews.apache.org/r/70841/diff/5/

Changes: https://reviews.apache.org/r/70841/diff/4-5/


Testing
---


Thanks,

Karen Coppage

[jira] [Created] (HIVE-21873) Hive Cann't drop table in different hdfs clusters

2019-06-14 Thread Deng ZiWu (JIRA)

Deng ZiWu created HIVE-21873:


 Summary: Hive Cann't drop table in different hdfs clusters
 Key: HIVE-21873
 URL: https://issues.apache.org/jira/browse/HIVE-21873
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 2.3.5, 2.3.4, 2.3.2, 2.3.1, 2.3.0, 2.3.3
Reporter: Deng ZiWu


Hive Cann't drop table in different hdfs clusters

org.apache.hadoop.hive.metastore.HiveMetaStore

in method

checkTrashPurgeCombination()

ShimLoader.getHadoopShims().createHdfsEncryptionShim(FileSystem.get(hiveConf), 
hiveConf);

 

should be

ShimLoader.getHadoopShims().createHdfsEncryptionShim(wh.getFs(pathToData), 
hiveConf);

 

Because you want to delete the target fs data, not the default fs data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 70841: HIVE-21576: Introduce CAST...FORMAT and limited list of SQL:2016 datetime formats

2019-06-14 Thread Karen Coppage via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70841/
---

(Updated June 14, 2019, 7:05 a.m.)


Review request for hive and Gabor Kaszab.


Changes
---

Patch 12


Bugs: HIVE-21576
https://issues.apache.org/jira/browse/HIVE-21576


Repository: hive-git


Description
---

Timestamp and date handling and formatting are currently implemented in Hive 
using (sometimes very specific) Java SimpleDateFormat patterns with both 
SimpleDateFormat and java.time.DateTimeFormatter, however, these patterns are 
not what most standard SQL systems use. For example see Vertica, Netezza, 
Oracle, and PostgreSQL.

**Cast...Format**

SQL:2016 introduced the FORMAT clause for CAST which is the standard way to do 
string <-> datetime conversions

For example:

CAST( AS  [FORMAT ])
CAST( AS  [FORMAT ])
cast(dt as string format 'DD-MM-')
cast('01-05-2017' as date format 'DD-MM-')
Stuff like this wouldn't need to happen.

**New SQL:2016 Patterns**

Some conflicting examples:

SimpleDateTime: 'MMM dd,  HH:mm:ss'
SQL:2016: 'mon dd,  hh24:mi:ss'

SimpleDateTime: '-MM-dd HH:mm:ss'
SQL:2016: '-mm-dd hh24:mi:ss'

For the full list of patterns, see subsection "Proposal for Impala’s datetime 
patterns" in this doc: 
https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit

**Continued usage of SimpleDateFormat patterns**

[Update] This feature will NOT be behind a flag in order to keep things simple 
for users. Existing Hive functions that accept SimpleDateFormat patterns as 
input will continue to do so. Please let me know if you disagree with this 
decision. These are the functions (afaik) affected:

from_unixtime(bigint unixtime[, string format])
unix_timestamp(string date, string pattern)
to_unix_timestamp(date[, pattern])
add_months(string start_date, int num_months, output_date_format)
date_format(date/timestamp/string ts, string fmt)
This description is a heavily edited description of IMPALA-4018.


Diffs (updated)
-

  
common/src/java/org/apache/hadoop/hive/common/format/datetime/HiveDateTimeFormatter.java
 PRE-CREATION 
  
common/src/java/org/apache/hadoop/hive/common/format/datetime/HiveJavaDateTimeFormatter.java
 PRE-CREATION 
  
common/src/java/org/apache/hadoop/hive/common/format/datetime/HiveSimpleDateFormatter.java
 PRE-CREATION 
  
common/src/java/org/apache/hadoop/hive/common/format/datetime/HiveSqlDateTimeFormatter.java
 PRE-CREATION 
  
common/src/java/org/apache/hadoop/hive/common/format/datetime/package-info.java 
PRE-CREATION 
  
common/src/test/org/apache/hadoop/hive/common/format/datetime/TestHiveJavaDateTimeFormatter.java
 PRE-CREATION 
  
common/src/test/org/apache/hadoop/hive/common/format/datetime/TestHiveSimpleDateFormatter.java
 PRE-CREATION 
  
common/src/test/org/apache/hadoop/hive/common/format/datetime/TestHiveSqlDateTimeFormatter.java
 PRE-CREATION 
  
common/src/test/org/apache/hadoop/hive/common/format/datetime/package-info.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d08b05fb68 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 58fe0cd32e 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCastFormat.java 
PRE-CREATION 
  
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFCastWithFormat.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/cast_datetime_with_sql_2016_format.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/cast_datetime_with_sql_2016_format.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 374e9c4fce 


Diff: https://reviews.apache.org/r/70841/diff/3/

Changes: https://reviews.apache.org/r/70841/diff/2-3/


Testing
---


Thanks,

Karen Coppage

[jira] [Created] (HIVE-21878) Metric for AM to show whether it is currently running a DAG

[jira] [Created] (HIVE-21877) Change HCatTableInfo to not be transient in PartInfo

[jira] [Created] (HIVE-21876) Documentation : collect_list documentation does not mention it silently ignores nulls

[jira] [Created] (HIVE-21875) Implement drop partition related methods on temporary tables

[jira] [Created] (HIVE-21874) Implement add partitions related methods on temporary table

Re: Review Request 70841: HIVE-21576: Introduce CAST...FORMAT and limited list of SQL:2016 datetime formats

[jira] [Created] (HIVE-21873) Hive Cann't drop table in different hdfs clusters

Re: Review Request 70841: HIVE-21576: Introduce CAST...FORMAT and limited list of SQL:2016 datetime formats

8 matches

Site Navigation

Mail list logo

Footer information