Re: JIRA "Affects version/s" and "Fix version/s"

2015-10-15 Thread Wei Zheng 
Looks like it's configurable.

https://confluence.atlassian.com/jira/specifying-field-behavior-185729654.html#SpecifyingFieldBehavior-Makingafieldrequiredoroptional

Thanks,
Wei







On 10/15/15, 13:58, "Eugene Koifman"  wrote:

>Can JIRA be made to enforce that these are set to something when the
>ticket is Resolved?
>
>
>On 10/15/15, 9:52 AM, "Xuefu Zhang"  wrote:
>
>>Hi folks,
>>
>>I know most of us have done a good job in updating JIRAs with respective
>>"Affects version/s" and "Fix version/s". However, I do see from time to
>>time that these fields are missing in some JIRAs even after those JIRAs
>>are
>>closed.
>>
>>These field are very important in helping users to determine if their
>>versions are affected or fixed.
>>
>>If for some reason either the field is not applicable, please use "n/a"
>>specifically.
>>
>>Please feel free to share your thoughts or comments on this.
>>
>>Thanks,
>>Xuefu
>
>

[jira] [Created] (HIVE-12201) Tez settings need to be shown in set -v output when execution engine is tez.

2015-10-15 Thread Vikram Dixit K (JIRA) 
Vikram Dixit K created HIVE-12201:
-

 Summary: Tez settings need to be shown in set -v output when 
execution engine is tez.
 Key: HIVE-12201
 URL: https://issues.apache.org/jira/browse/HIVE-12201
 Project: Hive
  Issue Type: Bug
  Components: Tez
Affects Versions: 1.2.1, 1.0.1
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K
Priority: Minor
 Attachments: HIVE-12201.1.patch

The set -v output currently shows configurations for yarn, hdfs etc. but does 
not show tez settings when tez is set as the execution engine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 39374: HIVE-12164

2015-10-15 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39374/
---

Review request for hive and pengcheng xiong.


Bugs: HIVE-12164
https://issues.apache.org/jira/browse/HIVE-12164


Repository: hive-git


Description
---

Remove jdbc stats collection mechanism


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java b16b231 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0fcd39b 
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/stats/DummyStatsAggregator.java
 eb3f6eb 
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/stats/KeyVerifyingStatsAggregator.java
 4e00316 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java b7c1267 
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 9775645 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 83b4969 
  ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanMapper.java 
2a7e979 
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 
b9863d9 
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorSpark.java 
4c01b25 
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java 
662c106 
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsPublisher.java 
e5f1400 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java b115daf 
  
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsCollectionTaskIndependent.java 
52c06c2 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java b9878a3 
  ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java 6dfc178 
  ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsPublisher.java aa2bf62 
  ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 
d8c9926 
  ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 
0318a8c 
  ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsSetupConstants.java 
e39fc5b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsUtils.java 59d94d5 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java d22d022 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestStatsPublisherEnhanced.java 
c257797 
  ql/src/test/queries/clientpositive/alter_table_update_status.q 4cc1a18 
  ql/src/test/queries/clientpositive/analyze_tbl_part.q 9040bd4 
  ql/src/test/queries/clientpositive/lb_fs_stats.q 7f31797 
  ql/src/test/queries/clientpositive/metadata_only_queries.q 70fac92 
  ql/src/test/queries/clientpositive/metadata_only_queries_with_filters.q 
a6f1148 
  ql/src/test/queries/clientpositive/stats19.q ccdad6b 
  ql/src/test/queries/clientpositive/stats_only_null.q b2bd4dc 
  ql/src/test/queries/clientpositive/statsfs.q 82a2295 

Diff: https://reviews.apache.org/r/39374/diff/


Testing
---

Existing tests.


Thanks,

Ashutosh Chauhan

Re: Review Request 38862: HIVE-11985 handle long typenames from Avro schema in metastore

2015-10-15 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38862/
---

(Updated Oct. 16, 2015, 1:21 a.m.)


Review request for hive, Ashutosh Chauhan and Xuefu Zhang.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0fcd39b 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
12f3f16 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 076791f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4058606 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 9f9b5bc 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 3d1ca93 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 3262887 
  serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java c5e78c5 
  serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 0e4e4c6 

Diff: https://reviews.apache.org/r/38862/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 39026: HIVE-11676 implement metastore API to do file footer PPD

2015-10-15 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39026/
---

(Updated Oct. 15, 2015, 11:12 p.m.)


Review request for hive, Daniel Dai, Alan Gates, and Thejas Nair.


Repository: hive-git


Description
---

.


Diffs (updated)
-

  metastore/pom.xml 2ab5cd3 
  
metastore/src/gen/protobuf/gen-java/org/apache/hadoop/hive/metastore/Metastore.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
ccb4c98 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java a02f179 
  
metastore/src/java/org/apache/hadoop/hive/metastore/PartitionExpressionProxy.java
 5195481 
  metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1968256 
  metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseReadWrite.java 
f69b4c7 
  metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
df0fac3 
  metastore/src/protobuf/org/apache/hadoop/hive/metastore/metastore.proto 
PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 0f3331a 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 126a2c2 
  
metastore/src/test/org/apache/hadoop/hive/metastore/MockPartitionExpressionForMetastore.java
 bae1391 
  metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java 
78a9ea0 
  metastore/src/test/org/apache/hadoop/hive/metastore/hbase/MockUtils.java 
6c288f4 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/FileMetaInfo.java 2853119 
  ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 
690b8c9 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionExpressionForMetastore.java
 9ffa177 

Diff: https://reviews.apache.org/r/39026/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-12200) INSERT INTO table using a select statement w/o a FROM clause fails

2015-10-15 Thread Jimmy Xiang (JIRA) 
Jimmy Xiang created HIVE-12200:
--

 Summary: INSERT INTO table using a select statement w/o a FROM 
clause fails
 Key: HIVE-12200
 URL: https://issues.apache.org/jira/browse/HIVE-12200
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


Here is the stack trace:

{noformat}
FailedPredicateException(regularBody,{$s.tree.getChild(1) !=null}?)
at 
org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41047)
at 
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40222)
at 
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40092)
at 
org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1656)
at 
org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1140)
at 
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
at 
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:407)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:312)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1162)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1215)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1091)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1081)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:225)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:177)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:388)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:323)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:731)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:704)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:633)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
FAILED: ParseException line 1:29 Failed to recognize predicate ''. Failed 
rule: 'regularBody' in statement
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 39026: HIVE-11676 implement metastore API to do file footer PPD

2015-10-15 Thread Sergey Shelukhin 


> On Oct. 15, 2015, 9:09 p.m., Alan Gates wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java, 
> > line 2385
> > 
> >
> > Is doing the evaluation locally rather than in an HBase filter a for 
> > now thing or a forever thing?  It seems like it would be much more 
> > efficient to do it in a filter.  I believe Daniel and Thejas have already 
> > written code to handle translating Hive partition expressions into HBase 
> > filters.  I'm not sure if you could re-use that here or not.

To be handled in HIVE-12051


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39026/#review102823
---


On Oct. 15, 2015, 1:47 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39026/
> ---
> 
> (Updated Oct. 15, 2015, 1:47 a.m.)
> 
> 
> Review request for hive, Daniel Dai, Alan Gates, and Thejas Nair.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   metastore/pom.xml 2ab5cd3 
>   
> metastore/src/gen/protobuf/gen-java/org/apache/hadoop/hive/metastore/Metastore.java
>  PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> ccb4c98 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> 136eff5 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/PartitionExpressionProxy.java
>  5195481 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1968256 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseReadWrite.java 
> f69b4c7 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
> df0fac3 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  0f3331a 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  126a2c2 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/MockPartitionExpressionForMetastore.java
>  bae1391 
>   metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java 
> 78a9ea0 
>   metastore/src/test/org/apache/hadoop/hive/metastore/hbase/MockUtils.java 
> 6c288f4 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/FileMetaInfo.java 2853119 
>   ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 
> 690b8c9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionExpressionForMetastore.java
>  9ffa177 
> 
> Diff: https://reviews.apache.org/r/39026/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-12199) LLAP: unnecessary preemption of the same stage

2015-10-15 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-12199:
---

 Summary: LLAP: unnecessary preemption of the same stage
 Key: HIVE-12199
 URL: https://issues.apache.org/jira/browse/HIVE-12199
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


6-node cluster x 16 executors, nothing else running on it
{code}
hive> select sum(l_extendedprice * l_discount) as revenue from testing.lineitem 
where l_shipdate >= '1993-01-01' and l_shipdate < '1994-01-01' ;
...
--
VERTICES  MODESTATUS  TOTAL  COMPLETED  RUNNING  PENDING  
FAILED  KILLED  
--
Map 1 ..  llap SUCCEEDED17617600
   0   8  
Reducer 2 ..  llap SUCCEEDED  1  100
   0   1 
{code}

those killed mappers are preempted:
{code}
015-10-15 18:07:07,154 INFO [Dispatcher thread: Central] 
history.HistoryEventHandler: 
[HISTORY][DAG:dag_1442254312093_2158_1][Event:TASK_ATTEMPT_FINISHED]: 
vertexName=Map 1, taskAttemptId=attempt_1442254312093_2158_1_00_000103_0, 
startTime=1444946823795, finishTime=1444946827152, timeTaken=3357, 
status=KILLED, errorEnum=EXTERNAL_PREEMPTION, diagnostics=Attempt preempted, 
lastDataEventSourceTA=null, lastDataEventTime=0, counters=Counters: 1, 
org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1
{code}

There's no reason for mappers of the same stage to preempt each other in this 
case
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12198) LLAP: reader failures when querying uncompressed data

2015-10-15 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-12198:
---

 Summary: LLAP: reader failures when querying uncompressed data
 Key: HIVE-12198
 URL: https://issues.apache.org/jira/browse/HIVE-12198
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Gopal V
Assignee: Sergey Shelukhin


{code}
hive> select sum(l_extendedprice * l_discount) as revenue from testing.lineitem 
where l_shipdate >= '1993-01-01' and l_shipdate < '1994-01-01' ;

Caused by: 
org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException: 
Failed to allocate 492; at 0 out of 1
at 
org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:176)
at 
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.preReadUncompressedStream(EncodedReaderImpl.java:882)
at 
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:319)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:413)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:194)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:191)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:191)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:74)
at 
org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
... 4 more
{code}

{code}
Caused by: java.lang.ClassCastException: 
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl$UncompressedCacheChunk
 cannot be cast to org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$BufferChunk
at 
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.copyAndReplaceUncompressedChunks(EncodedReaderImpl.java:962)
at 
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.preReadUncompressedStream(EncodedReaderImpl.java:890)
at 
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:319)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:413)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:194)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:191)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:191)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:74)
at 
org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
... 4 more
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12197) Parquet timestamps differ between Hive and Impala

2015-10-15 Thread Ryan Blue (JIRA) 
Ryan Blue created HIVE-12197:


 Summary: Parquet timestamps differ between Hive and Impala
 Key: HIVE-12197
 URL: https://issues.apache.org/jira/browse/HIVE-12197
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 1.1.1
Reporter: Ryan Blue


Timestamps several hundred years in the past differ between Hive and Impala 
when stored in Parquet:

{code}
Query: create table ts_parquet (id bigint, ts timestamp) stored as parquet
Query: insert into ts_parquet values (1, '1400-01-01 00:01:01.101'), (2, 
'1581-11-14 07:08:01.101'), (3, '1582-02-11 07:08:01.101'), (4, '1999-01-02 
14:15:34.101')
Query: select * from ts_parquet
++---+
| id | ts|
++---+
| 1  | 1400-01-01 00:01:01.10100 |
| 2  | 1581-11-14 07:08:01.10100 |
| 3  | 1582-02-11 07:08:01.10100 |
| 4  | 1999-01-02 14:15:34.10100 |
++---+

hive> select * from ts_parquet;
1   1399-12-24 00:01:01.101
2   1581-11-04 07:08:01.101
3   1582-02-01 07:08:01.101
4   1999-01-02 14:15:34.101
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12196) NPE when converting bad timestamp value

2015-10-15 Thread Ryan Blue (JIRA) 
Ryan Blue created HIVE-12196:


 Summary: NPE when converting bad timestamp value
 Key: HIVE-12196
 URL: https://issues.apache.org/jira/browse/HIVE-12196
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 1.1.1
Reporter: Ryan Blue


When I convert a timestamp value that is slightly wrong, the result is a NPE. 
Other queries correctly reject the timestamp:

{code}
hive> select from_utc_timestamp('2015-04-11-12:24:34.535', 'UTC');
FAILED: NullPointerException null
hive> select TIMESTAMP '2015-04-11-12:24:34.535';
FAILED: SemanticException Unable to convert time literal 
'2015-04-11-12:24:34.535' to time value.
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 39026: HIVE-11676 implement metastore API to do file footer PPD

2015-10-15 Thread Alan Gates 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39026/#review102823
---



metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java (line 603)


Some javadoc here would be good, as it is seriously non-obvious what you 
expect to be passed to some of these arguments.



metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java (line 
2385)


Is doing the evaluation locally rather than in an HBase filter a for now 
thing or a forever thing?  It seems like it would be much more efficient to do 
it in a filter.  I believe Daniel and Thejas have already written code to 
handle translating Hive partition expressions into HBase filters.  I'm not sure 
if you could re-use that here or not.


- Alan Gates


On Oct. 15, 2015, 1:47 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39026/
> ---
> 
> (Updated Oct. 15, 2015, 1:47 a.m.)
> 
> 
> Review request for hive, Daniel Dai, Alan Gates, and Thejas Nair.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> .
> 
> 
> Diffs
> -
> 
>   metastore/pom.xml 2ab5cd3 
>   
> metastore/src/gen/protobuf/gen-java/org/apache/hadoop/hive/metastore/Metastore.java
>  PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> ccb4c98 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> 136eff5 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/PartitionExpressionProxy.java
>  5195481 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1968256 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseReadWrite.java 
> f69b4c7 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
> df0fac3 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  0f3331a 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  126a2c2 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/MockPartitionExpressionForMetastore.java
>  bae1391 
>   metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java 
> 78a9ea0 
>   metastore/src/test/org/apache/hadoop/hive/metastore/hbase/MockUtils.java 
> 6c288f4 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/FileMetaInfo.java 2853119 
>   ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 
> 690b8c9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionExpressionForMetastore.java
>  9ffa177 
> 
> Diff: https://reviews.apache.org/r/39026/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-12195) Unknown zones should cause an error instead of silently failing

2015-10-15 Thread Ryan Blue (JIRA) 
Ryan Blue created HIVE-12195:


 Summary: Unknown zones should cause an error instead of silently 
failing
 Key: HIVE-12195
 URL: https://issues.apache.org/jira/browse/HIVE-12195
 Project: Hive
  Issue Type: Sub-task
Reporter: Ryan Blue


Using an unknown time zone with the {{from_utc_timestamp}} or 
{{to_utc_timetamp}} methods returns the time un-adjusted instead of throwing an 
error:

{code}
hive> select from_utc_timestamp('2015-04-11 12:24:34.535', 'panda');
2015-04-11 12:24:34.535
{code}

This should be an error because users may attempt to adjust to valid but 
unknown zones, like PDT or MDT. This would produce incorrect results with no 
warning or error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: JIRA "Affects version/s" and "Fix version/s"

2015-10-15 Thread Eugene Koifman 
Can JIRA be made to enforce that these are set to something when the
ticket is Resolved?


On 10/15/15, 9:52 AM, "Xuefu Zhang"  wrote:

>Hi folks,
>
>I know most of us have done a good job in updating JIRAs with respective
>"Affects version/s" and "Fix version/s". However, I do see from time to
>time that these fields are missing in some JIRAs even after those JIRAs
>are
>closed.
>
>These field are very important in helping users to determine if their
>versions are affected or fixed.
>
>If for some reason either the field is not applicable, please use "n/a"
>specifically.
>
>Please feel free to share your thoughts or comments on this.
>
>Thanks,
>Xuefu

[jira] [Created] (HIVE-12194) Incorrect result when using from_utc_timestamp with the local time zone.

2015-10-15 Thread Ryan Blue (JIRA) 
Ryan Blue created HIVE-12194:


 Summary: Incorrect result when using from_utc_timestamp with the 
local time zone.
 Key: HIVE-12194
 URL: https://issues.apache.org/jira/browse/HIVE-12194
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 1.1.1
Reporter: Ryan Blue


When I call the {{from_utc_timestamp}} function (or {{to_utc_timestamp}}) using 
my current time zone, the result is incorrect:

{code}
// CURRENT SERVER TIME ZONE IS PDT
hive> select to_utc_timestamp('2015-10-13 09:15:34.101', 'PDT');
2015-10-13 09:15:34.101 // NOT CHANGED!
hive> select to_utc_timestamp('2015-10-13 09:15:34.101', 'PST');
2015-10-13 16:15:34.101 // CORRECT VALUE FOR PST
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12193) Add a consistent unix_timestamp function.

2015-10-15 Thread Ryan Blue (JIRA) 
Ryan Blue created HIVE-12193:


 Summary: Add a consistent unix_timestamp function.
 Key: HIVE-12193
 URL: https://issues.apache.org/jira/browse/HIVE-12193
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 1.1.1
Reporter: Ryan Blue


The {{unix_timestamp}} function returns values with respect to the SQL session 
time zone (which is the default JVM time zone). This varies depending on server 
time zone. This is required by the documentation for the function and would be 
difficult to change.

For users that want consistent results across zones, Hive should include a 
{{utc_timestamp}} method that is zone independent and gives a result assuming 
the timestamp without time zone passed in is in UTC.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12192) Hive should carry out timestamp computations in UTC

2015-10-15 Thread Ryan Blue (JIRA) 
Ryan Blue created HIVE-12192:


 Summary: Hive should carry out timestamp computations in UTC
 Key: HIVE-12192
 URL: https://issues.apache.org/jira/browse/HIVE-12192
 Project: Hive
  Issue Type: Sub-task
Reporter: Ryan Blue


Hive currently uses the "local" time of a java.sql.Timestamp to represent the 
SQL data type TIMESTAMP WITHOUT TIME ZONE. The purpose is to be able to use 
{{Timestamp#getYear()}} and similar methods to implement SQL functions like 
{{year}}.

When the SQL session's time zone is a DST zone, such as America/Los_Angeles 
that alternates between PST and PDT, there are times that cannot be represented 
because the effective zone skips them.

{code}
hive> select TIMESTAMP '2015-03-08 02:10:00.101';
2015-03-08 03:10:00.101
{code}

Using UTC instead of the SQL session time zone as the underlying zone for a 
java.sql.Timestamp avoids this bug, while still returning correct values for 
{{getYear}} etc. Using UTC as the convenience representation (timestamp without 
time zone has no real zone) would make timestamp calculations more consistent 
and avoid similar problems in the future.

Notably, this would break the {{unix_timestamp}} UDF that specifies the result 
is with respect to ["the default timezone and default 
locale"|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-DateFunctions].
 That function would need to be updated to use the 
{{System.getProperty("user.timezone")}} zone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12191) Hive timestamp problems

2015-10-15 Thread Ryan Blue (JIRA) 
Ryan Blue created HIVE-12191:


 Summary: Hive timestamp problems
 Key: HIVE-12191
 URL: https://issues.apache.org/jira/browse/HIVE-12191
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 1.1.1
Reporter: Ryan Blue


This is an umbrella JIRA for problems found with Hive's timestamp (without time 
zone) implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12190) new attribute viewOriginalText on HCatTable class

2015-10-15 Thread DAG support (JIRA) 
DAG support created HIVE-12190:
--

 Summary: new attribute viewOriginalText on HCatTable class
 Key: HIVE-12190
 URL: https://issues.apache.org/jira/browse/HIVE-12190
 Project: Hive
  Issue Type: New Feature
  Components: HCatalog
Reporter: DAG support
Priority: Minor


The HCatTable object returned by the HCatClient does not have the view SQL(view 
definition).
As a workaround we are using the Hive Metastore API to get the view SQL from 
the Table (getViewOriginalText() method) object returned by the 
HiveMetaStoreClient.
Please add this new attribute viewOriginalText to the HCatTable class.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38292: HIVE-11768 java.io.DeleteOnExitHook leaks memory on long running Hive Server2 Instances

2015-10-15 Thread Thejas Nair 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38292/#review102808
---

Ship it!


Ship It!

- Thejas Nair


On Oct. 15, 2015, 12:35 a.m., Navis Ryu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38292/
> ---
> 
> (Updated Oct. 15, 2015, 12:35 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-11768
> https://issues.apache.org/jira/browse/HIVE-11768
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> More than 490,000 paths was added to java.io.DeleteOnExitHook on one of our 
> long running HiveServer2 instances,taken up more than 100MB on heap.
>   Most of the paths contains a suffix of ".pipeout".
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 7e4f386 
>   common/src/java/org/apache/hive/common/util/ShutdownHookManager.java 
> fd2f20a 
>   common/src/test/org/apache/hive/common/util/TestShutdownHookManager.java 
> fa30f15 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 92ac209 
>   
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
>  1d1e995 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 175348b 
> 
> Diff: https://reviews.apache.org/r/38292/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Navis Ryu
> 
>

[jira] [Created] (HIVE-12189) The list in pushdownPreds of ppd.ExprWalkerInfo should not be allowed to grow very large

2015-10-15 Thread Yongzhi Chen (JIRA) 
Yongzhi Chen created HIVE-12189:
---

 Summary: The list in pushdownPreds of ppd.ExprWalkerInfo should 
not be allowed to grow very large
 Key: HIVE-12189
 URL: https://issues.apache.org/jira/browse/HIVE-12189
 Project: Hive
  Issue Type: Bug
  Components: Logical Optimizer
Affects Versions: 1.1.0, 2.0.0
Reporter: Yongzhi Chen
Assignee: Yongzhi Chen


Some queries are very slow in compile time, for example following query
{noformat}
select * from tt1 nf 
join tt2 a1 on (nf.col1 = a1.col1 and nf.hdp_databaseid = a1.hdp_databaseid) 
join tt3 a2 on(a2.col2 = a1.col2 and a2.col3 = nf.col3 and 
a2.hdp_databaseid = nf.hdp_databaseid) 
join tt4 a3 on  (a3.col4 = a2.col4 and a3.col3 = a2.col3) 
join tt5 a4 on (a4.col4 = a2.col4 and a4.col5 = a2.col5 and a4.col3 = 
a2.col3 and a4.hdp_databaseid = nf.hdp_databaseid) 
join tt6 a5 on  (a5.col3 = a2.col3 and a5.col2 = a2.col2 and 
a5.hdp_databaseid = nf.hdp_databaseid) 
JOIN tt7 a6 ON (a2.col3 = a6.col3 and a2.col2 = a6.col2 and a6.hdp_databaseid = 
nf.hdp_databaseid) 
JOIN tt8 a7 ON (a2.col3 = a7.col3 and a2.col2 = a7.col2 and a7.hdp_databaseid = 
nf.hdp_databaseid)
where nf.hdp_databaseid = 102 limit 10;
{noformat}
takes around 120 seconds to compile in hive 1.1 when
hive.mapred.mode=strict;
hive.optimize.ppd=true;
and hive is not in test mode.
All the above tables are tables with one column as partition. But all the 
tables are empty table. If the tables are not empty, it is reported that the 
compile so slow that it looks like hive is hanging. 
In hive 2.0, the compile is much faster, explain takes 6.6 seconds. But it is 
still a lot of time. One of the problem slows ppd down is that list in 
pushdownPreds can grow very large which makes extractPushdownPreds bad 
performance:
{noformat}
public static ExprWalkerInfo extractPushdownPreds(OpWalkerInfo opContext,
Operator op, List preds)
{noformat}
During run the query above, in the following break point preds  has size of 
12051, and most entry of the list is: GenericUDFOPEqual(Column[hdp_databaseid], 
Const int 102), GenericUDFOPEqual(Column[hdp_databaseid], Const int 102), 
GenericUDFOPEqual(Column[hdp_databaseid], Const int 102), 
GenericUDFOPEqual(Column[hdp_databaseid], Const int 102), 
Following code in extractPushdownPreds will clone all the nodes in preds and do 
the walk. Hive 2.0 is faster because HIVE-11652 makes startWalking much faster, 
but we still clone thousands of nodes with same expression. Should we store so 
many same predicates in the list or just one is good enough?  

{noformat}
List startNodes = new ArrayList();
List clonedPreds = new ArrayList();
for (ExprNodeDesc node : preds) {
  ExprNodeDesc clone = node.clone();
  clonedPreds.add(clone);
  exprContext.getNewToOldExprMap().put(clone, node);
}
startNodes.addAll(clonedPreds);

egw.startWalking(startNodes, null);

{noformat}

Should we change java/org/apache/hadoop/hive/ql/ppd/ExprWalkerInfo.java
method 
public void addFinalCandidate(String alias, ExprNodeDesc expr) 
and
public void addPushDowns(String alias, List pushDowns) 

to only add expr which is not in the PushDown list for an alias?




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12188) DoAs does not work properly in non-kerberos secured HS2

2015-10-15 Thread Chaoyu Tang (JIRA) 
Chaoyu Tang created HIVE-12188:
--

 Summary: DoAs does not work properly in non-kerberos secured HS2
 Key: HIVE-12188
 URL: https://issues.apache.org/jira/browse/HIVE-12188
 Project: Hive
  Issue Type: Bug
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang


The case with following settings is valid but it seems still not work correctly 
in current HS2
==
hive.server2.authentication=NONE (or LDAP)
hive.server2.enable.doAs= true
hive.metastore.sasl.enabled=true (with HMS Kerberos enabled)
==
Currently HS2 is able to fetch the delegation token to a kerberos secured HMS 
only when itself is also kerberos secured.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12187) Release plan once a query is executed

2015-10-15 Thread Jimmy Xiang (JIRA) 
Jimmy Xiang created HIVE-12187:
--

 Summary: Release plan once a query is executed 
 Key: HIVE-12187
 URL: https://issues.apache.org/jira/browse/HIVE-12187
 Project: Hive
  Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


Some clients leave query operations open for a while so that they can retrieve 
the query results later. That means the allocated memory will be kept around 
too. We should release those resources not needed for query execution any more 
once it is executed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

JIRA "Affects version/s" and "Fix version/s"

2015-10-15 Thread Xuefu Zhang 
Hi folks,

I know most of us have done a good job in updating JIRAs with respective
"Affects version/s" and "Fix version/s". However, I do see from time to
time that these fields are missing in some JIRAs even after those JIRAs are
closed.

These field are very important in helping users to determine if their
versions are affected or fixed.

If for some reason either the field is not applicable, please use "n/a"
specifically.

Please feel free to share your thoughts or comments on this.

Thanks,
Xuefu

[jira] [Created] (HIVE-12186) Upgrade Hive to Calcite 1.5

2015-10-15 Thread Jesus Camacho Rodriguez (JIRA) 
Jesus Camacho Rodriguez created HIVE-12186:
--

 Summary: Upgrade Hive to Calcite 1.5
 Key: HIVE-12186
 URL: https://issues.apache.org/jira/browse/HIVE-12186
 Project: Hive
  Issue Type: Improvement
  Components: CBO, Logical Optimizer
Affects Versions: 2.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
 Fix For: 2.0.0


CLEAR LIBRARY CACHE

Upgrade Hive to Calcite 1.5.0-incubating.

There is currently a snapshot release, which is close to what will be in 1.5. 
First, we will test and check any possible issues against the snapshot, so we 
can upgrade quicker once the release is out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12185) setAutoCommit should only fail if auto commit is being disabled

2015-10-15 Thread Varadharajan (JIRA) 
Varadharajan created HIVE-12185:
---

 Summary: setAutoCommit should only fail if auto commit is being 
disabled
 Key: HIVE-12185
 URL: https://issues.apache.org/jira/browse/HIVE-12185
 Project: Hive
  Issue Type: Bug
  Components: JDBC
Affects Versions: 1.2.1, 1.1.0, 1.2.0
Reporter: Varadharajan
Assignee: Varadharajan
Priority: Minor


Auto Commit is enabled by default in hive as documented at 
https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Limitations.
 In accordance with this, HiveConnection.java:getAutoCommit method returns 
true. Similarly, setAutoCommit(true) should pass silently since it is already 
enabled and a SQL exception should be thrown if its called with false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)