Re: Review Request 40055: HIVE-12017

2015-11-18 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40055/
---

(Updated Nov. 18, 2015, 5:02 p.m.)


Review request for hive, Ashutosh Chauhan and John Pullokkaran.


Bugs: HIVE-12017
https://issues.apache.org/jira/browse/HIVE-12017


Repository: hive-git


Description (updated)
---

HIVE-12017


Diffs (updated)
-

  hbase-handler/src/test/results/positive/hbase_queries.q.out 
d044c7ed3874acaf521d83bdddfa02276bf71cb3 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOptUtil.java 
b4e7d47134357bc1e25af8642373ffb9babc015b 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateProjectMergeRule.java
 53f04ee72d8a614a602ada688f89d1febd467689 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
 a17fb9498557fc95f273240c1484d69f514fcad0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
de67b54a2c6cfd9bc4413ebf7f715e54c61b966f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
5323a7df342fa7b7f7b7f457d3d99d3407ed51a6 
  ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 
623c2e85a84919b41735913c3da32514f5d3ff22 
  ql/src/test/results/clientnegative/join_nonexistent_part.q.out 
391dd0592611d7af8484c52efde3a50fb7dfa44d 
  ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out 
aa380b20efee11a0a3a4c7acaeb9482444c1d3ce 
  ql/src/test/results/clientpositive/archive_excludeHadoop20.q.out 
c2b98727d21f4990ae7496a0a8fa9ac16598f4c0 
  ql/src/test/results/clientpositive/archive_multi.q.out 
0ad29d122153bd4adf4d19064188b0c4f94e05ab 
  ql/src/test/results/clientpositive/auto_join1.q.out 
48ad641788a6adfad5f7e4fcdfef3d67eac70a4e 
  ql/src/test/results/clientpositive/auto_join10.q.out 
fa6f62d18abbf517c4e49ac3fa9da190c23a119f 
  ql/src/test/results/clientpositive/auto_join11.q.out 
851920b9dce7d9fb8d105ef81404f3f67166ad15 
  ql/src/test/results/clientpositive/auto_join14.q.out 
47e1724ab18ac322a83f687fab37ea44c4fdf78a 
  ql/src/test/results/clientpositive/auto_join24.q.out 
5b573033d317e3e7dbf70f9b6ef253b35ac7c140 
  ql/src/test/results/clientpositive/auto_join26.q.out 
94ab76f750a2ce51a645012dcd5beb43b560445a 
  ql/src/test/results/clientpositive/auto_join32.q.out 
161ab6b377a644e62a94d69aa9d3bba02b8045e6 
  ql/src/test/results/clientpositive/auto_join_filters.q.out 
a6720d908f4c5a354cb4f3234f8c288249d35d2d 
  ql/src/test/results/clientpositive/auto_join_nulls.q.out 
4416f3e921a3590223658eb6b0e15c317733a7e2 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
f42b45e635ca5e271d48ea6bc48c8b0e45ac67d1 
  ql/src/test/results/clientpositive/auto_join_stats.q.out 
d75d6c42eba366905afb4e6e171402c50581ba05 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 
a0aefa3de8aa07ab7f4a634fcc22b29ba621a6c5 
  ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
1dc9cd07cddb5bce3b2369c1776b690bb239e050 
  ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out 
f1aadef724d6f10ca4a710a3d11382e2f01ca1e5 
  ql/src/test/results/clientpositive/auto_sortmerge_join_10.q.out 
fb1e6568de332e930e7836e09aef142f7f66eb17 
  ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out 
5dad0fb366d4e1fc21a9a7ba034d60c942e8664e 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
b1ba1483e1ab83c3f7ea71fddf5247bfc5dbde0b 
  ql/src/test/results/clientpositive/auto_sortmerge_join_14.q.out 
33c56fdc6d6f01377dd78e77b99c229ff437d802 
  ql/src/test/results/clientpositive/auto_sortmerge_join_15.q.out 
460e5b1b0f60c213f3a14172482a9a8f8e85454d 
  ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out 
a7a5faa8f8cc29ff53328e6db598cc0acf4cb68e 
  ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out 
dfb1a16529bb09de8eb154976386ae39b76420c8 
  ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out 
013bc07b6ef804d57632ab2840628e7903f3cb47 
  ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out 
d751e7052a14f2ca308699c3a52beb30f989d0a0 
  ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
853f6413ad519840c535e32ec680b6eedd13f457 
  ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out 
e2d797ba8d5083c2400df86aacaa94e22f71d809 
  ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out 
e3bb51d6ef8d69112acb75688c5eabefc628cab7 
  ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 
bbfa75608deef0a72df8104b6836105b296f7d29 
  ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out 
870ecdd37c3e4833dc1b182e98e6e4018f719fbe 
  ql/src/test/results/clientpositive/bucket_map_join_spark2.q.out 
33f5c46c2adf43a1e13f8af305da6c585998141a 
  ql/src/test/results/clientpositive/bucket_map_join_spark3.q.out 
067d12875a762d98126994f05090d4c4e4cd 
  ql/src/test/results/clientpositive/bucket_map_join_spark4.q.out

[jira] [Created] (HIVE-12455) Insert into Local Directory fails if partition exists without directory

2015-11-18 Thread rahul gulati (JIRA) 
rahul gulati created HIVE-12455:
---

 Summary: Insert into Local Directory fails if partition exists 
without directory
 Key: HIVE-12455
 URL: https://issues.apache.org/jira/browse/HIVE-12455
 Project: Hive
  Issue Type: Bug
  Components: Database/Schema
 Environment: Development
Reporter: rahul gulati


Insert Into local Directory fails if partition directory doesnot exist.
Command

Insert into Local Directory '/abc' select * from abc;

Above command fails but when i query the table like "select * from abc", it 
runs fine with no error.
Logs below :

Vertex failed, vertexName=Map 1, vertexId=vertex_1446903211205_3914_3_00, 
diagnostics=[Vertex vertex_1446903211205_3914_3_00 [Map 1] killed/failed due 
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: claim initializer failed, 
vertex=vertex_1446903211205_3914_3_00 [Map 1], 
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: 
hdfs://abc/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38663: HIVE-11878: ClassNotFoundException can possibly occur if multiple jars are registered one at a time in Hive

2015-11-18 Thread Anthony Hsu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38663/#review107063
---

Ship it!


Revision looks good to me.


ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java (lines 369 - 
370)


You could also use `new String[0]`.


- Anthony Hsu


On 十一月 18, 2015, 5:21 a.m., Ratandeep Ratti wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38663/
> ---
> 
> (Updated 十一月 18, 2015, 5:21 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-11878
> https://issues.apache.org/jira/browse/HIVE-11878
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11878: ClassNotFoundException can possibly occur if multiple jars are 
> registered one at a time in Hive
> 
> 
> Diffs
> -
> 
>   conf/ivysettings.xml bda842a89bb07710fdcd7180a00833a7388ada8f 
>   itests/custom-udfs/pom.xml PRE-CREATION 
>   itests/custom-udfs/udf-classloader-udf1/pom.xml PRE-CREATION 
>   
> itests/custom-udfs/udf-classloader-udf1/src/main/java/hive/it/custom/udfs/UDF1.java
>  PRE-CREATION 
>   itests/custom-udfs/udf-classloader-udf2/pom.xml PRE-CREATION 
>   
> itests/custom-udfs/udf-classloader-udf2/src/main/java/hive/it/custom/udfs/UDF2.java
>  PRE-CREATION 
>   itests/custom-udfs/udf-classloader-util/pom.xml PRE-CREATION 
>   
> itests/custom-udfs/udf-classloader-util/src/main/java/hive/it/custom/udfs/Util.java
>  PRE-CREATION 
>   itests/pom.xml 0686f1fd58c2be26b2ee645c4e244159aec565e5 
>   itests/qtest/pom.xml 8db6fb04d0a5d4600bc23543a0215d31c1cd0648 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/UDFClassLoader.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
> de2eb984159526048e8dacf71d3ff8b0647394a3 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> ff875df98e1dd64a8af3ad22f4b38dbc1d6a1923 
>   ql/src/test/queries/clientpositive/udf_classloader.q PRE-CREATION 
>   
> ql/src/test/queries/clientpositive/udf_classloader_dynamic_dependency_resolution.q
>  PRE-CREATION 
>   ql/src/test/results/clientpositive/udf_classloader.q.out PRE-CREATION 
>   
> ql/src/test/results/clientpositive/udf_classloader_dynamic_dependency_resolution.q.out
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38663/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ratandeep Ratti
> 
>

Re: hive transaction strange behaviour

2015-11-18 Thread Eugene Koifman 
can you send ls -l on the partition where you expect a base and don't see it?

From: Sanjeev Verma 
>
Reply-To: "u...@hive.apache.org" 
>
Date: Tuesday, November 17, 2015 at 10:27 PM
To: "u...@hive.apache.org" 
>
Cc: "dev@hive.apache.org" 
>
Subject: Re: hive transaction strange behaviour

Any help will be much appreciated.Thanks

On Tue, Nov 17, 2015 at 2:39 PM, Sanjeev Verma 
> wrote:
Thank Elliot, Eugene
I am able to see the Base file created in one of the partition, seems the 
Compactor kicked in and created it but it has not created base files in rest of 
the partition where delta files still exists.why compactor has not picked the 
other partition, when and how these partition will be picked up for compaction.

Thanks

On Sat, Nov 14, 2015 at 11:01 PM, Eugene Koifman 
> wrote:
When Compaction process runs, it will create base directory.
https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration

at a minimum you need 
hive.compactor.initiator.on=true
 and 
hive.compactor.worker.threads>0

Also, see 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionCompact
 on how to trigger compaction manually.

Eugene

From: Sanjeev Verma 
>
Reply-To: "u...@hive.apache.org" 
>
Date: Thursday, November 12, 2015 at 11:41 PM
To: "u...@hive.apache.org" 
>, 
"dev@hive.apache.org" 
>
Subject: hive transaction strange behaviour

I have enable the hive transaction and able to see the delta files created for 
some of the partition but i dont not see any base file created 
yet.it seems strange to me seeing so many delta files without 
any base file.
Could somebody let me know when Base file created.

Thanks

[jira] [Created] (HIVE-12456) QueryId can't be stored in the configuration of the SessionState since multiple queries can run in a single session

2015-11-18 Thread Aihua Xu (JIRA) 
Aihua Xu created HIVE-12456:
---

 Summary: QueryId can't be stored in the configuration of the 
SessionState since multiple queries can run in a single session
 Key: HIVE-12456
 URL: https://issues.apache.org/jira/browse/HIVE-12456
 Project: Hive
  Issue Type: Bug
  Components: Logging
Affects Versions: 2.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


Follow up on HIVE-11488 which stores the queryId in the sessionState conf. If 
multiple queries run at  the same time, then the logging will get wrong queryId 
from the sessionState.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40055: HIVE-12017

2015-11-18 Thread Jesús Camacho Rodríguez 


> On Nov. 17, 2015, 7:39 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java, lines 
> > 886-892
> > 
> >
> > All these rules don't need stats, it will be better to move them out of 
> > try block, so that we still leverage them even when stats are not available.

I think we should not remove them from there, as they might interact with the 
join ordering rules. But I could execute them in the catch block instead.


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40055/#review106922
---


On Nov. 16, 2015, 6:16 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40055/
> ---
> 
> (Updated Nov. 16, 2015, 6:16 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Bugs: HIVE-12017
> https://issues.apache.org/jira/browse/HIVE-12017
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Do not disable CBO by default when number of joins in a query is equal or 
> less than 1
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> d044c7ed3874acaf521d83bdddfa02276bf71cb3 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOptUtil.java 
> b4e7d47134357bc1e25af8642373ffb9babc015b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateProjectMergeRule.java
>  53f04ee72d8a614a602ada688f89d1febd467689 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
>  a17fb9498557fc95f273240c1484d69f514fcad0 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> de67b54a2c6cfd9bc4413ebf7f715e54c61b966f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 1ca113cba30b26a38abf4910aafd0bec2bbd9a51 
>   ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 
> 623c2e85a84919b41735913c3da32514f5d3ff22 
>   ql/src/test/results/clientnegative/join_nonexistent_part.q.out 
> 391dd0592611d7af8484c52efde3a50fb7dfa44d 
>   ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out 
> aa380b20efee11a0a3a4c7acaeb9482444c1d3ce 
>   ql/src/test/results/clientpositive/archive_excludeHadoop20.q.out 
> c2b98727d21f4990ae7496a0a8fa9ac16598f4c0 
>   ql/src/test/results/clientpositive/archive_multi.q.out 
> 0ad29d122153bd4adf4d19064188b0c4f94e05ab 
>   ql/src/test/results/clientpositive/auto_join1.q.out 
> 48ad641788a6adfad5f7e4fcdfef3d67eac70a4e 
>   ql/src/test/results/clientpositive/auto_join10.q.out 
> fa6f62d18abbf517c4e49ac3fa9da190c23a119f 
>   ql/src/test/results/clientpositive/auto_join11.q.out 
> 851920b9dce7d9fb8d105ef81404f3f67166ad15 
>   ql/src/test/results/clientpositive/auto_join14.q.out 
> 47e1724ab18ac322a83f687fab37ea44c4fdf78a 
>   ql/src/test/results/clientpositive/auto_join24.q.out 
> 5b573033d317e3e7dbf70f9b6ef253b35ac7c140 
>   ql/src/test/results/clientpositive/auto_join26.q.out 
> 94ab76f750a2ce51a645012dcd5beb43b560445a 
>   ql/src/test/results/clientpositive/auto_join32.q.out 
> 161ab6b377a644e62a94d69aa9d3bba02b8045e6 
>   ql/src/test/results/clientpositive/auto_join_filters.q.out 
> a6720d908f4c5a354cb4f3234f8c288249d35d2d 
>   ql/src/test/results/clientpositive/auto_join_nulls.q.out 
> 4416f3e921a3590223658eb6b0e15c317733a7e2 
>   ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
> f42b45e635ca5e271d48ea6bc48c8b0e45ac67d1 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out 
> d75d6c42eba366905afb4e6e171402c50581ba05 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 
> a0aefa3de8aa07ab7f4a634fcc22b29ba621a6c5 
>   ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
> 1dc9cd07cddb5bce3b2369c1776b690bb239e050 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out 
> f1aadef724d6f10ca4a710a3d11382e2f01ca1e5 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_10.q.out 
> fb1e6568de332e930e7836e09aef142f7f66eb17 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out 
> 5dad0fb366d4e1fc21a9a7ba034d60c942e8664e 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
> b1ba1483e1ab83c3f7ea71fddf5247bfc5dbde0b 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_14.q.out 
> 33c56fdc6d6f01377dd78e77b99c229ff437d802 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_15.q.out 
> 460e5b1b0f60c213f3a14172482a9a8f8e85454d 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out 
> a7a5faa8f8cc29ff53328e6db598cc0acf4cb68e 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out 
>

Re: Review Request 38211: HIVE-11684

2015-11-18 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/
---

(Updated Nov. 18, 2015, 10:40 a.m.)


Review request for hive and John Pullokkaran.


Bugs: HIVE-11684
https://issues.apache.org/jira/browse/HIVE-11684


Repository: hive-git


Description
---

Implement limit pushdown through outer join in CBO


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
953e52c4de49bbc083647033b370a1ca80f5cb97 
  itests/src/test/resources/testconfiguration.properties 
402914cdfb4d72998a9c219e812d561e67612897 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java
 82fdc0e04011aa585912ce5aae5e3245e5e5bb01 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
 728c5aa91f8d6eab6bcc2696c059ae1bdbeea689 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java
 715f24fc1872a54637a47bf80bb29e4a2e053779 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
de67b54a2c6cfd9bc4413ebf7f715e54c61b966f 
  ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION 
  ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION 
  ql/src/test/results/clientpositive/join32_lessSize.q.out 
5ea40241d329617220bbe775dcba91a9d7fb2f89 
  ql/src/test/results/clientpositive/join_merging.q.out 
b42bdc5ee92e9d725b7c0b8323c7763a55f3d059 
  ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/join32_lessSize.q.out 
937e8fcb0c7b89c13548a01e182176102ebcb681 
  ql/src/test/results/clientpositive/spark/join_merging.q.out 
6c5089b4868e839a2e72546a39a7a5d4741464ed 
  ql/src/test/results/clientpositive/subquery_notin.q.out 
5563794f9761a8269a56c7e716545c5edb1e4076 
  ql/src/test/results/clientpositive/subquery_notin_having.q.java1.7.out 
9689ae37dfedc8c3becd6915e1e4cd5a5da29f9b 
  ql/src/test/results/clientpositive/tez/explainuser_1.q.out 
fa29dfec8fe38fcbec2cfa22859821898e07dbaa 
  ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/38211/diff/


Testing
---

Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only 
Tez)


Thanks,

Jesús Camacho Rodríguez

[jira] [Created] (HIVE-12454) in tez model when i use add jar xxx query will fail

2015-11-18 Thread Feng Yuan (JIRA) 
Feng Yuan created HIVE-12454:


 Summary: in tez model when i use add jar xxx query will fail
 Key: HIVE-12454
 URL: https://issues.apache.org/jira/browse/HIVE-12454
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 0.14.0
Reporter: Feng Yuan
 Fix For: 1.3.0


whatever hql only if i use add jar udf.jar hql will throws out these:

Status: Running (Executing on YARN cluster with App id 
application_1447448264041_0723)


VERTICES  STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED

Map 1 FAILED -1  00   -1   0   0
Map 2 FAILED -1  00   -1   0   0

VERTICES: 00/02  [>>--] 0%ELAPSED TIME: 0.27 s 

Status: Failed
Vertex failed, vertexName=Map 2, vertexId=vertex_1447448264041_0723_1_00, 
diagnostics=[Vertex vertex_1447448264041_0723_1_00 [Map 2] killed/failed due 
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: mid_bdi_customer_online initializer 
failed, vertex=vertex_1447448264041_0723_1_00 [Map 2], 
java.lang.NoSuchMethodError: 
org.apache.hadoop.hive.shims.HadoopShims.getMergedCredentials(Lorg/apache/hadoop/mapred/JobConf;)V
at 
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:104)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:245)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:239)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:239)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
]
Vertex failed, vertexName=Map 1, vertexId=vertex_1447448264041_0723_1_01, 
diagnostics=[Vertex vertex_1447448264041_0723_1_01 [Map 1] killed/failed due 
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: raw_kafka_event_pageview_dt0 
initializer failed, vertex=vertex_1447448264041_0723_1_01 [Map 1], 
java.lang.NoSuchMethodError: 
org.apache.hadoop.hive.shims.HadoopShims.getMergedCredentials(Lorg/apache/hadoop/mapred/JobConf;)V
at 
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:104)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:245)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:239)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:239)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
]
DAG failed due to vertex failure. failedVertices:2 killedVertices:0
FAILED: Execution Error, return code 2 from 
org.apache.hadoop.hive.ql.exec.tez.TezTask

sometime is NoSuchMethodError other package such as  
org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogBegin(Ljava/lang/String;Ljava/lang/String;)V
or something.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 40467: HIVE-12075 analyze for file metadata

2015-11-18 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40467/
---

Review request for hive, Alan Gates and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2185f85 
  itests/src/test/resources/testconfiguration.properties a33e720 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 1d0fdf0 
  metastore/if/hive_metastore.thrift bb754f1 
  metastore/src/java/org/apache/hadoop/hive/metastore/FileMetadataHandler.java 
7c3525a 
  metastore/src/java/org/apache/hadoop/hive/metastore/FileMetadataManager.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
a835f6a 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
c5e7a5f 
  metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
aa96f77 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
02cbd76 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 803c6e7 
  
metastore/src/java/org/apache/hadoop/hive/metastore/PartitionExpressionProxy.java
 ed59829 
  metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 5b36b03 
  
metastore/src/java/org/apache/hadoop/hive/metastore/filemeta/OrcFileMetadataHandler.java
 14189da 
  metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseReadWrite.java 
2fb3e8f 
  metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
98e6c75 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 9a1d159 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 8dde0af 
  
metastore/src/test/org/apache/hadoop/hive/metastore/MockPartitionExpressionForMetastore.java
 d72bf76 
  metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java 
9089d1c 
  metastore/src/test/org/apache/hadoop/hive/metastore/hbase/MockUtils.java 
983129a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9ab3e98 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 488d923 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionExpressionForMetastore.java
 f9978b4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/AnalyzeCommandUtils.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
543bc0f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java eea2fcc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g e9fbfb1 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 009e72e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
0affe84 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CacheMetadataDesc.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java a4c3db1 
  ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java af7e43e 
  ql/src/test/queries/clientpositive/stats_filemetadata.q PRE-CREATION 
  ql/src/test/results/clientpositive/tez/stats_filemetadata.q.out PRE-CREATION 
  shims/common/src/main/java/org/apache/hadoop/hive/io/HdfsUtils.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/40467/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-12460) Fix branch-1 build

2015-11-18 Thread Xuefu Zhang (JIRA) 
Xuefu Zhang created HIVE-12460:
--

 Summary: Fix branch-1 build
 Key: HIVE-12460
 URL: https://issues.apache.org/jira/browse/HIVE-12460
 Project: Hive
  Issue Type: Bug
  Components: Build Infrastructure
Affects Versions: 1.3.0
Reporter: Xuefu Zhang


Caused by a merge.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12458) remove identity_udf.jar from source

2015-11-18 Thread Thejas M Nair (JIRA) 
Thejas M Nair created HIVE-12458:


 Summary: remove identity_udf.jar from source
 Key: HIVE-12458
 URL: https://issues.apache.org/jira/browse/HIVE-12458
 Project: Hive
  Issue Type: Bug
  Components: Test
Reporter: Thejas M Nair
Assignee: Vaibhav Gumashta


We should not be checking in jars into the source repo.
We could use hive-contrib jar like its used in 

./ql/src/test/queries/clientpositive/add_jar_pfile.q 
add jar pfile://${system:test.tmp.dir}/hive-contrib-${system:hive.version}.jar;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Dev meetup Nov 2015

2015-11-18 Thread Thejas Nair 
Thanks to all who attended the meetup. There were great talks and discussions!

Here are the slides -
https://cwiki.apache.org/confluence/display/Hive/Presentations#Presentations-November2015HiveContributorMeetup


On Tue, Nov 10, 2015 at 12:27 PM, Xuefu Zhang  wrote:
> HI folks,
>
> Here is the talks that are scheduled for Monday's meetup. As you can see
> that we have a full schedule, speakers please constrain your talk within
> the given slot.
>
> 1. Lessons learned from enterprise users (Xuefu, 5m)
> 2. New Hive metrics and monitoring for HS2/HMS (TBD, 10m)
> 3. Hardening Hive - Design/Code Reviews (John Pullokkaran, 15m)
> 4. Correctness issue reporting and tracking guideline (Carl, 5m)
> 5. Dynamic dependency resolution and classloader issues (Anthony, 10m)
> 6. A query syntax validation tool for Hive. (Szehon, 15m)
> 7. Improved integration test for Hive (HIVE-12316) (Alan Gates, 30m)
> 8. Uber's plan for Hive and our expected feature from the overall dev
> community (Mohammad Islam,  15m)
> 9. Update on LLAP (Gopal V Vijayaraghavan, 10m)
>
> See you all there!
>
> --Xuefu
>
> On Fri, Nov 6, 2015 at 2:11 PM, Xuefu Zhang  wrote:
>
>> Hi all,
>>
>> As an update, here is the link to the meetup:
>> http://www.meetup.com/Hive-Contributors-Group/events/226495286/. Please
>> RSVP as soon as possible because the organizers do need a headcount before
>> the event.
>>
>> Also, we are finalizing a list of talks. If you plan to give a talk,
>> please let the organizer know.
>>
>> Thanks,
>> Xuefu
>>
>>
>> On Mon, Nov 2, 2015 at 10:09 AM, Xuefu Zhang  wrote:
>>
>>> Hi all,
>>>
>>> Apache Hive community is considering a developer meetup in the middle of
>>> this month. While we are finalizing the venue, I'd like to invite you to
>>> plan to join, and even better, to give talk in the event.
>>>
>>> We consider the center of the meetup would be around the quality aspect
>>> of the Project, and like to establish a collaborative effort in improving
>>> Hive's quality. In the meantime, any topic will be considered.
>>>
>>> Please let me or Thejas know if you'd like to give a talk.
>>>
>>> Detailed venue and agenda will be given soon. Look forward to meeting you
>>> in two weeks!
>>>
>>> Regards,
>>>
>>> Xuefu
>>>
>>
>>

Re: Dev meetup Nov 2015

2015-11-18 Thread Gopal Vijayaraghavan 
Hi,

I've attached the LLAP Slides from the meetup to that.

For those who missed the meeting, I've recreated the demo as a video.

http://people.apache.org/~gopalv/LLAP.gif

TL;DR - The data-set is the TPC-DS 1000Gb scale data-set & I'm running
2x160Gb LLAP instances (16 physical cores each), with 16Gb caches (so
net-cache size of 32Gb).


LLAP  - 890 milliseconds
Tez   - 9.47 seconds
MR2   - 183 seconds


Cheers,
Gopal

On 11/18/15, 2:00 PM, "Thejas Nair"  wrote:

>Thanks to all who attended the meetup. There were great talks and
>discussions!
>
>Here are the slides -
>https://cwiki.apache.org/confluence/display/Hive/Presentations#Presentatio
>ns-November2015HiveContributorMeetup
>
>
>On Tue, Nov 10, 2015 at 12:27 PM, Xuefu Zhang  wrote:
>> HI folks,
>>
>> Here is the talks that are scheduled for Monday's meetup. As you can see
>> that we have a full schedule, speakers please constrain your talk within
>> the given slot.
>>
>> 1. Lessons learned from enterprise users (Xuefu, 5m)
>> 2. New Hive metrics and monitoring for HS2/HMS (TBD, 10m)
>> 3. Hardening Hive - Design/Code Reviews (John Pullokkaran, 15m)
>> 4. Correctness issue reporting and tracking guideline (Carl, 5m)
>> 5. Dynamic dependency resolution and classloader issues (Anthony, 10m)
>> 6. A query syntax validation tool for Hive. (Szehon, 15m)
>> 7. Improved integration test for Hive (HIVE-12316) (Alan Gates, 30m)
>> 8. Uber's plan for Hive and our expected feature from the overall dev
>> community (Mohammad Islam,  15m)
>> 9. Update on LLAP (Gopal V Vijayaraghavan, 10m)
>>
>> See you all there!
>>
>> --Xuefu
>>
>> On Fri, Nov 6, 2015 at 2:11 PM, Xuefu Zhang  wrote:
>>
>>> Hi all,
>>>
>>> As an update, here is the link to the meetup:
>>> http://www.meetup.com/Hive-Contributors-Group/events/226495286/. Please
>>> RSVP as soon as possible because the organizers do need a headcount
>>>before
>>> the event.
>>>
>>> Also, we are finalizing a list of talks. If you plan to give a talk,
>>> please let the organizer know.
>>>
>>> Thanks,
>>> Xuefu
>>>
>>>
>>> On Mon, Nov 2, 2015 at 10:09 AM, Xuefu Zhang 
>>>wrote:
>>>
 Hi all,

 Apache Hive community is considering a developer meetup in the middle
of
 this month. While we are finalizing the venue, I'd like to invite you
to
 plan to join, and even better, to give talk in the event.

 We consider the center of the meetup would be around the quality
aspect
 of the Project, and like to establish a collaborative effort in
improving
 Hive's quality. In the meantime, any topic will be considered.

 Please let me or Thejas know if you'd like to give a talk.

 Detailed venue and agenda will be given soon. Look forward to meeting
you
 in two weeks!

 Regards,

 Xuefu

>>>
>>>
>

[jira] [Created] (HIVE-12459) Tez startup dislikes spaces in classpath on Windows

2015-11-18 Thread john lilley (JIRA) 
john lilley created HIVE-12459:
--

 Summary: Tez startup dislikes spaces in classpath on Windows
 Key: HIVE-12459
 URL: https://issues.apache.org/jira/browse/HIVE-12459
 Project: Hive
  Issue Type: Bug
  Components: Tez
Affects Versions: 0.14.0
Reporter: john lilley
Assignee: Vikram Dixit K


We are seeing an issue that I summarize as “Tez doesn’t like spaces in the 
classpath”, but I wanted to check with the group before submitting a JIRA.  
This is showing when we try to access Hive on HDP 2.3 from a Windows client, 
where we’ve put the client jars in a classpath that contains spaces.  

The causing line in our code is:
state = SessionState.start(hiveConf);
where SessionState is in org.apache.hadoop.hive.ql.session

The exception stack is:

net/redpoint/hiveclient/DMHCatClient.newInstance:java.lang.RuntimeException: 
java.io.FileNotFoundException: File 
file:/C:/Program%20Files/RedPointDM7/hadoop/clusters/hds-cent6/lib/hive-exec-1.2.1.2.3.0.0-2557.jar
 does not exist
Additional message: 
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:535)
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:466)
net.redpoint.hiveclient.DMHCatClient.(DMHCatClient.java:255)
net.redpoint.hiveclient.DMHCatClient.newInstance(DMHCatClient.java:59)
Caused by: java.io.FileNotFoundException: File 
file:/C:/Program%20Files/RedPointDM7/hadoop/clusters/hds-cent6/lib/hive-exec-1.2.1.2.3.0.0-2557.jar
 does not exist

org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:606)

org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:819)

org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:596)

org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)

org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:140)
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)

org.apache.hadoop.hive.ql.exec.tez.TezSessionState.getSha(TezSessionState.java:356)

org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:332)

org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:151)

org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:116)
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:532)
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:466)
net.redpoint.hiveclient.DMHCatClient.(DMHCatClient.java:255)
net.redpoint.hiveclient.DMHCatClient.newInstance(DMHCatClient.java:59)

It sure looks like something in the client code is turning “C:/Program Files” 
into “C:/Program%20Files”.  I am certain that it is not our code, because we 
otherwise access all of the jars and Java classes just fine.

Furthermore, disabling Tez for client-side Hive query in the configuration 
seems to fix or avoid the issue:
theConfiguration.set("hive.execution.engine", "mr");

The stack trace doesn’t make sense to me, because we use FileSystem all over 
the place and it doesn’t run into this problem when accessing HDFS, only when 
connecting to Hive.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12461) Branch-1 -Phadoop-1 build is broken

2015-11-18 Thread Xuefu Zhang (JIRA) 
Xuefu Zhang created HIVE-12461:
--

 Summary: Branch-1 -Phadoop-1 build is broken
 Key: HIVE-12461
 URL: https://issues.apache.org/jira/browse/HIVE-12461
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.3.0
Reporter: Xuefu Zhang


{code}
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ hive-exec ---
[INFO] Compiling 2423 source files to 
/Users/xzhang/apache/hive-git-commit/ql/target/classes
[INFO] -
[ERROR] COMPILATION ERROR : 
[INFO] -
[ERROR] 
/Users/xzhang/apache/hive-git-commit/ql/src/java/org/apache/hadoop/hive/ql/Context.java:[352,10]
 error: cannot find symbol
[INFO] 1 error
[INFO] -
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Hive ... SUCCESS [  2.636 s]
[INFO] Hive Shims Common .. SUCCESS [  3.270 s]
[INFO] Hive Shims 0.20S ... SUCCESS [  1.052 s]
[INFO] Hive Shims 0.23  SUCCESS [  3.550 s]
[INFO] Hive Shims Scheduler ... SUCCESS [  1.076 s]
[INFO] Hive Shims . SUCCESS [  1.472 s]
[INFO] Hive Common  SUCCESS [  5.989 s]
[INFO] Hive Serde . SUCCESS [  6.923 s]
[INFO] Hive Metastore . SUCCESS [ 19.424 s]
[INFO] Hive Ant Utilities . SUCCESS [  0.516 s]
[INFO] Spark Remote Client  SUCCESS [  3.305 s]
[INFO] Hive Query Language  FAILURE [ 34.276 s]
[INFO] Hive Service ... SKIPPED
{code}

Part of the code that's being complained:
{code}
343   /**
344* Remove any created scratch directories.
345*/
346   public void removeScratchDir() {
347 for (Map.Entry entry : fsScratchDirs.entrySet()) {
348   try {
349 Path p = entry.getValue();
350 FileSystem fs = p.getFileSystem(conf);
351 fs.delete(p, true);
352 fs.cancelDeleteOnExit(p);
353   } catch (Exception e) {
354 LOG.warn("Error Removing Scratch: "
355 + StringUtils.stringifyException(e));
356   }
{code}
might be related to HIVE-12268.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Dev meetup Nov 2015

2015-11-18 Thread Dmitry Tolpeko 
Thejas, thank you for sharing the slides!

On Thu, Nov 19, 2015 at 1:00 AM, Thejas Nair  wrote:

> Thanks to all who attended the meetup. There were great talks and
> discussions!
>
> Here are the slides -
>
> https://cwiki.apache.org/confluence/display/Hive/Presentations#Presentations-November2015HiveContributorMeetup
>
>
> On Tue, Nov 10, 2015 at 12:27 PM, Xuefu Zhang  wrote:
> > HI folks,
> >
> > Here is the talks that are scheduled for Monday's meetup. As you can see
> > that we have a full schedule, speakers please constrain your talk within
> > the given slot.
> >
> > 1. Lessons learned from enterprise users (Xuefu, 5m)
> > 2. New Hive metrics and monitoring for HS2/HMS (TBD, 10m)
> > 3. Hardening Hive - Design/Code Reviews (John Pullokkaran, 15m)
> > 4. Correctness issue reporting and tracking guideline (Carl, 5m)
> > 5. Dynamic dependency resolution and classloader issues (Anthony, 10m)
> > 6. A query syntax validation tool for Hive. (Szehon, 15m)
> > 7. Improved integration test for Hive (HIVE-12316) (Alan Gates, 30m)
> > 8. Uber's plan for Hive and our expected feature from the overall dev
> > community (Mohammad Islam,  15m)
> > 9. Update on LLAP (Gopal V Vijayaraghavan, 10m)
> >
> > See you all there!
> >
> > --Xuefu
> >
> > On Fri, Nov 6, 2015 at 2:11 PM, Xuefu Zhang  wrote:
> >
> >> Hi all,
> >>
> >> As an update, here is the link to the meetup:
> >> http://www.meetup.com/Hive-Contributors-Group/events/226495286/. Please
> >> RSVP as soon as possible because the organizers do need a headcount
> before
> >> the event.
> >>
> >> Also, we are finalizing a list of talks. If you plan to give a talk,
> >> please let the organizer know.
> >>
> >> Thanks,
> >> Xuefu
> >>
> >>
> >> On Mon, Nov 2, 2015 at 10:09 AM, Xuefu Zhang 
> wrote:
> >>
> >>> Hi all,
> >>>
> >>> Apache Hive community is considering a developer meetup in the middle
> of
> >>> this month. While we are finalizing the venue, I'd like to invite you
> to
> >>> plan to join, and even better, to give talk in the event.
> >>>
> >>> We consider the center of the meetup would be around the quality aspect
> >>> of the Project, and like to establish a collaborative effort in
> improving
> >>> Hive's quality. In the meantime, any topic will be considered.
> >>>
> >>> Please let me or Thejas know if you'd like to give a talk.
> >>>
> >>> Detailed venue and agenda will be given soon. Look forward to meeting
> you
> >>> in two weeks!
> >>>
> >>> Regards,
> >>>
> >>> Xuefu
> >>>
> >>
> >>
>

[jira] [Created] (HIVE-12462) DPP: DPP optimizers need to run on the TS operators not FIL

2015-11-18 Thread Gopal V (JIRA) 
Gopal V created HIVE-12462:
--

 Summary: DPP: DPP optimizers need to run on the TS operators not 
FIL 
 Key: HIVE-12462
 URL: https://issues.apache.org/jira/browse/HIVE-12462
 Project: Hive
  Issue Type: Bug
  Components: Tez
Reporter: Gopal V
Assignee: Gopal V
Priority: Critical


HIVE-11398 + HIVE-11791, the partition-condition-remover became more effective.

This removes predicates from the FilterExpression which involve partition 
columns, causing a miss for dynamic-partition pruning if the DPP relies on 
FilterDesc.

The TS desc will have the correct predicate in that condition.

{code}
$hdt$_0:$hdt$_1:a
  TableScan (TS_2)
alias: a
filterExpr: (((account_id = 22) and year(dt) is not null) and (year(dt)) IN 
(RS[6])) (type: boolean)
Filter Operator (FIL_20)
  predicate: ((account_id = 22) and year(dt) is not null) (type: boolean)
  Select Operator (SEL_4)
expressions: dt (type: date)
outputColumnNames: _col1
Reduce Output Operator (RS_8)
  key expressions: year(_col1) (type: int)
  sort order: +
  Map-reduce partition columns: year(_col1) (type: int)
  Join Operator (JOIN_9)
condition map:
 Inner Join 0 to 1
keys:
  0 year(_col1) (type: int)
  1 year(_col1) (type: int)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12457) Problems with SERDEPROPERTIES ('serialization.encoding'='ISO-8859-1')

2015-11-18 Thread hebert honorio maia de sousa (JIRA) 
hebert honorio maia de sousa created HIVE-12457:
---

 Summary: Problems with SERDEPROPERTIES 
('serialization.encoding'='ISO-8859-1')
 Key: HIVE-12457
 URL: https://issues.apache.org/jira/browse/HIVE-12457
 Project: Hive
  Issue Type: Bug
Reporter: hebert honorio maia de sousa


When I run this comand below, the accents work, but appears more records then 
original way:

Command:
ALTER TABLE tablename  SET SERDEPROPERTIES 
('serialization.encoding'='ISO-8859-1')

Result after command:
9884@TO@Monte do Carmo@77585000@0@M@@Mte 
Carmo@1713601ocantins@1713205712801040002@arães@

Result before command:
9884@TO@Monte do Carmo@77585000@0@M@@Mte Carmo@1713601



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)