[jira] [Created] (HIVE-17181) HCatOutputFormat should expose complete output-schema (including partition-keys) for dynamic-partitioning MR jobs

2017-07-26 Thread Mithun Radhakrishnan (JIRA) 
Mithun Radhakrishnan created HIVE-17181:
---

 Summary: HCatOutputFormat should expose complete output-schema 
(including partition-keys) for dynamic-partitioning MR jobs
 Key: HIVE-17181
 URL: https://issues.apache.org/jira/browse/HIVE-17181
 Project: Hive
  Issue Type: Bug
  Components: HCatalog
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan


Map/Reduce jobs that use HCatalog APIs to write to Hive tables using Dynamic 
partitioning are expected to call the following API methods:
# {{HCatOutputFormat.setOutput()}} to indicate which table/partitions to write 
to. This call populates the {{OutputJobInfo}} with details fetched from the 
Metastore.
# {{HCatOutputFormat.setSchema()}} to indicate the output-schema for the data 
being written.

It is a common mistake to invoke {{HCatOUtputFormat.setSchema()}} as follows:
{code:java}
HCatOutputFormat.setSchema(conf, HCatOutputFormat.getTableSchema(conf));
{code}

Unfortunately, {{getTableSchema()}} returns only the record-schema, not the 
entire table's schema. We'll need a better API for use in M/R jobs to get the 
complete table-schema.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17180) LLAP: Task scheduler: Add support for scheduling tasks in 1:1 edges

2017-07-26 Thread Rajesh Balamohan (JIRA) 
Rajesh Balamohan created HIVE-17180:
---

 Summary: LLAP: Task scheduler: Add support for scheduling tasks in 
1:1 edges
 Key: HIVE-17180
 URL: https://issues.apache.org/jira/browse/HIVE-17180
 Project: Hive
  Issue Type: Improvement
  Components: llap
Reporter: Rajesh Balamohan


With HIVE-17037, hive on tez can create 1:1 edges (e.g q95 in tpcds). 

{{LlapTaskSchedulerService:allocateTask}} currently ignores containerId details 
during scheduling. It would be good to add support for scheduling tasks in 1:1 
edges.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-07-26 Thread Vineet Garg 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16811
https://issues.apache.org/jira/browse/HIVE-16811


Repository: hive-git


Description
---

This patch introduces estimation of statistics if stats doesn't already exist.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 05f6cc9592 
  itests/src/test/resources/testconfiguration.properties f66e19be3e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
1d49568c8c 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 3b9ab41bed 
  ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
156be41502 
  ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
  ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
  ql/src/test/results/clientpositive/filter_join_breaktask.q.out 8f9b6363f4 
  ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
  ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
  ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
f77c75c232 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
f04519f8c5 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
f15d536945 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
1ec5dc415c 
  ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
  ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 8bc280cedd 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
7e3fe4c862 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
278c55b3e4 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 649afb502a 
  ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out feb3bbcfc4 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 922f23a29e 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_7.q.out 
8691d0d34e 
  
ql/src/test/results/clientpositive/llap/columnStatsUpdateForStatsOptimizer_1.q.out
 6d941fd4f2 
  ql/src/test/results/clientpositive/llap/column_access_stats.q.out c56c818617 
  ql/src/test/results/clientpositive/llap/column_table_stats.q.out c7726fec30 
  ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 
3de40ce871 
  ql/src/test/results/clientpositive/llap/dynamic_partition_pruning_2.q.out 
e78429fc7f 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out 
1dc9ed5a8c 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization2.q.out 
90756c6776 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out 
788854aa04 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out b36fe2a83f 
  ql/src/test/results/clientpositive/llap/filter_join_breaktask.q.out 
c5cdb77afc 
  ql/src/test/results/clientpositive/llap/join_reordering_no_stats.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/llap_partitioned.q.out d35501e82b 
  ql/src/test/results/clientpositive/llap/mergejoin.q.out 9df5e42df2 
  ql/src/test/results/clientpositive/llap/metadata_only_queries.q.out 
594e7e7fd6 
  ql/src/test/results/clientpositive/llap/partition_shared_scan.q.out 
bd3d247d48 
  ql/src/test/results/clientpositive/llap/ppd_union_view.q.out 543d17630a 
  ql/src/test/results/clientpositive/llap/reduce_deduplicate.q.out d74e4c7c78 
  ql/src/test/results/clientpositive/llap/sample10.q.out a3da78754f 
  ql/src/test/results/clientpositive/llap/schema_evol_orc_nonvec_part.q.out 
0ebb378d08 
  
ql/src/test/results/clientpositive/llap/schema_evol_orc_nonvec_part_all_complex.q.out
 22c7745345 
  
ql/src/test/results/clientpositive/llap/schema_evol_orc_nonvec_part_all_primitive.q.out
 4e94322705 
  ql/src/test/results/clientpositive/llap/schema_evol_orc_vec_part.q.out 
c6fc7b5dc6 
  
ql/src/test/results/clientpositive/llap/schema_evol_orc_vec_part_all_complex.q.out
 f57481dcc1 
  
ql/src/test/results/clientpositive/llap/schema_evol_orc_vec_part_all_primitive.q.out
 3a5232ad2e 
  ql/src/test/results/clientpositive/llap/schema_evol_text_nonvec_part.q.out 
8daee2883f 
  
ql/src/test/results/clientpositive/llap/schema_evol_text_nonvec_part_all_complex.q.out
 12c7db4af8 
  
ql/src/test/results/clientpositive/llap/schema_evol_text_nonvec_part_all_primitive.q.out
 757ea3a6c0

Re: Review Request 61164: HIVE-17006 LLAP: Parquet caching

2017-07-26 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61164/#review181494
---




ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
Lines 17 (patched)


lots of spurious imports, will remove in the next iter


- Sergey Shelukhin


On July 26, 2017, 9:53 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61164/
> ---
> 
> (Updated July 26, 2017, 9:53 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java e8a3a7a49e 
>   itests/src/test/resources/testconfiguration.properties f66e19be3e 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 
> 42129b7511 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java
>  0cbc8f6f4c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> 35b9d1f942 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java 9b8b76102a 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 21394c6aab 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/LlapCacheOnlyInputFormatInterface.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java
>  f4fadbb61b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java
>  322178a2f7 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetFooterInputFromCache.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
>  6a7a219dfe 
>   ql/src/test/queries/clientpositive/parquet_types_vectorization.q bb0e5b258f 
>   ql/src/test/results/clientpositive/llap/parquet_types_vectorization.q.out 
> PRE-CREATION 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 
> PRE-CREATION 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/MemoryBufferOrBuffers.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/61164/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Review Request 61164: HIVE-17006 LLAP: Parquet caching

2017-07-26 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61164/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java e8a3a7a49e 
  itests/src/test/resources/testconfiguration.properties f66e19be3e 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 
42129b7511 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java 
0cbc8f6f4c 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
35b9d1f942 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java 9b8b76102a 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 21394c6aab 
  
ql/src/java/org/apache/hadoop/hive/ql/io/LlapCacheOnlyInputFormatInterface.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java 
f4fadbb61b 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java
 322178a2f7 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetFooterInputFromCache.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
 6a7a219dfe 
  ql/src/test/queries/clientpositive/parquet_types_vectorization.q bb0e5b258f 
  ql/src/test/results/clientpositive/llap/parquet_types_vectorization.q.out 
PRE-CREATION 
  storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 
PRE-CREATION 
  
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/MemoryBufferOrBuffers.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/61164/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 61164: HIVE-17006 LLAP: Parquet caching

2017-07-26 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61164/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java e8a3a7a49e 
  itests/src/test/resources/testconfiguration.properties f66e19be3e 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 
42129b7511 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java 
0cbc8f6f4c 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
35b9d1f942 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java 9b8b76102a 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 21394c6aab 
  
ql/src/java/org/apache/hadoop/hive/ql/io/LlapCacheOnlyInputFormatInterface.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java 
f4fadbb61b 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java
 322178a2f7 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetFooterInputFromCache.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
 6a7a219dfe 
  ql/src/test/queries/clientpositive/parquet_types_vectorization.q bb0e5b258f 
  ql/src/test/results/clientpositive/llap/parquet_types_vectorization.q.out 
PRE-CREATION 
  storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 
PRE-CREATION 
  
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/MemoryBufferOrBuffers.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/61164/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-17179) Add InterfaceAudience and InterfaceStability annotations for Hook APIs

2017-07-26 Thread Sahil Takiar (JIRA) 
Sahil Takiar created HIVE-17179:
---

 Summary: Add InterfaceAudience and InterfaceStability annotations 
for Hook APIs
 Key: HIVE-17179
 URL: https://issues.apache.org/jira/browse/HIVE-17179
 Project: Hive
  Issue Type: Sub-task
  Components: Hooks
Reporter: Sahil Takiar
Assignee: Sahil Takiar






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17178) Spark Partition Pruning Sink Operator can't target multiple Works

2017-07-26 Thread Sahil Takiar (JIRA) 
Sahil Takiar created HIVE-17178:
---

 Summary: Spark Partition Pruning Sink Operator can't target 
multiple Works
 Key: HIVE-17178
 URL: https://issues.apache.org/jira/browse/HIVE-17178
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Reporter: Sahil Takiar
Assignee: Sahil Takiar


A Spark Partition Pruning Sink Operator cannot be used to target multiple Map 
Work objects. The entire DPP subtree (SEL-GBY-SPARKPRUNINGSINK) is duplicated 
if a single table needs to be used to target multiple Map Works.

The following query shows the issue:

{code}
set hive.spark.dynamic.partition.pruning=true;
set hive.auto.convert.join=true;

create table part_table_1 (col int) partitioned by (part_col int);
create table part_table_2 (col int) partitioned by (part_col int);
create table regular_table (col int);

insert into table regular_table values (1);

alter table part_table_1 add partition (part_col=1);
insert into table part_table_1 partition (part_col=1) values (1), (2), (3), (4);

alter table part_table_1 add partition (part_col=2);
insert into table part_table_1 partition (part_col=2) values (1), (2), (3), (4);

alter table part_table_2 add partition (part_col=1);
insert into table part_table_2 partition (part_col=1) values (1), (2), (3), (4);

alter table part_table_2 add partition (part_col=2);
insert into table part_table_2 partition (part_col=2) values (1), (2), (3), (4);

explain select * from regular_table, part_table_1, part_table_2 where 
regular_table.col = part_table_1.part_col and regular_table.col = 
part_table_2.part_col;
{code}

The explain plan is

{code}
STAGE DEPENDENCIES:
  Stage-2 is a root stage
  Stage-1 depends on stages: Stage-2
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-2
Spark
 A masked pattern was here 
  Vertices:
Map 1 
Map Operator Tree:
TableScan
  alias: regular_table
  Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE 
Column stats: NONE
  Filter Operator
predicate: col is not null (type: boolean)
Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE 
Column stats: NONE
Select Operator
  expressions: col (type: int)
  outputColumnNames: _col0
  Statistics: Num rows: 1 Data size: 1 Basic stats: 
COMPLETE Column stats: NONE
  Spark HashTable Sink Operator
keys:
  0 _col0 (type: int)
  1 _col1 (type: int)
  2 _col1 (type: int)
  Select Operator
expressions: _col0 (type: int)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 1 Basic stats: 
COMPLETE Column stats: NONE
Group By Operator
  keys: _col0 (type: int)
  mode: hash
  outputColumnNames: _col0
  Statistics: Num rows: 1 Data size: 1 Basic stats: 
COMPLETE Column stats: NONE
  Spark Partition Pruning Sink Operator
partition key expr: part_col
Statistics: Num rows: 1 Data size: 1 Basic stats: 
COMPLETE Column stats: NONE
target column name: part_col
target work: Map 2
  Select Operator
expressions: _col0 (type: int)
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 1 Basic stats: 
COMPLETE Column stats: NONE
Group By Operator
  keys: _col0 (type: int)
  mode: hash
  outputColumnNames: _col0
  Statistics: Num rows: 1 Data size: 1 Basic stats: 
COMPLETE Column stats: NONE
  Spark Partition Pruning Sink Operator
partition key expr: part_col
Statistics: Num rows: 1 Data size: 1 Basic stats: 
COMPLETE Column stats: NONE
target column name: part_col
target work: Map 3
Local Work:
  Map Reduce Local Work
Map 3 
Map Operator Tree:
TableScan
  alias: part_table_2
  Statistics: Num rows: 8 Data size: 8 Basic stats: COMPLETE 
Column stats: NONE
  Select Operator
expressions: col (type: int), part_col (type: int)
outputColumnNames: _col0, _col1
Statistics: Num rows: 8 Data

Re: Review Request 60006: HIVE-14746: Remove branch and repositories from profiles by sending them from ptest-client

2017-07-26 Thread Sergio Pena 


> On June 14, 2017, 4:03 p.m., Sergio Pena wrote:
> > testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java
> > Lines 108 (patched)
> > 
> >
> > Why is a workingDirectoryWrapper needed?
> > 
> > Currently, Ptest works on a 'working' directory for any branch detected 
> > on the profile. Why do we want to change that for different branches?
> 
> Barna Zsombor Klara wrote:
> I did this based on Siddharth Seth's comments on the Jira, where he 
> mentioned that associating the working dir with the branch could prevent 
> frequent rebasing when the checkout occurs. We don't need it for the original 
> intent of the Jira. We can discuss if we want it or if the rebase would not 
> be that problematic.

It's a good time savings. However, it's done only once. I don't know how much 
time we do save. Anyway, can we have this as a follow up jira?


- Sergio


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60006/#review177910
---


On June 13, 2017, 2:28 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60006/
> ---
> 
> (Updated June 13, 2017, 2:28 p.m.)
> 
> 
> Review request for hive, Sergio Pena and Siddharth Seth.
> 
> 
> Bugs: HIVE-14746
> https://issues.apache.org/jira/browse/HIVE-14746
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14746: Remove branch and repositories from profiles by sending them from 
> ptest-client
> 
> 
> Diffs
> -
> 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java
>  8e2604d372ac29b94445b269f08423b058308efe 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/request/TestStartRequest.java
>  8deed52ae0307d4fc075654a4d75e6cb09a5d9db 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
>  b2c61f03c5bf5f170894141848c89fc26129115a 
>   testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
> 1cdfdb309acd8282e593abd7ed10c87721926c60 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/CloudExecutionContextProvider.java
>  8b82497bdaf43694e0e1552e125b5ffdce40f56c 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/ExecutionContext.java
>  b09de1d4d930cf2d4d26b500f3457cea3fffa9ce 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/FixedExecutionContextProvider.java
>  f7b50d6a61962d2727b5181215be9de2e64b05b7 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/WorkingDirWrapper.java
>  PRE-CREATION 
>   
> testutils/ptest2/src/test/java/org/apache/hive/ptest/api/server/TestTestExecutor.java
>  a4a789b579305d9ed573d8c1fd0b6ce75787d50f 
>   
> testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestTestConfiguration.java
>  848faf27af1ed8945d7013b6562bab544605e4bc 
> 
> 
> Diff: https://reviews.apache.org/r/60006/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested the PTestClient with and without the branch argument.
> Updated and ran the unit tests.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

Re: Issue while creating Hive External table on S3 location

2017-07-26 Thread Sarath Kumar Sivan 
Hi Leela,



The Secret Access Key property name should be *fs.s3n.awsSecretAccessKey*
NOT *fs.s3.awsSecretAccessKey*
Access Key property name should be *fs.s3n.awsAccessKeyId* NOT
*fs.s3.awsAccessKeyId*



Please try this:



hive> set fs.s3n.awsSecretAccessKey=zvpZS8KxMBLK0w73w8yhzWaW0Ove10
Pk+fHeit/I;

hive> set fs.s3n.awsAccessKeyId=AKIAINIHCWDZMRWJEI3A;



Best Regards,

Sarath Kumar Sivan

Email: sarathkumarsi...@gmail.com

On Wed, Jul 26, 2017 at 2:38 AM, leela prasad Gorrepati <
leelaprasad.gorrep...@gmail.com> wrote:

> Hi All,
> I am unable to create Hive external table on s3 location. I have followed
> the steps mentioned in "
> https://cwiki.apache.org/confluence/display/Hive/HiveAws+HivingS3nRemotely
> "
>
> My Hive commands and received error is,
>
> hive> set hadoop.socks.server=localhost:2600;
> hive> set
> hadoop.rpc.socket.factory.class.default=org.apache.
> hadoop.net.SocksSocketFactory;
> hive> set hadoop.job.ugi=root,root;
> hive> set mapred.map.tasks=40;
> hive> set mapred.reduce.tasks=-1;
> hive> set fs.s3.awsSecretAccessKey=zvpZS8KxMBLK0w73w8yhzWaW0Ove10
> Pk+fHeit/I;
> hive> set fs.s3.awsAccessKeyId=AKIAINIHCWDZMRWJEI3A;
> hive> create external table empl (id int, name string, location string)
> location 's3://leela-2507-test/employee';
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.IllegalArgumentException: AWS Access Key
> ID
> and Secret Access Key must be specified by setting the fs.s3.awsAccessKeyId
> and fs.s3.awsSecretAccessKey properties (respectively).)
>
>
> Any one can try with the above valid access key, I have created for trail
> purpose and test data is in s3://leela-2507-test/employee
>
> Configuring in hive-site.xml also results in same error.
>
> Any suggestions are appreciated.
>
> Thanks in Advance.
>
> Regards,
> Leela Prasad
>

Issue while creating Hive External table on S3 location

2017-07-26 Thread leela prasad Gorrepati 
Hi All,
I am unable to create Hive external table on s3 location. I have followed
the steps mentioned in "
https://cwiki.apache.org/confluence/display/Hive/HiveAws+HivingS3nRemotely;

My Hive commands and received error is,

hive> set hadoop.socks.server=localhost:2600;
hive> set
hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.SocksSocketFactory;
hive> set hadoop.job.ugi=root,root;
hive> set mapred.map.tasks=40;
hive> set mapred.reduce.tasks=-1;
hive> set fs.s3.awsSecretAccessKey=zvpZS8KxMBLK0w73w8yhzWaW0Ove10Pk+fHeit/I;
hive> set fs.s3.awsAccessKeyId=AKIAINIHCWDZMRWJEI3A;
hive> create external table empl (id int, name string, location string)
location 's3://leela-2507-test/employee';
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID
and Secret Access Key must be specified by setting the fs.s3.awsAccessKeyId
and fs.s3.awsSecretAccessKey properties (respectively).)


Any one can try with the above valid access key, I have created for trail
purpose and test data is in s3://leela-2507-test/employee

Configuring in hive-site.xml also results in same error.

Any suggestions are appreciated.

Thanks in Advance.

Regards,
Leela Prasad

Re: trouble starting hiveserver2 with hive2.1.1

2017-07-26 Thread Amit Kumar 
Hi,

when running hive --service llap, on hive2.3, It throw below error

Failed: java.io.IOException: Target /tmp/staging-slider-hpJkzz/lib/tez is a
directory
java.util.concurrent.ExecutionException: java.io.IOException: Target
/tmp/staging-slider-hpJkzz/lib/tez is a directory
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.hadoop.hive.llap.cli.LlapServiceDriver.run(
LlapServiceDriver.java:556)
at org.apache.hadoop.hive.llap.cli.LlapServiceDriver.main(
LlapServiceDriver.java:116)
Caused by: java.io.IOException: Target /tmp/staging-slider-hpJkzz/lib/tez
is a directory
at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:500)
at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:502)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:348)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(
FileSystem.java:1965)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(
FileSystem.java:1933)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(
FileSystem.java:1898)
at org.apache.hadoop.hive.llap.cli.LlapServiceDriver$3.call(
LlapServiceDriver.java:450)
at org.apache.hadoop.hive.llap.cli.LlapServiceDriver$3.call(
LlapServiceDriver.java:404)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.
call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
INFO cli.LlapServiceDriver: LLAP service driver finished

Thanks and Regards,

Amit

Re: Review Request 60433: HIVE-16784: Missing lineage information when hive.blobstore.optimizations.enabled is true

2017-07-26 Thread Barna Zsombor Klara 


> On July 25, 2017, 9:29 p.m., Sahil Takiar wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java
> > Lines 57 (patched)
> > 
> >
> > Whats this change for?

This is needed to prevent an NPE in GenMapRedUtils#mergeMovePaths. If the 
Session is not initialized Session.get() will return null.


- Barna Zsombor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60433/#review181386
---


On July 17, 2017, 2:08 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60433/
> ---
> 
> (Updated July 17, 2017, 2:08 p.m.)
> 
> 
> Review request for hive, Marta Kuczora, Sergio Pena, and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16784: Missing lineage information when 
> hive.blobstore.optimizations.enabled is true
> 
> 
> Diffs
> -
> 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_blobstore_to_blobstore.q.out
>  46518999358be9ec5191ad2c3b37f555efbec263 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_empty_into_blobstore.q.out
>  8e5e09627b3a0f5c66a01fdd096b4b17b0b10689 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> 663a57274840a84a5f04777540d09d9f9bcfb5da 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  8052fd86eefaff1fe7d024d3f58f6896e7223dd9 
>   
> itests/hive-blobstore/src/test/results/clientpositive/orc_format_nonpart.q.out
>  f4705687ff4b7b29d9bd6b81dfcb4065364c2965 
>   itests/hive-blobstore/src/test/results/clientpositive/orc_format_part.q.out 
> 1ef98104c5a711ef4b6553ca06fccf9003b99361 
>   
> itests/hive-blobstore/src/test/results/clientpositive/orc_nonstd_partitions_loc.q.out
>  9de4190072cc779b34e556a0e774ce7403324525 
>   
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_nonpart.q.out
>  44a1f11fa2c7bd32c636b9bad4e4e35d79989dcf 
>   
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_part.q.out
>  defca3bb78d4d97a327c389650d256a2cd880899 
>   
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_nonstd_partitions_loc.q.out
>  5db9c7ef90c81524eae3eb6ec5346321e38b2a9e 
>   
> itests/hive-blobstore/src/test/results/clientpositive/zero_rows_blobstore.q.out
>  590f947add9b34d3adb05a7e22b0eb7d011c9a0d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 88bf8253346bc92dac17c07373e6360be232 
>   ql/src/java/org/apache/hadoop/hive/ql/session/LineageState.java 
> 0f9506376c283ec6e6ab659b20a5718807268969 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java
>  68ccda9ef61744e369638ceec5c083c77524c2db 
> 
> 
> Diff: https://reviews.apache.org/r/60433/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

Re: Review Request 60433: HIVE-16784: Missing lineage information when hive.blobstore.optimizations.enabled is true

2017-07-26 Thread Barna Zsombor Klara 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60433/#review181440
---




ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java
Lines 57 (patched)


This is needed to prevent an NPE in GenMapRedUtils#mergeMovePaths. If the 
Session is not initialized Session.get() will return null.


- Barna Zsombor Klara


On July 17, 2017, 2:08 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60433/
> ---
> 
> (Updated July 17, 2017, 2:08 p.m.)
> 
> 
> Review request for hive, Marta Kuczora, Sergio Pena, and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16784: Missing lineage information when 
> hive.blobstore.optimizations.enabled is true
> 
> 
> Diffs
> -
> 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_blobstore_to_blobstore.q.out
>  46518999358be9ec5191ad2c3b37f555efbec263 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_empty_into_blobstore.q.out
>  8e5e09627b3a0f5c66a01fdd096b4b17b0b10689 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> 663a57274840a84a5f04777540d09d9f9bcfb5da 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  8052fd86eefaff1fe7d024d3f58f6896e7223dd9 
>   
> itests/hive-blobstore/src/test/results/clientpositive/orc_format_nonpart.q.out
>  f4705687ff4b7b29d9bd6b81dfcb4065364c2965 
>   itests/hive-blobstore/src/test/results/clientpositive/orc_format_part.q.out 
> 1ef98104c5a711ef4b6553ca06fccf9003b99361 
>   
> itests/hive-blobstore/src/test/results/clientpositive/orc_nonstd_partitions_loc.q.out
>  9de4190072cc779b34e556a0e774ce7403324525 
>   
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_nonpart.q.out
>  44a1f11fa2c7bd32c636b9bad4e4e35d79989dcf 
>   
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_part.q.out
>  defca3bb78d4d97a327c389650d256a2cd880899 
>   
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_nonstd_partitions_loc.q.out
>  5db9c7ef90c81524eae3eb6ec5346321e38b2a9e 
>   
> itests/hive-blobstore/src/test/results/clientpositive/zero_rows_blobstore.q.out
>  590f947add9b34d3adb05a7e22b0eb7d011c9a0d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 88bf8253346bc92dac17c07373e6360be232 
>   ql/src/java/org/apache/hadoop/hive/ql/session/LineageState.java 
> 0f9506376c283ec6e6ab659b20a5718807268969 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java
>  68ccda9ef61744e369638ceec5c083c77524c2db 
> 
> 
> Diff: https://reviews.apache.org/r/60433/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

[jira] [Created] (HIVE-17177) move TestSuite.java to the right position

2017-07-26 Thread Saijin Huang (JIRA) 
Saijin Huang created HIVE-17177:
---

 Summary: move TestSuite.java to the right position
 Key: HIVE-17177
 URL: https://issues.apache.org/jira/browse/HIVE-17177
 Project: Hive
  Issue Type: Bug
Reporter: Saijin Huang
Assignee: Saijin Huang
Priority: Minor
 Fix For: 3.0.0


TestSuite.java is currently not belong to the package 
org.apache.hive.storage.jdbc.Move it to the right position.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17176) Add ASF header for LlapAllocatorBuffer.java

2017-07-26 Thread Saijin Huang (JIRA) 
Saijin Huang created HIVE-17176:
---

 Summary: Add ASF header for LlapAllocatorBuffer.java
 Key: HIVE-17176
 URL: https://issues.apache.org/jira/browse/HIVE-17176
 Project: Hive
  Issue Type: Bug
Reporter: Saijin Huang
Assignee: Saijin Huang
Priority: Minor
 Fix For: 3.0.0


Reproduce the problem from hive-16233,find the asf header missed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)