Open Source Hive using AWS Glue as Metastore

2017-12-11 Thread Goden Yao 
Accoring to EMR doc:
http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hive-metastore-glue.html


They support using AWS Glue as metastore.
I wonder if community has seen an demand to support it in OSS as well.
I found this JIRA:
HIVE-12679 

But the priority seems low and not too much traction from the community so
far.
Does that mean if I want to use Hive in AWS but not running EMR, I need to
develop it myself?
-- 
Goden

[jira] [Created] (HIVE-18263) Ptest execution are multiple times slower sometimes due to dying executor slaves

2017-12-11 Thread Adam Szita (JIRA) 
Adam Szita created HIVE-18263:
-

 Summary: Ptest execution are multiple times slower sometimes due 
to dying executor slaves
 Key: HIVE-18263
 URL: https://issues.apache.org/jira/browse/HIVE-18263
 Project: Hive
  Issue Type: Bug
  Components: Testing Infrastructure
Reporter: Adam Szita
Assignee: Adam Szita






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 64524: HIVE-18250

2017-12-11 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64524/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-18250
https://issues.apache.org/jira/browse/HIVE-18250


Repository: hive-git


Description
---

HIVE-18250


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
ba64f97105eaaa0a5d3f2ad1883db42608c95ae8 
  ql/src/test/queries/clientpositive/groupby_multialias.q PRE-CREATION 
  ql/src/test/results/clientpositive/groupby_multialias.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/64524/diff/1/


Testing
---


Thanks,

Jesús Camacho Rodríguez

Re: Review Request 64524: HIVE-18250

2017-12-11 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64524/
---

(Updated Dec. 12, 2017, 2:32 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-18250
https://issues.apache.org/jira/browse/HIVE-18250


Repository: hive-git


Description
---

HIVE-18250


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
ba64f97105eaaa0a5d3f2ad1883db42608c95ae8 
  ql/src/test/queries/clientpositive/groupby_multialias.q PRE-CREATION 
  ql/src/test/results/clientpositive/groupby_multialias.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/64524/diff/2/

Changes: https://reviews.apache.org/r/64524/diff/1-2/


Testing
---


Thanks,

Jesús Camacho Rodríguez

Re: Review Request 64324: HIVE-18153 refactor reopen and file management in TezTask

2017-12-11 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64324/
---

(Updated Dec. 12, 2017, 2:10 a.m.)


Review request for hive, Prasanth_J and Siddharth Seth.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 88a75edd35 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 6c1afa6555 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java 3bcf657ac4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
8417ebb7d5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolSession.java 
b3ccd24fd6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 
dd879fc5e8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 8795cfcee1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
dbdbbf25db 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezJobMonitor.java 
9726af1506 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
4148a8aa3a 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
52484540ff 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
829ea8cecc 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java 47aa936845 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
c58e4507f2 


Diff: https://reviews.apache.org/r/64324/diff/4/

Changes: https://reviews.apache.org/r/64324/diff/3-4/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 64510: HIVE-18068

2017-12-11 Thread Jesús Camacho Rodríguez 


> On Dec. 11, 2017, 10:27 p.m., Ashutosh Chauhan wrote:
> > accumulo-handler/src/test/results/positive/accumulo_queries.q.out
> > Line 292 (original), 292 (patched)
> > 
> >
> > Regression?

In fact it is an improvement _key > 100 and key < 120_. It gets the _key > 100_ 
from the other branch.


> On Dec. 11, 2017, 10:27 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/explainuser_1.q.out
> > Line 450 (original), 450 (patched)
> > 
> >
> > Is this predicate simplification correct?

The original condition is: ((col3 > 0) or (col1 >= 0)) and ((col1 >= 1) or 
(col4 >= 1))
>From below, we have that col1 i.e. c_int, is col1 > 0. Thus, _((col3 > 0) or 
>(col1 >= 0))_ folds to _true_. Hence, we keep only _((col1 >= 1) or (col4 >= 
>1))_.


> On Dec. 11, 2017, 10:27 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/masking_10.q.out
> > Line 122 (original)
> > 
> >
> > Seems like this predicate folded to false. But that seems incorrect.

_alias01.key = alias02.a_ where _a=2017_ (in join condition) _
and
((alias01.key % 2) = 0) and (alias01.key < 10)_ (coming from left input)
fold to
_false_

on the right input for the left outer join (observe that it does not propagate 
to left input, since it is a left outer join).


> On Dec. 11, 2017, 10:27 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/ppd_gby_join.q.out
> > Line 70 (original)
> > 
> >
> > Does this predicate folded to true. That doesn't look correct.

_key < '400' and (key < '50' or key > '50')_ -> _(key < '400' or key < '50') 
and (key < '400' or key > '50')_ -> _key < '400' and (key < '400' or key > 
'50')_ -> _key < '400'_ (note it is lexicographic comparison).


> On Dec. 11, 2017, 10:27 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/ppd_join5.q.out
> > Line 180 (original), 180 (patched)
> > 
> >
> > Does this predicate got simplified to false?

_id1 <= 1_ is specified below and the condition above is _id1 > 1_. Hence it 
folds to _false_.


> On Dec. 11, 2017, 10:27 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/ppd_join5.q.out
> > Line 194 (original), 194 (patched)
> > 
> >
> > Does this predicate got simplified to false?

_d <= 1_ is specified below and the condition above is _d > 1_. Hence it folds 
to _false_.


> On Dec. 11, 2017, 10:27 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/ppd_join5.q.out
> > Line 256 (original)
> > 
> >
> > Does this predicate got simplified to false?

_d <= 1_ is specified below and the condition above is _d > 1_. Hence it folds 
to _false_.


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64510/#review193459
---


On Dec. 11, 2017, 9:45 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64510/
> ---
> 
> (Updated Dec. 11, 2017, 9:45 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-18068
> https://issues.apache.org/jira/browse/HIVE-18068
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18068
> 
> 
> Diffs
> -
> 
>   accumulo-handler/src/test/results/positive/accumulo_queries.q.out 
> bd79eefb87b19ab6409238305a0256cf17af06cf 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> d6ec14e3759958d7e41dfa813fe239aaa860885d 
>   pom.xml a6b5663b7677c9d3d088a1d563e6c3b78199ba02 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
>  77a7b0f26bd21f5e36a2814a929513b3bdc6c9ce 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelDistribution.java
>  653f1c535c2789e337c95d26bc6d3b2d5657d5c0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateProjectMergeRule.java
>  b5f17754cb43bdc8fabb20ee77279bad50fd2061 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregatePullUpConstantsRule.java
>  370c0eca47b335aed9934dea5b50d519dd22ff57 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  9a5d8e2826f309ac274976f5ffbf374901c5b16b 
>

Review Request 64510: HIVE-18068

2017-12-11 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64510/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-18068
https://issues.apache.org/jira/browse/HIVE-18068


Repository: hive-git


Description
---

HIVE-18068


Diffs
-

  accumulo-handler/src/test/results/positive/accumulo_queries.q.out 
bd79eefb87b19ab6409238305a0256cf17af06cf 
  hbase-handler/src/test/results/positive/hbase_queries.q.out 
d6ec14e3759958d7e41dfa813fe239aaa860885d 
  pom.xml a6b5663b7677c9d3d088a1d563e6c3b78199ba02 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
 77a7b0f26bd21f5e36a2814a929513b3bdc6c9ce 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelDistribution.java
 653f1c535c2789e337c95d26bc6d3b2d5657d5c0 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateProjectMergeRule.java
 b5f17754cb43bdc8fabb20ee77279bad50fd2061 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregatePullUpConstantsRule.java
 370c0eca47b335aed9934dea5b50d519dd22ff57 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
 9a5d8e2826f309ac274976f5ffbf374901c5b16b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
ba64f97105eaaa0a5d3f2ad1883db42608c95ae8 
  ql/src/test/results/clientpositive/druid/druidmini_test1.q.out 
9577e72ee4a4eee4cbf62aba7689127ee7491e2a 
  ql/src/test/results/clientpositive/druid_basic2.q.out 
5a5f7541ecd7b131f424a94e1bcc9a572d880bff 
  ql/src/test/results/clientpositive/druid_basic3.q.out 
c174a5cc54429857dcba3af7f7027c7e94843200 
  ql/src/test/results/clientpositive/druid_intervals.q.out 
3cd28b513136c667e62f734741a1db65e43ff62d 
  ql/src/test/results/clientpositive/druid_timeseries.q.out 
330c068a02cafb9935a2e02a544702fedf8dece8 
  ql/src/test/results/clientpositive/druid_topn.q.out 
3e2b477dfc7e515a715882a8f9f7d51de1e0 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 
5adf401b25fcc1e548aed510d73177f9df62bc50 
  ql/src/test/results/clientpositive/masking_10.q.out 
e1a398af99120d2e17d5b126e8a78a28ed42d775 
  ql/src/test/results/clientpositive/perf/spark/query11.q.out 
17b2309379e3511d5712ad5608bedba18974d48b 
  ql/src/test/results/clientpositive/perf/spark/query4.q.out 
71154ff8a76b8a717d996708456f2d809511d7df 
  ql/src/test/results/clientpositive/perf/spark/query74.q.out 
497e792bc2c8608618aadffc8955c9eb4fdf2f5f 
  ql/src/test/results/clientpositive/perf/tez/query11.q.out 
69ced2611c7ff13e2247b0d38e4d109903de80a8 
  ql/src/test/results/clientpositive/perf/tez/query4.q.out 
81867b967551dea1269461d55407e3c18cd8e29e 
  ql/src/test/results/clientpositive/perf/tez/query74.q.out 
a75aaa129441c4a87b60e5db34bc4f6e7fd3010e 
  ql/src/test/results/clientpositive/ppd_gby_join.q.out 
75d0a6270da36c91409b7948b01f0af47bf1a48a 
  ql/src/test/results/clientpositive/ppd_join.q.out 
02aa5c2670056f5e26ef13705d702bd4857d6b65 
  ql/src/test/results/clientpositive/ppd_join5.q.out 
ca17699ca39bbfe0901e99c2e6d588d1f3eec4d3 
  ql/src/test/results/clientpositive/spark/ppd_gby_join.q.out 
1dc1a2594f4f6d805fcd282e1d4e94ea29228138 
  ql/src/test/results/clientpositive/spark/ppd_join.q.out 
dd2c394d48c398c9b3de35aa42877670eeb13daf 
  ql/src/test/results/clientpositive/spark/ppd_join5.q.out 
41817d8087e66a0912288c971cb2b7bce5875d0d 
  ql/src/test/results/clientpositive/spark/spark_explainuser_1.q.out 
6a4bea1bd4e4ae60c014469206de35ac56001508 


Diff: https://reviews.apache.org/r/64510/diff/1/


Testing
---


Thanks,

Jesús Camacho Rodríguez

Re: Review Request 64510: HIVE-18068

2017-12-11 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64510/#review193459
---




accumulo-handler/src/test/results/positive/accumulo_queries.q.out
Line 292 (original), 292 (patched)


Regression?



ql/src/test/results/clientpositive/llap/explainuser_1.q.out
Line 450 (original), 450 (patched)


Is this predicate simplification correct?



ql/src/test/results/clientpositive/masking_10.q.out
Line 122 (original)


Seems like this predicate folded to false. But that seems incorrect.



ql/src/test/results/clientpositive/ppd_gby_join.q.out
Line 70 (original)


Does this predicate folded to true. That doesn't look correct.



ql/src/test/results/clientpositive/ppd_join5.q.out
Line 180 (original), 180 (patched)


Does this predicate got simplified to false?



ql/src/test/results/clientpositive/ppd_join5.q.out
Line 194 (original), 194 (patched)


Does this predicate got simplified to false?



ql/src/test/results/clientpositive/ppd_join5.q.out
Line 256 (original)


Does this predicate got simplified to false?


- Ashutosh Chauhan


On Dec. 11, 2017, 9:45 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64510/
> ---
> 
> (Updated Dec. 11, 2017, 9:45 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-18068
> https://issues.apache.org/jira/browse/HIVE-18068
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18068
> 
> 
> Diffs
> -
> 
>   accumulo-handler/src/test/results/positive/accumulo_queries.q.out 
> bd79eefb87b19ab6409238305a0256cf17af06cf 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> d6ec14e3759958d7e41dfa813fe239aaa860885d 
>   pom.xml a6b5663b7677c9d3d088a1d563e6c3b78199ba02 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
>  77a7b0f26bd21f5e36a2814a929513b3bdc6c9ce 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelDistribution.java
>  653f1c535c2789e337c95d26bc6d3b2d5657d5c0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateProjectMergeRule.java
>  b5f17754cb43bdc8fabb20ee77279bad50fd2061 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregatePullUpConstantsRule.java
>  370c0eca47b335aed9934dea5b50d519dd22ff57 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  9a5d8e2826f309ac274976f5ffbf374901c5b16b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> ba64f97105eaaa0a5d3f2ad1883db42608c95ae8 
>   ql/src/test/results/clientpositive/druid/druidmini_test1.q.out 
> 9577e72ee4a4eee4cbf62aba7689127ee7491e2a 
>   ql/src/test/results/clientpositive/druid_basic2.q.out 
> 5a5f7541ecd7b131f424a94e1bcc9a572d880bff 
>   ql/src/test/results/clientpositive/druid_basic3.q.out 
> c174a5cc54429857dcba3af7f7027c7e94843200 
>   ql/src/test/results/clientpositive/druid_intervals.q.out 
> 3cd28b513136c667e62f734741a1db65e43ff62d 
>   ql/src/test/results/clientpositive/druid_timeseries.q.out 
> 330c068a02cafb9935a2e02a544702fedf8dece8 
>   ql/src/test/results/clientpositive/druid_topn.q.out 
> 3e2b477dfc7e515a715882a8f9f7d51de1e0 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 
> 5adf401b25fcc1e548aed510d73177f9df62bc50 
>   ql/src/test/results/clientpositive/masking_10.q.out 
> e1a398af99120d2e17d5b126e8a78a28ed42d775 
>   ql/src/test/results/clientpositive/perf/spark/query11.q.out 
> 17b2309379e3511d5712ad5608bedba18974d48b 
>   ql/src/test/results/clientpositive/perf/spark/query4.q.out 
> 71154ff8a76b8a717d996708456f2d809511d7df 
>   ql/src/test/results/clientpositive/perf/spark/query74.q.out 
> 497e792bc2c8608618aadffc8955c9eb4fdf2f5f 
>   ql/src/test/results/clientpositive/perf/tez/query11.q.out 
> 69ced2611c7ff13e2247b0d38e4d109903de80a8 
>   ql/src/test/results/clientpositive/perf/tez/query4.q.out 
> 81867b967551dea1269461d55407e3c18cd8e29e 
>   ql/src/test/results/clientpositive/perf/tez/query74.q.out 
> a75aaa129441c4a87b60e5db34bc4f6e7fd3010e 
>   ql/src/test/results/clientpositive/ppd_gby_join.q.out 
> 75d0a6270da36c91409b7948b01f0af47bf1a48a 
>   ql/src/test/results/clientpositive/ppd_join.q.out 
> 02aa5c2670056f5e26ef13705d702bd4857d6b65 
>   ql/src/test/results/clientpositive/ppd_join5.q.out 
>

Review Request 64511: HIVE-18252 Limit the size of the object inspector caches

2017-12-11 Thread Jason Dere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64511/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-18252
https://issues.apache.org/jira/browse/HIVE-18252


Repository: hive-git


Description
---

Patch to limit size of the object inspector cache for complex types, and to not 
cache complex object inspectors that contain constants.


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/LRUCache.java PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/TestLRUCache.java PRE-CREATION 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
 cb63d59 
  
serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorFactory.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/64511/diff/1/


Testing
---

Added Junit tests


Thanks,

Jason Dere

[jira] [Created] (HIVE-18264) CachedStore: Store cached partitions within the table cache

2017-12-11 Thread Vaibhav Gumashta (JIRA) 
Vaibhav Gumashta created HIVE-18264:
---

 Summary: CachedStore: Store cached partitions within the table 
cache  
 Key: HIVE-18264
 URL: https://issues.apache.org/jira/browse/HIVE-18264
 Project: Hive
  Issue Type: Bug
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta


Currently we have a separate cache for partitions and partition col stats which 
results in some calls iterating through each of these for retrieving/updating. 
We can get better performance by organizing hierarchically. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: apache hive builds

2017-12-11 Thread Adam Szita 
Hi all,

There are two things related to this:
1, We have recently applied a fix on the ptest component so we did a
restart this morning. We also re-triggered all pending jobs at that time.
More details: https://issues.apache.org/jira/browse/HIVE-18212
2, It seems like that for some time (even before Yetus integration was
added) ptest was suffering from very slow executions from time to time.
Today we looked in this as well in depth. I have found that some executor
slaves are getting shut down in the middle of the testing. I already have a
fix ready and will shortly upload it at:
https://issues.apache.org/jira/browse/HIVE-18263

Thanks,
Adam


On 11 December 2017 at 10:06, Anishek Agarwal  wrote:

> Hello
>
> The apache builds seem to be taking very long and most of them just being
> either manually shut or getting killed
> Builds after https://builds.apache.org/job/PreCommit-HIVE-Build/8160/
> seem
> to be showing the problem. Problems are surfacing after TestExecDriver is
> being run. Is there someone  here with better infrastructure understanding
> to help figure out what is happening.
>
> Thanks
> Anishek
>

Hive JDBC and invalid SSL certificates

2017-12-11 Thread Daisuke Goto 
Hello: I am a Hive JDBC user and I am new to Hive and Java.  I am currently
trying to use the JDBC driver to pull data from a test server that issues a
SSL certificate for *.azurehdinsight.net but I can only reach the server
through an internal ip address 10.*.*.*.  I figured that I need to modify
\jdbc\src\java\org\apache\hive\jdbc\HiveConnection.java to accept invalid
certificates.  I found that lines 465 - 506 are responsible for
establishing a SSL connection.  I am wondering if anyone could either
modify the code for me to accept an invalid certificate or give me some
pointers.  I'd never want to ask for something without making contributions
first but at this moment, I need your help.  I'd greatly appreciate your
input.

DG

// Configure http client for SSL
if (useSsl) {
  String useTwoWaySSL = sessConfMap.get(JdbcConnectionParams.USE_TWO_
WAY_SSL);
  String sslTrustStorePath = sessConfMap.get(JdbcConnectionParams.SSL_
TRUST_STORE);
  String sslTrustStorePassword = sessConfMap.get(
JdbcConnectionParams.SSL_TRUST_STORE_PASSWORD);
  KeyStore sslTrustStore;
  SSLConnectionSocketFactory socketFactory;
  SSLContext sslContext;
  /**
   * The code within the try block throws: SSLInitializationException,
KeyStoreException,
   * IOException, NoSuchAlgorithmException, CertificateException,
KeyManagementException &
   * UnrecoverableKeyException. We don't want the client to retry on
any of these,
   * hence we catch all and throw a SQLException.
   */
  try {
if (useTwoWaySSL != null &&
useTwoWaySSL.equalsIgnoreCase(JdbcConnectionParams.TRUE))
{
  socketFactory = getTwoWaySSLSocketFactory();
} else if (sslTrustStorePath == null ||
sslTrustStorePath.isEmpty()) {
  // Create a default socket factory based on standard JSSE trust
material
  socketFactory = SSLConnectionSocketFactory.getSocketFactory();
} else {
  // Pick trust store config from the given path
  sslTrustStore = KeyStore.getInstance(JdbcConnectionParams.SSL_
TRUST_STORE_TYPE);
  try (FileInputStream fis = new FileInputStream(sslTrustStorePath))
{
sslTrustStore.load(fis, sslTrustStorePassword.toCharArray());
  }
  sslContext = SSLContexts.custom().loadTrustMaterial(sslTrustStore,
null).build();
  socketFactory =
  new SSLConnectionSocketFactory(sslContext, new
DefaultHostnameVerifier(null));
}
final Registry registry =
RegistryBuilder.
create().register("https", socketFactory)
.build();
httpClientBuilder.setConnectionManager(new
BasicHttpClientConnectionManager(registry));
  } catch (Exception e) {
String msg =
"Could not create an https connection to " + jdbcUriString + ".
" + e.getMessage();
throw new SQLException(msg, " 08S01", e);
  }
}
return httpClientBuilder.build();

[jira] [Created] (HIVE-18262) ConcurrentModificationException in QueryPlan.extractCounters

2017-12-11 Thread Marta Kuczora (JIRA) 
Marta Kuczora created HIVE-18262:


 Summary: ConcurrentModificationException in 
QueryPlan.extractCounters
 Key: HIVE-18262
 URL: https://issues.apache.org/jira/browse/HIVE-18262
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 2.3.2
Reporter: Marta Kuczora


Intermittently experiencing an issue where the query fails with the following 
error:
{noformat}
2017-10-27 03:05:48,194 ERROR org.apache.hadoop.hive.ql.Driver: 
[HiveServer2-Handler-Pool: Thread-10925531]: FAILED: Hive Internal Error: 
java.util.ConcurrentModificationException(null)
java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
at java.util.ArrayList$Itr.next(ArrayList.java:851)
at org.apache.hadoop.hive.ql.QueryPlan.extractCounters(QueryPlan.java:381)
at org.apache.hadoop.hive.ql.QueryPlan.getQueryPlan(QueryPlan.java:436)
at 
org.apache.hadoop.hive.ql.exec.Utilities.setWorkflowAdjacencies(Utilities.java:471)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1950)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1684)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1421)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1205)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1200)
at 
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:187)
at 
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:217)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:337)
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:425)
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:396)
at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at 
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at 
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1912)
at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy19.executeStatement(Unknown Source)
at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:247)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:500)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

apache hive builds

2017-12-11 Thread Anishek Agarwal 
Hello

The apache builds seem to be taking very long and most of them just being
either manually shut or getting killed
Builds after https://builds.apache.org/job/PreCommit-HIVE-Build/8160/  seem
to be showing the problem. Problems are surfacing after TestExecDriver is
being run. Is there someone  here with better infrastructure understanding
to help figure out what is happening.

Thanks
Anishek