Re: Review Request 62360: HIVE-16898: Validation of source file after distcp in repl load

2017-09-18 Thread anishek 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62360/#review185647
---




ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
Line 61 (original), 69 (patched)


use lambdas



ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
Lines 82 (patched)


Path not required on right hand side



ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
Lines 153 (patched)


use lambda


- anishek


On Sept. 19, 2017, 12:14 a.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62360/
> ---
> 
> (Updated Sept. 19, 2017, 12:14 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-16898
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
> 88d6a7a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 54746d3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java 28e7bcb 
> 
> 
> Diff: https://reviews.apache.org/r/62360/diff/3/
> 
> 
> Testing
> ---
> 
> Manually test it with debugger: setup a breakpoint right before copy, and 
> drop table in another session.
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>

Re: Review Request 62360: HIVE-16898: Validation of source file after distcp in repl load

2017-09-18 Thread anishek 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62360/#review185643
---




ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
Lines 51 (patched)


This can be private



ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
Lines 86 (patched)


we can start with

if(!srcFile.isUseSourcePath() && 
sourceFs.exists(srcFile.getEffectivePath())){continue;} 

as the first statement inside the for loop for srcFiles, that way it 
wouldnt have to try and get checksum etc for files in CM. Check to see 
existence of CM is to make sure that while doing the copy the cleaner thread 
did not clean the CM.

This will also make a few subsequent conditional checking irrelevant.



ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
Lines 120 (patched)


May be i am missing something but even when we are using the CM path in 
copyAndVerify towards the last, the file from CM itself might be deleted via 
the CM cleaner thread, in that case the doCopy will quietly fail and move on 
rather than failing replication.



ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
Lines 130 (patched)


copyAsUser and hiveConf are member variables, they dont need to be passed 
to functions.


- anishek


On Sept. 19, 2017, 12:14 a.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62360/
> ---
> 
> (Updated Sept. 19, 2017, 12:14 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-16898
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
> 88d6a7a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 54746d3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java 28e7bcb 
> 
> 
> Diff: https://reviews.apache.org/r/62360/diff/3/
> 
> 
> Testing
> ---
> 
> Manually test it with debugger: setup a breakpoint right before copy, and 
> drop table in another session.
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>

Re: New committer : Barna Zsombor Klara

2017-09-18 Thread Vaibhav Gumashta 
Congratulations Zsombor!

From: Prasanth Jayachandran 
Sent: Monday, September 18, 2017 5:52 PM
To: dev@hive.apache.org
Subject: Re: New committer : Barna Zsombor Klara

Congratulations Zsombor!!

Thanks
Prasanth
> On Sep 18, 2017, at 5:50 PM, Chetna C  wrote:
>
> Congratulations Zsombor!
>
> Thanks,
> Chetna Chaudhari
>
> On 19 September 2017 at 10:37, Sergio Pena  wrote:
>
>> Great, congratulations Zsombor !!!
>>
>> On Mon, Sep 18, 2017 at 12:44 PM, Sahil Takiar 
>> wrote:
>>
>>> Congrats Zsombor!
>>>
>>> On Mon, Sep 18, 2017 at 9:50 AM, Vihang Karajgaonkar <
>> vih...@cloudera.com>
>>> wrote:
>>>
 Congratulations Zsombor!

 On Mon, Sep 18, 2017 at 8:15 AM, Zoltan Haindrich <
 zhaindr...@hortonworks.com> wrote:

> Congratulations!!
>
> On 16 Sep 2017 00:52, Ashutosh Chauhan  wrote:
> The Project Management Committee (PMC) for Apache Hive has invited
>>> Barna
> Zsombor Klara to become a committer and we are pleased to announce
>> that
 he
> has accepted.
>
> Barna, thank you for your contributions, and we look forward your
> further interactions
> with the community!
>
> Welcome, Barna!
>
> Thanks,
> Ashutosh (on behalf of Apache Hive PMC)
>
>

>>>
>>>
>>>
>>> --
>>> Sahil Takiar
>>> Software Engineer at Cloudera
>>> takiar.sa...@gmail.com | (510) 673-0309
>>>
>>

Re: New committer : Barna Zsombor Klara

2017-09-18 Thread Prasanth Jayachandran 
Congratulations Zsombor!!

Thanks
Prasanth
> On Sep 18, 2017, at 5:50 PM, Chetna C  wrote:
> 
> Congratulations Zsombor!
> 
> Thanks,
> Chetna Chaudhari
> 
> On 19 September 2017 at 10:37, Sergio Pena  wrote:
> 
>> Great, congratulations Zsombor !!!
>> 
>> On Mon, Sep 18, 2017 at 12:44 PM, Sahil Takiar 
>> wrote:
>> 
>>> Congrats Zsombor!
>>> 
>>> On Mon, Sep 18, 2017 at 9:50 AM, Vihang Karajgaonkar <
>> vih...@cloudera.com>
>>> wrote:
>>> 
 Congratulations Zsombor!
 
 On Mon, Sep 18, 2017 at 8:15 AM, Zoltan Haindrich <
 zhaindr...@hortonworks.com> wrote:
 
> Congratulations!!
> 
> On 16 Sep 2017 00:52, Ashutosh Chauhan  wrote:
> The Project Management Committee (PMC) for Apache Hive has invited
>>> Barna
> Zsombor Klara to become a committer and we are pleased to announce
>> that
 he
> has accepted.
> 
> Barna, thank you for your contributions, and we look forward your
> further interactions
> with the community!
> 
> Welcome, Barna!
> 
> Thanks,
> Ashutosh (on behalf of Apache Hive PMC)
> 
> 
 
>>> 
>>> 
>>> 
>>> --
>>> Sahil Takiar
>>> Software Engineer at Cloudera
>>> takiar.sa...@gmail.com | (510) 673-0309
>>> 
>>

Re: New committer : Barna Zsombor Klara

2017-09-18 Thread Chetna C 
Congratulations Zsombor!

Thanks,
Chetna Chaudhari

On 19 September 2017 at 10:37, Sergio Pena  wrote:

> Great, congratulations Zsombor !!!
>
> On Mon, Sep 18, 2017 at 12:44 PM, Sahil Takiar 
> wrote:
>
> > Congrats Zsombor!
> >
> > On Mon, Sep 18, 2017 at 9:50 AM, Vihang Karajgaonkar <
> vih...@cloudera.com>
> > wrote:
> >
> > > Congratulations Zsombor!
> > >
> > > On Mon, Sep 18, 2017 at 8:15 AM, Zoltan Haindrich <
> > > zhaindr...@hortonworks.com> wrote:
> > >
> > > > Congratulations!!
> > > >
> > > > On 16 Sep 2017 00:52, Ashutosh Chauhan  wrote:
> > > > The Project Management Committee (PMC) for Apache Hive has invited
> > Barna
> > > > Zsombor Klara to become a committer and we are pleased to announce
> that
> > > he
> > > > has accepted.
> > > >
> > > > Barna, thank you for your contributions, and we look forward your
> > > > further interactions
> > > > with the community!
> > > >
> > > > Welcome, Barna!
> > > >
> > > > Thanks,
> > > > Ashutosh (on behalf of Apache Hive PMC)
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Sahil Takiar
> > Software Engineer at Cloudera
> > takiar.sa...@gmail.com | (510) 673-0309
> >
>

Re: Review Request 62373: [HIVE-17483] HS2 kill command to kill queries using query id

2017-09-18 Thread Thejas Nair 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62373/#review185634
---




itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java
Lines 1425 (patched)


remove ?



itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java
Lines 1439 (patched)


parallel statement execution within same connection can have issues. Better 
to use 2nd connection for kill query



jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java
Lines 181 (patched)


Can we re-use methods used by regular HiveConnection constructor code path 
to get the list of URLS ?

The duplication of code to read from ZK again here can be a maintainence 
problem.



service/src/java/org/apache/hive/service/cli/operation/OperationManager.java
Lines 386 (patched)


maintaining a hashmap of queryid to operation will help with performance, 
and also the case of killing from same connection.


- Thejas Nair


On Sept. 18, 2017, 4:57 a.m., Teddy Choi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62373/
> ---
> 
> (Updated Sept. 18, 2017, 4:57 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-17483
> https://issues.apache.org/jira/browse/HIVE-17483
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> For administrators, it is important to be able to kill queries if required. 
> Currently, there is no clean way to do it.
> It would help to have a "kill query " command that can be run using 
> odbc/jdbc against a HiveServer2 instance, to kill a query with that queryid 
> running in that instance.
> Authorization will have to be done to ensure that the user that is invoking 
> the API is allowed to perform this action.
> In case of SQL std authorization, this would require admin role.
> 
> 
> Diffs
> -
> 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
> 1108934df2 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 4a9af80fdc 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestServiceDiscoveryWithMiniHS2.java
>  e8051e40f2 
>   itests/src/test/resources/testconfiguration.properties d472bb3f9e 
>   jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 1311d2d88c 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java c6bd41feb7 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4e7c80f184 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryState.java fa7c32386b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 8aa2d90b76 
>   ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java 131c1e1bb5 
>   ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java da8c1e2305 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 251decac9b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g b5792ac485 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 429e0d995a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 003e09fd13 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
> 553dd64b5f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 2b9e897a54 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java e1f1f53c3c 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/KillQueryDesc.java PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/AuthorizationUtils.java
>  04e5565506 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveOperationType.java
>  3af97ea02f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HivePrivilegeObject.java
>  41983f1b4c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java
>  da99972e0c 
>   ql/src/java/org/apache/hadoop/hive/ql/session/KillQuery.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/session/NullKillQuery.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 8b64407d53 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestSQL11ReservedKeyWordsNegative.java
>  1a89eb1263 
>   ql/src/test/queries/clientnegative/authorization_kill_query.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/kill_query.q PRE-CREATION 
>   ql/src/test/results/clientnegative/authorization_kill_query.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/kill_query.q.out PRE-CREATION 
>   service-rpc/if/TCLIService.thrift 976ca9b6b3 
>

Re: Review Request 62392: HIVE-17535: Support CBO for queries without source table

2017-09-18 Thread Vineet Garg 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62392/
---

(Updated Sept. 19, 2017, 12:27 a.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Addressed review comments


Bugs: HIVE-17535
https://issues.apache.org/jira/browse/HIVE-17535


Repository: hive-git


Description
---

This patch provide CBO support for queries with no source e.g. select 1. This 
patch updates the code to bypass metadata calls for dummy table, which is used 
for queries with no source.


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties d472bb3f9e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 2645fab4c8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java db29e3aa5d 
  ql/src/test/queries/clientnegative/subquery_missing_from.q 3b49ac6a0a 
  ql/src/test/queries/clientnegative/subquery_select_no_source.q 75cae51e6a 
  ql/src/test/results/clientnegative/subquery_missing_from.q.out b09a8e311f 
  ql/src/test/results/clientnegative/subquery_select_no_source.q.out 37c4e57813 
  ql/src/test/results/clientpositive/beeline/mapjoin2.q.out 08e398ad50 
  ql/src/test/results/clientpositive/beeline/select_dummy_source.q.out 
b6fa03f808 
  ql/src/test/results/clientpositive/decimal_precision2.q.out 384ba80a58 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 03ebe375b8 
  ql/src/test/results/clientpositive/llap/mapjoin2.q.out cdee70b065 
  ql/src/test/results/clientpositive/llap/select_dummy_source.q.out 7efd2ff685 
  ql/src/test/results/clientpositive/mapjoin2.q.out 08e398ad50 
  ql/src/test/results/clientpositive/select_dummy_source.q.out b6fa03f808 
  ql/src/test/results/clientpositive/timestamptz_1.q.out 475603357c 
  ql/src/test/results/clientpositive/udtf_stack.q.out 97dcd3a1cd 
  ql/src/test/results/clientpositive/vector_tablesample_rows.q.out 283762a3eb 


Diff: https://reviews.apache.org/r/62392/diff/2/

Changes: https://reviews.apache.org/r/62392/diff/1-2/


Testing
---

Updated existing tests


Thanks,

Vineet Garg

[jira] [Created] (HIVE-17553) CBO wrongly type cast decimal literal to int

2017-09-18 Thread Vineet Garg (JIRA) 
Vineet Garg created HIVE-17553:
--

 Summary: CBO wrongly type cast decimal literal to int
 Key: HIVE-17553
 URL: https://issues.apache.org/jira/browse/HIVE-17553
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Reporter: Vineet Garg


{code:sql}explain select 100.000BD from f{code}
{noformat}
STAGE PLANS:
  Stage: Stage-0
Fetch Operator
  limit: -1
  Processor Tree:
TableScan
  alias: f
  Select Operator
expressions: 100 (type: int)
outputColumnNames: _col0
ListSink
{noformat}

Notice that the expression 100.000BD is of type int instead of decimal.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62360: HIVE-16898: Validation of source file after distcp in repl load

2017-09-18 Thread Daniel Dai 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62360/
---

(Updated Sept. 19, 2017, 12:14 a.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-16898


Diffs (updated)
-

  metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
88d6a7a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 54746d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java 28e7bcb 


Diff: https://reviews.apache.org/r/62360/diff/3/

Changes: https://reviews.apache.org/r/62360/diff/2-3/


Testing
---

Manually test it with debugger: setup a breakpoint right before copy, and drop 
table in another session.


Thanks,

Daniel Dai

Re: Review Request 62360: HIVE-16898: Validation of source file after distcp in repl load

2017-09-18 Thread Daniel Dai 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62360/
---

(Updated Sept. 18, 2017, 10:55 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-16898


Diffs (updated)
-

  metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
88d6a7a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 54746d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java 28e7bcb 


Diff: https://reviews.apache.org/r/62360/diff/2/

Changes: https://reviews.apache.org/r/62360/diff/1-2/


Testing
---

Manually test it with debugger: setup a breakpoint right before copy, and drop 
table in another session.


Thanks,

Daniel Dai

Re: Review Request 62360: HIVE-16898: Validation of source file after distcp in repl load

2017-09-18 Thread Daniel Dai 


> On Sept. 18, 2017, 4:49 a.m., anishek wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
> > Lines 73 (patched)
> > 
> >
> > Evaluation of doing a regularCopy or distCp can be done in the inner 
> > most function call, this will reduce passing in another variable from the 
> > top which can be evaluated later

I need to cache useRegularCopy and pass it to multiple doCopyRetry, that's why 
I put in the outer function.


> On Sept. 18, 2017, 4:49 a.m., anishek wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
> > Lines 92 (patched)
> > 
> >
> > I think eventually we have to move to a model of doing the checksum on 
> > sourceFS vs destinationFS as you have done here, though certain FS 
> > configurations change the value of checksum and unless we can guarantee 
> > that we calculate the checksum on the data by reading the data this might 
> > lead to more failures,
> > 
> > I thought the idea for now was that,
> > 
> > 1>> we get the checksum of the file on sourceFS before copy
> > 2>> we do the copy
> > 3>> we get the checksum on the file on sourceFS again 
> > 4>> we compare the checksum in 1 and 3 and if its not changed then 
> > during our copy the value wouldnt have either. 
> > 
> > until we can figure out the acutal solution to this, the fall back of 
> > doing the check on sourceFS might be the way to go.

Yes, that's right. The checksum of the file is in _files.


> On Sept. 18, 2017, 4:49 a.m., anishek wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
> > Lines 116 (patched)
> > 
> >
> > As a part of doing copy if the copy fails due to fileNotFoundException 
> > for a file location to actual location on hdfs then we should retry with 
> > the corresponding CMRoot Path for this file since it was moved while we 
> > were in the porcess of doing the copy.
> > 
> > Also if this happnes for a CM root file then there is an issue in our 
> > configuration such that the CM root FS is cleaned before the copy is done 
> > and we should log this as an error as the cleaner thread for CMroot is not 
> > configured for the right time. i did rather fail repl load, instead of just 
> > logging the error else we might not know how many such instances might 
> > happen before we realize that replication is broken.

Retry with CM path is part of copyAndVerify. doCopyRetry is shared between 
regular import and repl load, it does not deal with CM logic.


- Daniel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62360/#review185534
---


On Sept. 15, 2017, 6:10 p.m., Daniel Dai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62360/
> ---
> 
> (Updated Sept. 15, 2017, 6:10 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See HIVE-16898
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 
> 88d6a7a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 54746d3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java 28e7bcb 
> 
> 
> Diff: https://reviews.apache.org/r/62360/diff/1/
> 
> 
> Testing
> ---
> 
> Manually test it with debugger: setup a breakpoint right before copy, and 
> drop table in another session.
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>

Re: New committer : Barna Zsombor Klara

2017-09-18 Thread Sergio Pena 
Great, congratulations Zsombor !!!

On Mon, Sep 18, 2017 at 12:44 PM, Sahil Takiar 
wrote:

> Congrats Zsombor!
>
> On Mon, Sep 18, 2017 at 9:50 AM, Vihang Karajgaonkar 
> wrote:
>
> > Congratulations Zsombor!
> >
> > On Mon, Sep 18, 2017 at 8:15 AM, Zoltan Haindrich <
> > zhaindr...@hortonworks.com> wrote:
> >
> > > Congratulations!!
> > >
> > > On 16 Sep 2017 00:52, Ashutosh Chauhan  wrote:
> > > The Project Management Committee (PMC) for Apache Hive has invited
> Barna
> > > Zsombor Klara to become a committer and we are pleased to announce that
> > he
> > > has accepted.
> > >
> > > Barna, thank you for your contributions, and we look forward your
> > > further interactions
> > > with the community!
> > >
> > > Welcome, Barna!
> > >
> > > Thanks,
> > > Ashutosh (on behalf of Apache Hive PMC)
> > >
> > >
> >
>
>
>
> --
> Sahil Takiar
> Software Engineer at Cloudera
> takiar.sa...@gmail.com | (510) 673-0309
>

Re: Review Request 62392: HIVE-17535: Support CBO for queries without source table

2017-09-18 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62392/#review185625
---




ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Lines 4166-4178 (original), 4166-4178 (patched)


May want to get rid of this comment now.



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Line 10727 (original), 10733 (patched)


better to use protected.



ql/src/test/results/clientpositive/decimal_precision2.q.out
Line 40 (original), 40 (patched)


This doesn't look correct. query had BD qualifier so this should be decimal.


- Ashutosh Chauhan


On Sept. 18, 2017, 9:40 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62392/
> ---
> 
> (Updated Sept. 18, 2017, 9:40 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-17535
> https://issues.apache.org/jira/browse/HIVE-17535
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch provide CBO support for queries with no source e.g. select 1. This 
> patch updates the code to bypass metadata calls for dummy table, which is 
> used for queries with no source.
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties d472bb3f9e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 2645fab4c8 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> db29e3aa5d 
>   ql/src/test/queries/clientnegative/subquery_missing_from.q 3b49ac6a0a 
>   ql/src/test/queries/clientnegative/subquery_select_no_source.q 75cae51e6a 
>   ql/src/test/results/clientnegative/subquery_missing_from.q.out b09a8e311f 
>   ql/src/test/results/clientnegative/subquery_select_no_source.q.out 
> 37c4e57813 
>   ql/src/test/results/clientpositive/beeline/mapjoin2.q.out 08e398ad50 
>   ql/src/test/results/clientpositive/beeline/select_dummy_source.q.out 
> b6fa03f808 
>   ql/src/test/results/clientpositive/decimal_precision2.q.out 384ba80a58 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 03ebe375b8 
>   ql/src/test/results/clientpositive/llap/mapjoin2.q.out cdee70b065 
>   ql/src/test/results/clientpositive/llap/select_dummy_source.q.out 
> 7efd2ff685 
>   ql/src/test/results/clientpositive/mapjoin2.q.out 08e398ad50 
>   ql/src/test/results/clientpositive/select_dummy_source.q.out b6fa03f808 
>   ql/src/test/results/clientpositive/timestamptz_1.q.out 475603357c 
>   ql/src/test/results/clientpositive/udtf_stack.q.out 97dcd3a1cd 
>   ql/src/test/results/clientpositive/vector_tablesample_rows.q.out 283762a3eb 
> 
> 
> Diff: https://reviews.apache.org/r/62392/diff/1/
> 
> 
> Testing
> ---
> 
> Updated existing tests
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>

Review Request 62392: HIVE-17535: Support CBO for queries without source table

2017-09-18 Thread Vineet Garg 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62392/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-17535
https://issues.apache.org/jira/browse/HIVE-17535


Repository: hive-git


Description
---

This patch provide CBO support for queries with no source e.g. select 1. This 
patch updates the code to bypass metadata calls for dummy table, which is used 
for queries with no source.


Diffs
-

  itests/src/test/resources/testconfiguration.properties d472bb3f9e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 2645fab4c8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java db29e3aa5d 
  ql/src/test/queries/clientnegative/subquery_missing_from.q 3b49ac6a0a 
  ql/src/test/queries/clientnegative/subquery_select_no_source.q 75cae51e6a 
  ql/src/test/results/clientnegative/subquery_missing_from.q.out b09a8e311f 
  ql/src/test/results/clientnegative/subquery_select_no_source.q.out 37c4e57813 
  ql/src/test/results/clientpositive/beeline/mapjoin2.q.out 08e398ad50 
  ql/src/test/results/clientpositive/beeline/select_dummy_source.q.out 
b6fa03f808 
  ql/src/test/results/clientpositive/decimal_precision2.q.out 384ba80a58 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 03ebe375b8 
  ql/src/test/results/clientpositive/llap/mapjoin2.q.out cdee70b065 
  ql/src/test/results/clientpositive/llap/select_dummy_source.q.out 7efd2ff685 
  ql/src/test/results/clientpositive/mapjoin2.q.out 08e398ad50 
  ql/src/test/results/clientpositive/select_dummy_source.q.out b6fa03f808 
  ql/src/test/results/clientpositive/timestamptz_1.q.out 475603357c 
  ql/src/test/results/clientpositive/udtf_stack.q.out 97dcd3a1cd 
  ql/src/test/results/clientpositive/vector_tablesample_rows.q.out 283762a3eb 


Diff: https://reviews.apache.org/r/62392/diff/1/


Testing
---

Updated existing tests


Thanks,

Vineet Garg

Re: New committer : Barna Zsombor Klara

2017-09-18 Thread Sahil Takiar 
Congrats Zsombor!

On Mon, Sep 18, 2017 at 9:50 AM, Vihang Karajgaonkar 
wrote:

> Congratulations Zsombor!
>
> On Mon, Sep 18, 2017 at 8:15 AM, Zoltan Haindrich <
> zhaindr...@hortonworks.com> wrote:
>
> > Congratulations!!
> >
> > On 16 Sep 2017 00:52, Ashutosh Chauhan  wrote:
> > The Project Management Committee (PMC) for Apache Hive has invited Barna
> > Zsombor Klara to become a committer and we are pleased to announce that
> he
> > has accepted.
> >
> > Barna, thank you for your contributions, and we look forward your
> > further interactions
> > with the community!
> >
> > Welcome, Barna!
> >
> > Thanks,
> > Ashutosh (on behalf of Apache Hive PMC)
> >
> >
>



-- 
Sahil Takiar
Software Engineer at Cloudera
takiar.sa...@gmail.com | (510) 673-0309

[jira] [Created] (HIVE-17552) Enable bucket map join by default

2017-09-18 Thread Deepak Jaiswal (JIRA) 
Deepak Jaiswal created HIVE-17552:
-

 Summary: Enable bucket map join by default
 Key: HIVE-17552
 URL: https://issues.apache.org/jira/browse/HIVE-17552
 Project: Hive
  Issue Type: Bug
Reporter: Deepak Jaiswal
Assignee: Deepak Jaiswal


Currently bucket map join is disabled by default, however, it is potentially 
most optimal join we have. Need to enable it by default.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17551) HiveReduceExpressionsWithStatsRule should handle when stats are not available

2017-09-18 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-17551:
---

 Summary: HiveReduceExpressionsWithStatsRule should handle when 
stats are not available
 Key: HIVE-17551
 URL: https://issues.apache.org/jira/browse/HIVE-17551
 Project: Hive
  Issue Type: Bug
Reporter: Zoltan Haindrich


it seems to me that it might be possible that in case stats  are not available; 
then instead of those null checks; the array will be empty - and a AIOBE will 
be raised...

https://github.com/apache/hive/blob/a51ae9c8f5700638d0cc8c1d7f085ebf061f996f/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsWithStatsRule.java#L267





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: New committer : Barna Zsombor Klara

2017-09-18 Thread Vihang Karajgaonkar 
Congratulations Zsombor!

On Mon, Sep 18, 2017 at 8:15 AM, Zoltan Haindrich <
zhaindr...@hortonworks.com> wrote:

> Congratulations!!
>
> On 16 Sep 2017 00:52, Ashutosh Chauhan  wrote:
> The Project Management Committee (PMC) for Apache Hive has invited Barna
> Zsombor Klara to become a committer and we are pleased to announce that he
> has accepted.
>
> Barna, thank you for your contributions, and we look forward your
> further interactions
> with the community!
>
> Welcome, Barna!
>
> Thanks,
> Ashutosh (on behalf of Apache Hive PMC)
>
>

Re: New committer : Barna Zsombor Klara

2017-09-18 Thread Zoltan Haindrich 
Congratulations!!

On 16 Sep 2017 00:52, Ashutosh Chauhan  wrote:
The Project Management Committee (PMC) for Apache Hive has invited Barna
Zsombor Klara to become a committer and we are pleased to announce that he
has accepted.

Barna, thank you for your contributions, and we look forward your
further interactions
with the community!

Welcome, Barna!

Thanks,
Ashutosh (on behalf of Apache Hive PMC)

[jira] [Created] (HIVE-17550) Remove unreferenced q.out-s

2017-09-18 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-17550:
---

 Summary: Remove unreferenced q.out-s
 Key: HIVE-17550
 URL: https://issues.apache.org/jira/browse/HIVE-17550
 Project: Hive
  Issue Type: Improvement
  Components: Tests
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


it's sometimes a bit misleading to see q.out-s which are never even used..

I'll also add a small utility which is able to remove them - and add a test 
which will help to avoid them in the future



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17549) Use SHA-256 for RowContainer to improve security

2017-09-18 Thread Saijin Huang (JIRA) 
Saijin Huang created HIVE-17549:
---

 Summary: Use SHA-256 for RowContainer to improve security
 Key: HIVE-17549
 URL: https://issues.apache.org/jira/browse/HIVE-17549
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Saijin Huang
Assignee: Saijin Huang


Use SHA-256 to replace md5 for RowContainer to improve security



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)