[jira] [Comment Edited] (HIVE-8045) SQL standard auth with cli - Errors and configuration issues

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14134579#comment-14134579
 ] 

Thejas M Nair edited comment on HIVE-8045 at 9/17/14 6:59 AM:
--

bq. 1 Even if hive.security.authorization.enabled=false, authorization checks 
were happening for non sql statements as add/detete/dfs/compile, which results 
in MetaStoreAuthzAPIAuthorizerEmbedOnly throwing an error.

CommandUtil. authorizeCommand(..) has been fixed to consider the value of 
hive.security.authorization.enabled . Added  new test case -
authorization_cli_nonsql.q

bq. 2 Create table from hive-cli ended up calling metastore server api call 
(getRoles) and resulted in MetaStoreAuthzAPIAuthorizerEmbedOnly throwing an 
error.

Removed the getRoles call from authorization from  SQLStdHiveAccessController 
constructor. It is already explicitly initialized before roles are used from 
other methods. Added test authorization_cli_createtab_noauthzapi.q to verify 
that no authorization api calls are being made for create table commands.

bq. 3 Some users prefer to enable authorization using hive-site.xml for 
hive-server2 (hive.security.authorization.enabled param). If this file is 
shared by hive-cli and hive-server2, SQL std authorizer throws an error because 
its use in hive-cli is not allowed.
Created SQLStdConfOnlyAuthorizerFactory that can be set in hive-cli, its 
authorization checkprivileges calls are no-op calls. The sql standard 
authorizer can be set for hiveserver2 using commandline or hiveserver2-site.xml 
. authorization_cli_stdconfigauth.q tests this.






was (Author: thejas):
bq. 1 Even if hive.security.authorization.enabled=false, authorization checks 
were happening for non sql statements as add/detete/dfs/compile, which results 
in MetaStoreAuthzAPIAuthorizerEmbedOnly throwing an error.

CommandUtil. authorizeCommand(..) has been fixed to consider the value of 
hive.security.authorization.enabled . Added  new test case -
authorization_cli_nonsql.q

bq. 2 Create table from hive-cli ended up calling metastore server api call 
(getRoles) and resulted in MetaStoreAuthzAPIAuthorizerEmbedOnly throwing an 
error.

Removed the getRoles call from authorization from  SQLStdHiveAccessController 
constructor. It is already explicitly initialized before roles are used from 
other methods. Added test authorization_cli_createtab_noauthzapi.q to verify 
that no authorization api calls are being made for create table commands.

bq. 3 Some users prefer to enable authorization using hive-site.xml for 
hive-server2 (hive.security.authorization.enabled param). If this file is 
shared by hive-cli and hive-server2, SQL std authorizer throws an error because 
its use in hive-cli is not allowed.
Created SQLStdConfOnlyAuthorizerFactory that can be set in hive-cli, its 
authorization checkprivileges calls are no-op calls. The sql standard 
authorizer can be set for hiveserver2 using commandline or hiveserver2-site.xml 
. 




> SQL standard auth with cli - Errors and configuration issues
> 
>
> Key: HIVE-8045
> URL: https://issues.apache.org/jira/browse/HIVE-8045
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization
>Reporter: Jagruti Varia
>Assignee: Thejas M Nair
>
> HIVE-7533 enabled sql std authorization to be set in hive cli (without 
> enabling authorization checks). This updates hive configuration so that 
> create-table and create-views set permissions appropriately for the owner of 
> the table.
> HIVE-7209 added a metastore authorization provider that can be used to 
> restricts calls made to the authorization api, so that only HS2 can make 
> those calls (when HS2 uses embedded metastore).
> Some issues were found with this.
> # Even if hive.security.authorization.enabled=false, authorization checks 
> were happening for non sql statements as add/detete/dfs/compile, which 
> results in MetaStoreAuthzAPIAuthorizerEmbedOnly throwing an error.
> # Create table from hive-cli ended up calling metastore server api call 
> (getRoles) and resulted in  MetaStoreAuthzAPIAuthorizerEmbedOnly throwing an 
> error.
> # Some users prefer to enable authorization using hive-site.xml for 
> hive-server2 (hive.security.authorization.enabled param). If this file is 
> shared by hive-cli and hive-server2,  SQL std authorizer throws an error 
> because is use in hive-cli is not allowed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7946) CBO: Merge CBO changes to Trunk

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-7946:
-
Status: Patch Available  (was: Open)

> CBO: Merge CBO changes to Trunk
> ---
>
> Key: HIVE-7946
> URL: https://issues.apache.org/jira/browse/HIVE-7946
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-7946.1.patch, HIVE-7946.10.patch, 
> HIVE-7946.2.patch, HIVE-7946.3.patch, HIVE-7946.4.patch, HIVE-7946.5.patch, 
> HIVE-7946.6.patch, HIVE-7946.7.patch, HIVE-7946.8.patch, HIVE-7946.9.patch, 
> HIVE-7946.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7946) CBO: Merge CBO changes to Trunk

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-7946:
-
Status: Open  (was: Patch Available)

> CBO: Merge CBO changes to Trunk
> ---
>
> Key: HIVE-7946
> URL: https://issues.apache.org/jira/browse/HIVE-7946
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-7946.1.patch, HIVE-7946.10.patch, 
> HIVE-7946.2.patch, HIVE-7946.3.patch, HIVE-7946.4.patch, HIVE-7946.5.patch, 
> HIVE-7946.6.patch, HIVE-7946.7.patch, HIVE-7946.8.patch, HIVE-7946.9.patch, 
> HIVE-7946.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7946) CBO: Merge CBO changes to Trunk

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-7946:
-
Attachment: HIVE-7946.10.patch

> CBO: Merge CBO changes to Trunk
> ---
>
> Key: HIVE-7946
> URL: https://issues.apache.org/jira/browse/HIVE-7946
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-7946.1.patch, HIVE-7946.10.patch, 
> HIVE-7946.2.patch, HIVE-7946.3.patch, HIVE-7946.4.patch, HIVE-7946.5.patch, 
> HIVE-7946.6.patch, HIVE-7946.7.patch, HIVE-7946.8.patch, HIVE-7946.9.patch, 
> HIVE-7946.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HIVE-8157) remove unnecessary conf file on cbo branch

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner resolved HIVE-8157.
--
Resolution: Fixed

committed to cbo branch.

> remove unnecessary conf file on cbo branch
> --
>
> Key: HIVE-8157
> URL: https://issues.apache.org/jira/browse/HIVE-8157
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
>
> brought in hive-default.xml.template with old merge.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8157) remove unnecessary conf file on cbo branch

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8157:
-
Description: brought in hive-default.xml.template with old merge.

> remove unnecessary conf file on cbo branch
> --
>
> Key: HIVE-8157
> URL: https://issues.apache.org/jira/browse/HIVE-8157
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
>
> brought in hive-default.xml.template with old merge.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8157) remove unnecessary conf file on cbo branch

[Gunther Hagleitner (JIRA)]

Gunther Hagleitner created HIVE-8157:


 Summary: remove unnecessary conf file on cbo branch
 Key: HIVE-8157
 URL: https://issues.apache.org/jira/browse/HIVE-8157
 Project: Hive
  Issue Type: Sub-task
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7812) Disable CombineHiveInputFormat when ACID format is used

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136812#comment-14136812
 ] 

Hive QA commented on HIVE-7812:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12669096/HIVE-7812.patch

{color:green}SUCCESS:{color} +1 6279 tests passed

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/834/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/834/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-834/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12669096

> Disable CombineHiveInputFormat when ACID format is used
> ---
>
> Key: HIVE-7812
> URL: https://issues.apache.org/jira/browse/HIVE-7812
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 0.14.0
>
> Attachments: HIVE-7812.patch, HIVE-7812.patch, HIVE-7812.patch, 
> HIVE-7812.patch
>
>
> Currently the HiveCombineInputFormat complains when called on an ACID 
> directory. Modify HiveCombineInputFormat so that HiveInputFormat is used 
> instead if the directory is ACID format.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8112) Change reporting string to reflect update in Tez

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8112:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Failure is unrelated. Committed to trunk. Thanks [~vikram.dixit]

> Change reporting string to reflect update in Tez
> 
>
> Key: HIVE-8112
> URL: https://issues.apache.org/jira/browse/HIVE-8112
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
>Priority: Minor
> Attachments: HIVE-8112.1.patch
>
>
> We're now printing:
> Status: Running (application id: Executing on YARN cluster with App id 
> application_1410822917153_0001)
> Where Tez just used to print the id it now has the full "Executing on 
> YARN..." string.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8112) Change reporting string to reflect update in Tez

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8112:
-
Fix Version/s: 0.14.0

> Change reporting string to reflect update in Tez
> 
>
> Key: HIVE-8112
> URL: https://issues.apache.org/jira/browse/HIVE-8112
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
>Priority: Minor
> Fix For: 0.14.0
>
> Attachments: HIVE-8112.1.patch
>
>
> We're now printing:
> Status: Running (application id: Executing on YARN cluster with App id 
> application_1410822917153_0001)
> Where Tez just used to print the id it now has the full "Executing on 
> YARN..." string.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8105) booleans and nulls not handled properly in insert/values

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136803#comment-14136803
 ] 

Eugene Koifman commented on HIVE-8105:
--

I'm wondering if it's not better to disallow NULL until we create the input 
format as you suggest.  As it is, this works for numeric types, but for 
boolean/string types this will take NULL from user input and insert an 
FALSE/empty string.  This doesn't seem right.
Is it possible to check the target type and throw if it's not numeric?

> booleans and nulls not handled properly in insert/values
> 
>
> Key: HIVE-8105
> URL: https://issues.apache.org/jira/browse/HIVE-8105
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Attachments: HIVE-8105.2.patch, HIVE-8105.patch
>
>
> Doing an insert/values with a boolean always results in a value of true, 
> regardless of whether true or false is given in the query.
> Doing an insert/values with a null for a column value results in a semantic 
> error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-6705) hive jdbc can not used by jmeter, because of unsupported auto commit feature

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136795#comment-14136795
 ] 

Thejas M Nair commented on HIVE-6705:
-

+1

> hive jdbc can not used by jmeter, because of unsupported auto commit feature
> 
>
> Key: HIVE-6705
> URL: https://issues.apache.org/jira/browse/HIVE-6705
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.12.0
> Environment: CentOS_X86_64 
> JMeter 2.11
>Reporter: Ben
>Assignee: Navis
> Attachments: HIVE-6705.1.patch.txt, HIVE-6705.2.patch.txt
>
>
> In apache jmeter ,the autocommit property is required.
> but in the hive jdbc the auto commit is unsupported method.
> in 
> /jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java
> {quote}
>  public void setAutoCommit(boolean autoCommit) throws SQLException {
> // TODO Auto-generated method stub
>   throw new {color:red}  SQLException("Method not supported");
> {color}
>   }
> {quote}
> so ,should  we make a mock to support  the auto commit property == false ?
> {quote}
> public void setAutoCommit(boolean autoCommit) throws SQLException {
>   // TODO Auto-generated method stub
>  {color:red}if(autoCommit) {color}
>   throw new SQLException("Method not supported");
>  else
>   return;
>  }
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7984) AccumuloOutputFormat Configuration items from StorageHandler not re-set in Configuration in Tez

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136787#comment-14136787
 ] 

Josh Elser commented on HIVE-7984:
--

Test failure appears unrelated to me. Can anyone give this a review? It's a 
rather straightforward change.

> AccumuloOutputFormat Configuration items from StorageHandler not re-set in 
> Configuration in Tez
> ---
>
> Key: HIVE-7984
> URL: https://issues.apache.org/jira/browse/HIVE-7984
> Project: Hive
>  Issue Type: Bug
>  Components: StorageHandler, Tez
>Reporter: Josh Elser
>Assignee: Josh Elser
> Fix For: 0.14.0
>
> Attachments: HIVE-7984-1.diff, HIVE-7984-1.patch, HIVE-7984.1.patch
>
>
> Ran AccumuloStorageHandler queries with Tez and found that configuration 
> elements that are pulled from the {{-hiveconf}} and passed to the 
> inputJobProperties or outputJobProperties by the AccumuloStorageHandler 
> aren't available inside of the Tez container.
> I'm guessing that there is a disconnect from the configuration that the 
> StorageHandler creates and what the Tez container sees.
> The HBaseStorageHandler likely doesn't run into this because it expects to 
> have hbase-site.xml available via tmpjars (and can extrapolate connection 
> information from that file). Accumulo's site configuration file is not meant 
> to be shared with consumers which means that this exact approach is not 
> sufficient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7950) StorageHandler resources aren't added to Tez Session if already Session is already Open

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136784#comment-14136784
 ] 

Josh Elser commented on HIVE-7950:
--

Test failures appear unrelated to me. Can anyone give this a review for me?

> StorageHandler resources aren't added to Tez Session if already Session is 
> already Open
> ---
>
> Key: HIVE-7950
> URL: https://issues.apache.org/jira/browse/HIVE-7950
> Project: Hive
>  Issue Type: Bug
>  Components: StorageHandler, Tez
>Reporter: Josh Elser
>Assignee: Josh Elser
> Fix For: 0.14.0
>
> Attachments: HIVE-7950-1.diff, HIVE-7950.2.patch, HIVE-7950.3.patch, 
> hive-7950-tez-WIP.diff
>
>
> Was trying to run some queries using the AccumuloStorageHandler when using 
> the Tez execution engine. Some things that classes which were added to 
> tmpjars weren't making it into the container. When a Tez Session is already 
> open, as is the normal case when simply using the `hive` command, the 
> resources aren't added.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8104) Insert statements against ACID tables NPE when vectorization is on

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136771#comment-14136771
 ] 

Eugene Koifman commented on HIVE-8104:
--

+1 pending tests

> Insert statements against ACID tables NPE when vectorization is on
> --
>
> Key: HIVE-8104
> URL: https://issues.apache.org/jira/browse/HIVE-8104
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor, Vectorization
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Attachments: HIVE-8104.2.patch, HIVE-8104.patch
>
>
> Doing an insert against a table that is using ACID format with the 
> transaction manager set to DbTxnManager and vectorization turned on results 
> in an NPE.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-6705) hive jdbc can not used by jmeter, because of unsupported auto commit feature

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136751#comment-14136751
 ] 

Hive QA commented on HIVE-6705:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12669000/HIVE-6705.2.patch.txt

{color:green}SUCCESS:{color} +1 6279 tests passed

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/833/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/833/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-833/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12669000

> hive jdbc can not used by jmeter, because of unsupported auto commit feature
> 
>
> Key: HIVE-6705
> URL: https://issues.apache.org/jira/browse/HIVE-6705
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.12.0
> Environment: CentOS_X86_64 
> JMeter 2.11
>Reporter: Ben
>Assignee: Navis
> Attachments: HIVE-6705.1.patch.txt, HIVE-6705.2.patch.txt
>
>
> In apache jmeter ,the autocommit property is required.
> but in the hive jdbc the auto commit is unsupported method.
> in 
> /jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java
> {quote}
>  public void setAutoCommit(boolean autoCommit) throws SQLException {
> // TODO Auto-generated method stub
>   throw new {color:red}  SQLException("Method not supported");
> {color}
>   }
> {quote}
> so ,should  we make a mock to support  the auto commit property == false ?
> {quote}
> public void setAutoCommit(boolean autoCommit) throws SQLException {
>   // TODO Auto-generated method stub
>  {color:red}if(autoCommit) {color}
>   throw new SQLException("Method not supported");
>  else
>   return;
>  }
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8156) Vectorized reducers need to avoid memory build-up during a single key

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-8156:
--
Status: Patch Available  (was: Open)

> Vectorized reducers need to avoid memory build-up during a single key
> -
>
> Key: HIVE-8156
> URL: https://issues.apache.org/jira/browse/HIVE-8156
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 0.14.0
>
> Attachments: HIVE-8156.1.patch, HIVE-8156.2.patch
>
>
> When encountering a skewed key with a large number of values, the vectorized 
> reducer will not release memory within the loop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8106) Enable vectorization for spark [spark branch]

[Chinna Rao Lalam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinna Rao Lalam updated HIVE-8106:
---
Status: Open  (was: Patch Available)

> Enable vectorization for spark [spark branch]
> -
>
> Key: HIVE-8106
> URL: https://issues.apache.org/jira/browse/HIVE-8106
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-8106-spark.patch, HIVE-8106.1-spark.patch
>
>
> Enable the vectorization optimization on spark



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8156) Vectorized reducers need to avoid memory build-up during a single key

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-8156:
--
Attachment: HIVE-8156.2.patch

> Vectorized reducers need to avoid memory build-up during a single key
> -
>
> Key: HIVE-8156
> URL: https://issues.apache.org/jira/browse/HIVE-8156
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 0.14.0
>
> Attachments: HIVE-8156.1.patch, HIVE-8156.2.patch
>
>
> When encountering a skewed key with a large number of values, the vectorized 
> reducer will not release memory within the loop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8156) Vectorized reducers need to avoid memory build-up during a single key

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-8156:
--
Status: Open  (was: Patch Available)

> Vectorized reducers need to avoid memory build-up during a single key
> -
>
> Key: HIVE-8156
> URL: https://issues.apache.org/jira/browse/HIVE-8156
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 0.14.0
>
> Attachments: HIVE-8156.1.patch
>
>
> When encountering a skewed key with a large number of values, the vectorized 
> reducer will not release memory within the loop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8106) Enable vectorization for spark [spark branch]

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136719#comment-14136719
 ] 

Hive QA commented on HIVE-8106:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12669139/HIVE-8106.1-spark.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/134/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/134/console
Test logs: 
http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-134/

Messages:
{noformat}
 This message was trimmed, see log for full details 
As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_CASE KW_ARRAY" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_NULL LESSTHAN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_NOT KW_DATE" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:115:5: 
Decision can match input such as "KW_CLUSTER KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:127:5: 
Decision can match input such as "KW_PARTITION KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:138:5: 
Decision can match input such as "KW_DISTRIBUTE KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:149:5: 
Decision can match input such as "KW_SORT KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:166:7: 
Decision can match input such as "STAR" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_ARRAY" using multiple alternatives: 2, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_STRUCT" using multiple alternatives: 4, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_UNIONTYPE" using multiple alternatives: 5, 
6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_NULL" using multiple alternatives: 1, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_TRUE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_FALSE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_DATE StringLiteral" using multiple 
alternatives: 2, 3

As a result, alternative(s) 3 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_ORDER 
KW_BY" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE 
KW_BY" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER 
KW_BY" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL 
KW_VIEW" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT 
KW_INTO" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_MAP LPAREN" 
using multiple alternatives: 2, 9

[jira] [Updated] (HIVE-8106) Enable vectorization for spark [spark branch]

[Chinna Rao Lalam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinna Rao Lalam updated HIVE-8106:
---
Status: Patch Available  (was: Open)

> Enable vectorization for spark [spark branch]
> -
>
> Key: HIVE-8106
> URL: https://issues.apache.org/jira/browse/HIVE-8106
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-8106-spark.patch, HIVE-8106.1-spark.patch
>
>
> Enable the vectorization optimization on spark



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8156) Vectorized reducers need to avoid memory build-up during a single key

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-8156:
--
Release Note: Cleanup memory within the vectorized reducer loop
  Status: Patch Available  (was: Open)

> Vectorized reducers need to avoid memory build-up during a single key
> -
>
> Key: HIVE-8156
> URL: https://issues.apache.org/jira/browse/HIVE-8156
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 0.14.0
>
> Attachments: HIVE-8156.1.patch
>
>
> When encountering a skewed key with a large number of values, the vectorized 
> reducer will not release memory within the loop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8156) Vectorized reducers need to avoid memory build-up during a single key

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-8156:
--
Attachment: HIVE-8156.1.patch

> Vectorized reducers need to avoid memory build-up during a single key
> -
>
> Key: HIVE-8156
> URL: https://issues.apache.org/jira/browse/HIVE-8156
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 0.14.0
>
> Attachments: HIVE-8156.1.patch
>
>
> When encountering a skewed key with a large number of values, the vectorized 
> reducer will not release memory within the loop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8156) Vectorized reducers need to avoid memory build-up during a single key

[Gopal V (JIRA)]

Gopal V created HIVE-8156:
-

 Summary: Vectorized reducers need to avoid memory build-up during 
a single key
 Key: HIVE-8156
 URL: https://issues.apache.org/jira/browse/HIVE-8156
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.14.0
Reporter: Gopal V
Assignee: Gopal V
 Fix For: 0.14.0


When encountering a skewed key with a large number of values, the vectorized 
reducer will not release memory within the loop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-5744) Implement support for BETWEEN in SELECT list

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136694#comment-14136694
 ] 

Hive QA commented on HIVE-5744:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12668997/HIVE-5744.1.patch.txt

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6279 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/832/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/832/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-832/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12668997

> Implement support for BETWEEN in SELECT list
> 
>
> Key: HIVE-5744
> URL: https://issues.apache.org/jira/browse/HIVE-5744
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Eric Hanson
>Assignee: Navis
> Attachments: HIVE-5744.1.patch.txt
>
>
> Queries like 
> SELECT col1 BETWEEN 0 and 10 from T;
> fail in vectorized mode. Support needs to be implemented for a BETWEEN 
> expression in the SELECT list, comparable to how it was added for comparison 
> operators (<, >, ...). These were done by adding new, templates that return a 
> value for a comparison instead of applying a filter. See 
> ColumnCompareScalar.txt under ql/src/gen for an example.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8078) ORC Delta encoding corrupts data when delta overflows long

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-8078:
-
Attachment: HIVE-8078.2.patch

Fixes HIVE QA test failures.

> ORC Delta encoding corrupts data when delta overflows long
> --
>
> Key: HIVE-8078
> URL: https://issues.apache.org/jira/browse/HIVE-8078
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.14.0, 0.13.1
>Reporter: Tim Patterson
>Priority: Critical
> Attachments: HIVE-8078-testcase.patch, HIVE-8078.1.patch, 
> HIVE-8078.2.patch
>
>
> There is an issue with the integer encoding that can cause corruption in 
> certain cases.
> The following 3 longs cause this failure.
> 4513343538618202711
> 2911390882471569739
> -9181829309989854913
> I believe that even though the numbers are in decreasing order, the delta 
> between the last two numbers overflows causing a positive delta, in this case 
> the last digit ends up being corrupted (the delta is applied for the wrong 
> sign resulting in -3442132998776557225 instead of -9181829309989854913.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 25711: Create root scratch dir with 733 instead of 777 perms

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25711/#review53644
---

Ship it!


Ship It!

- Gopal V


On Sept. 16, 2014, 10:20 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25711/
> ---
> 
> (Updated Sept. 16, 2014, 10:20 p.m.)
> 
> 
> Review request for hive, Gopal V and Thejas Nair.
> 
> 
> Bugs: HIVE-8143
> https://issues.apache.org/jira/browse/HIVE-8143
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-8143
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 185ea95 
>   itests/hive-unit/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 
> b430cae 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 241d315 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java d7227ed 
> 
> Diff: https://reviews.apache.org/r/25711/diff/
> 
> 
> Testing
> ---
> 
> Unit + manual
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

[jira] [Commented] (HIVE-8143) Create root scratch dir with 733 instead of 777 perms

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136675#comment-14136675
 ] 

Gopal V commented on HIVE-8143:
---

+1 - works with 777 and 733 /tmp/hive dirs.

> Create root scratch dir with 733 instead of 777 perms
> -
>
> Key: HIVE-8143
> URL: https://issues.apache.org/jira/browse/HIVE-8143
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 0.14.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 0.14.0
>
> Attachments: HIVE-8143.1.patch, HIVE-8143.2.patch
>
>
> hive.exec.scratchdir which is treated as the root scratch directory on hdfs 
> only needs to be writable by all. We can use 733 instead of 777 for doing 
> that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8155) In select statement after * any random characters are allowed in hive but in RDBMS its not allowed

[Ferdinand Xu (JIRA)]

Ferdinand Xu created HIVE-8155:
--

 Summary:  In select statement after * any random characters are 
allowed in hive but in RDBMS its not allowed
 Key: HIVE-8155
 URL: https://issues.apache.org/jira/browse/HIVE-8155
 Project: Hive
  Issue Type: Improvement
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
Priority: Critical


In select statement after * any random characters are allowed in hive but in 
RDBMS its not allowed. 
Steps:
In the below query "abcdef" is random characters.

In RDBMS(oracle): 
select *abcdef from mytable;
Output: 
ERROR prepare() failed with: ORA-00923: FROM keyword not found where expected

In Hive:
select *abcdef from mytable;
Output: 
Query worked fine and display all the records of mytable.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HIVE-8134) Umbrella JIRA to track concurrency improvements

[Ferdinand Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ferdinand Xu reassigned HIVE-8134:
--

Assignee: Ferdinand Xu

> Umbrella JIRA to track concurrency improvements
> ---
>
> Key: HIVE-8134
> URL: https://issues.apache.org/jira/browse/HIVE-8134
> Project: Hive
>  Issue Type: Improvement
>Reporter: Brock Noland
>Assignee: Ferdinand Xu
>
> The goal of this JIRA is track supportability issues with concurrent users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8154) HadoopThriftAuthBridge20S.getHadoopSaslProperties is incompatible with Hadoop 2.4.1 and later

[Yu Gao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136653#comment-14136653
 ] 

Yu Gao commented on HIVE-8154:
--

Hadoop 2.4.1 has refactored the sasl related classes, moving sasl properties 
related code including sasl QOP setting from method SaslRpcServer.init to a new 
class named SaslPropertiesResolver. However, the 
HadoopThriftAuthBridge20S.getHadoopSaslProperties method still replies on the 
old implementation, which will not work for newer versions of Hadoop.

> HadoopThriftAuthBridge20S.getHadoopSaslProperties is incompatible with Hadoop 
> 2.4.1 and later
> -
>
> Key: HIVE-8154
> URL: https://issues.apache.org/jira/browse/HIVE-8154
> Project: Hive
>  Issue Type: Bug
>  Components: Shims
>Affects Versions: 0.13.1
> Environment: Kerberos
> Hadoop 2.4.1
>Reporter: Yu Gao
>
> Enabled Kerberos in Hadoop 2.4.1 and Hive 0.13.1, with all kerberos 
> properties and principals/keytabs configured correctly. Hadoop cluster is 
> healthy but Hive Server2 is not able to start, due to following error in 
> hive.log:
> 2014-09-16 13:52:32,964 ERROR thrift.ThriftCLIService 
> (ThriftBinaryCLIService.java:run(93)) - Error: 
> java.lang.IllegalArgumentException: Unknown auth type: null Allowed values 
> are: [auth-int, auth-conf, auth]
>   at org.apache.hive.service.auth.SaslQOP.fromString(SaslQOP.java:56)
>   at 
> org.apache.hive.service.auth.HiveAuthFactory.getSaslProperties(HiveAuthFactory.java:118)
>   at 
> org.apache.hive.service.auth.HiveAuthFactory.getAuthTransFactory(HiveAuthFactory.java:133)
>   at 
> org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:43)
>   at java.lang.Thread.run(Thread.java:853)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8154) HadoopThriftAuthBridge20S.getHadoopSaslProperties is incompatible with Hadoop 2.4.1 and later

[Yu Gao (JIRA)]

Yu Gao created HIVE-8154:


 Summary: HadoopThriftAuthBridge20S.getHadoopSaslProperties is 
incompatible with Hadoop 2.4.1 and later
 Key: HIVE-8154
 URL: https://issues.apache.org/jira/browse/HIVE-8154
 Project: Hive
  Issue Type: Bug
  Components: Shims
Affects Versions: 0.13.1
 Environment: Kerberos
Hadoop 2.4.1
Reporter: Yu Gao


Enabled Kerberos in Hadoop 2.4.1 and Hive 0.13.1, with all kerberos properties 
and principals/keytabs configured correctly. Hadoop cluster is healthy but Hive 
Server2 is not able to start, due to following error in hive.log:

2014-09-16 13:52:32,964 ERROR thrift.ThriftCLIService 
(ThriftBinaryCLIService.java:run(93)) - Error: 
java.lang.IllegalArgumentException: Unknown auth type: null Allowed values are: 
[auth-int, auth-conf, auth]
at org.apache.hive.service.auth.SaslQOP.fromString(SaslQOP.java:56)
at 
org.apache.hive.service.auth.HiveAuthFactory.getSaslProperties(HiveAuthFactory.java:118)
at 
org.apache.hive.service.auth.HiveAuthFactory.getAuthTransFactory(HiveAuthFactory.java:133)
at 
org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:43)
at java.lang.Thread.run(Thread.java:853)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8125) CBO Trunk Merge: On Failure Fall Back to Non CBO

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136628#comment-14136628
 ] 

Hive QA commented on HIVE-8125:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12668980/HIVE-8125.9.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/831/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/831/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-831/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-831/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target 
shims/0.20S/target shims/0.23/target shims/aggregator/target 
shims/common/target shims/common-secure/target packaging/target 
hbase-handler/target testutils/target jdbc/target metastore/target 
itests/target itests/hcatalog-unit/target itests/test-serde/target 
itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target 
itests/hive-unit/target itests/custom-serde/target itests/util/target 
hcatalog/target hcatalog/core/target hcatalog/streaming/target 
hcatalog/server-extensions/target hcatalog/hcatalog-pig-adapter/target 
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target 
accumulo-handler/target hwi/target common/target common/src/gen service/target 
contrib/target serde/target beeline/target odbc/target cli/target 
ql/dependency-reduced-pom.xml ql/target
+ svn update

Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1625449.

At revision 1625449.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12668980

> CBO Trunk Merge: On Failure Fall Back to Non CBO
> 
>
> Key: HIVE-8125
> URL: https://issues.apache.org/jira/browse/HIVE-8125
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-8125.9.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8112) Change reporting string to reflect update in Tez

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136624#comment-14136624
 ] 

Hive QA commented on HIVE-8112:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12668907/HIVE-8112.1.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6279 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_analyze
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/830/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/830/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-830/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12668907

> Change reporting string to reflect update in Tez
> 
>
> Key: HIVE-8112
> URL: https://issues.apache.org/jira/browse/HIVE-8112
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
>Priority: Minor
> Attachments: HIVE-8112.1.patch
>
>
> We're now printing:
> Status: Running (application id: Executing on YARN cluster with App id 
> application_1410822917153_0001)
> Where Tez just used to print the id it now has the full "Executing on 
> YARN..." string.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8043) Support merging small files [Spark Branch]

[Rui Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136618#comment-14136618
 ] 

Rui Li commented on HIVE-8043:
--

Thank you [~xuefuz]. I'll take a look.

> Support merging small files [Spark Branch]
> --
>
> Key: HIVE-8043
> URL: https://issues.apache.org/jira/browse/HIVE-8043
> Project: Hive
>  Issue Type: Task
>  Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Rui Li
>  Labels: Spark-M1
>
> Hive currently supports merging small files with MR as the execution engine. 
> There are options available for this, such as 
> {code}
> hive.merge.mapfiles
> hive.merge.mapredfiles
> {code}
> Hive.merge.sparkfiles is already introduced in HIVE-7810. To make it work, we 
> might need a little more research and design on this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 25524: HIVE-8047 Lazy char/varchar are not using escape char defined in serde params

[Thejas Nair]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25524/#review53634
---

Ship it!


Ship It!

- Thejas Nair


On Sept. 12, 2014, 3:22 a.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25524/
> ---
> 
> (Updated Sept. 12, 2014, 3:22 a.m.)
> 
> 
> Review request for hive and Thejas Nair.
> 
> 
> Bugs: HIVE-8047
> https://issues.apache.org/jira/browse/HIVE-8047
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Update lazy char/varchar to support escape character
> 
> 
> Diffs
> -
> 
>   data/files/data_with_escape.txt PRE-CREATION 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
> ae128a9 
>   ql/src/test/queries/clientpositive/escape3.q PRE-CREATION 
>   ql/src/test/results/clientpositive/escape3.q.out PRE-CREATION 
>   serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyHiveChar.java ef469eb 
>   serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyHiveVarchar.java 
> bc8d41e 
>   serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyString.java 28b3f86 
>   serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1d62422 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyHiveCharObjectInspector.java
>  65fb1ab 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyHiveVarcharObjectInspector.java
>  c802ed0 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java
>  734b9d8 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableHiveCharObjectInspector.java
>  c340dcb 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableHiveVarcharObjectInspector.java
>  7bbfd1f 
> 
> Diff: https://reviews.apache.org/r/25524/diff/
> 
> 
> Testing
> ---
> 
> New q file test
> 
> 
> Thanks,
> 
> Jason Dere
> 
>

[jira] [Commented] (HIVE-8047) Lazy char/varchar are not using escape char defined in serde params

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136603#comment-14136603
 ] 

Thejas M Nair commented on HIVE-8047:
-

+1

> Lazy char/varchar are not using escape char defined in serde params
> ---
>
> Key: HIVE-8047
> URL: https://issues.apache.org/jira/browse/HIVE-8047
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-8047.1.patch, HIVE-8047.2.patch, HIVE-8047.3.patch, 
> HIVE-8047.4.patch
>
>
> A table with char/varchar columns that specifies an escape character for 
> string data still has the escape characters in the char/varchar columns.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8141) Refactor the GraphTran code by moving union handling logic to UnionTran [Spark Branch]

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136589#comment-14136589
 ] 

Hive QA commented on HIVE-8141:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12669270/HIVE-8141.1-spark.patch

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 6437 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/133/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/133/console
Test logs: 
http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-133/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12669270

> Refactor the GraphTran code by moving union handling logic to UnionTran 
> [Spark Branch]
> --
>
> Key: HIVE-8141
> URL: https://issues.apache.org/jira/browse/HIVE-8141
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Na Yang
>Assignee: Na Yang
>  Labels: Spark-M1
> Attachments: HIVE-8141.1-spark.patch
>
>
> In the current hive on spark code, union logic is handled in the GraphTran 
> class. The Union logic could be moved to the UnionTran class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136584#comment-14136584
 ] 

Sergey Shelukhin commented on HIVE-8111:


Another solution that might work is casting to minimum required type for CBO, 
instead of result expression type. 

> CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
> 
>
> Key: HIVE-8111
> URL: https://issues.apache.org/jira/browse/HIVE-8111
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>
> Original test failure: looks like column type changes to different decimals 
> in most cases. In one case it causes the integer part to be too big to fit, 
> so the result becomes null it seems.
> What happens is that CBO adds casts to arithmetic expressions to make them 
> type compatible; these casts become part of new AST, and then Hive adds casts 
> on top of these casts. This (the first part) also causes lots of out file 
> changes. It's not clear how to best fix it so far, in addition to incorrect 
> decimal width and sometimes nulls when width is larger than allowed in Hive.
> Option one - don't add those for numeric ops - cannot be done if numeric op 
> is a part of compare, for which CBO needs correct types.
> Option two - unwrap casts when determining type in Hive - hard or impossible 
> to tell apart CBO-added casts and user casts. 
> Option three - don't change types in Hive if CBO has run - seems hacky and 
> hard to ensure it's applied everywhere.
> Option four - map all expressions precisely between two trees and remove 
> casts again after optimization, will be pretty difficult.
> Option five - somehow mark those casts. Not sure about how yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8148) HDFS Path named with file:// instead of file:/// results in Unit test failures in Windows

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136578#comment-14136578
 ] 

Jason Dere commented on HIVE-8148:
--

Can you put this on RB?
I thought that on Windows, URI.getPath() would return a path starting with "/" 
(for example, new URI("file:///c:/test/test.py").getPath() returns 
"/c:/test/test.py"), and this allowed a lot of the stuff with "file://" to work 
properly. Would you be able to make changes like that where possible - for 
Path/File try using toUri().getPath() to get a proper path string.


> HDFS Path named with file:// instead of file:/// results in Unit test 
> failures in Windows
> -
>
> Key: HIVE-8148
> URL: https://issues.apache.org/jira/browse/HIVE-8148
> Project: Hive
>  Issue Type: Bug
> Environment: Windows
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-8148.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7100) Users of hive should be able to specify skipTrash when dropping tables.

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136572#comment-14136572
 ] 

Xuefu Zhang commented on HIVE-7100:
---

+1 pending on test result.

> Users of hive should be able to specify skipTrash when dropping tables.
> ---
>
> Key: HIVE-7100
> URL: https://issues.apache.org/jira/browse/HIVE-7100
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 0.13.0
>Reporter: Ravi Prakash
>Assignee: david serafini
> Attachments: HIVE-7100.1.patch, HIVE-7100.10.patch, 
> HIVE-7100.2.patch, HIVE-7100.3.patch, HIVE-7100.4.patch, HIVE-7100.5.patch, 
> HIVE-7100.8.patch, HIVE-7100.9.patch, HIVE-7100.patch
>
>
> Users of our clusters are often running up against their quota limits because 
> of Hive tables. When they drop tables, they have to then manually delete the 
> files from HDFS using skipTrash. This is cumbersome and unnecessary. We 
> should enable users to skipTrash directly when dropping tables.
> We should also be able to provide this functionality without polluting SQL 
> syntax.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8151) Dynamic partition sort optimization inserts record wrongly to partition when used with GroupBy

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-8151:
-
Description: 
HIVE-6455 added dynamic partition sort optimization. It added startGroup() 
method to FileSink operator to look for changes in reduce key for creating 
partition directories. This method however is not reliable as the key called 
with startGroup() is different from the key called with processOp(). 
startGroup() is called with newly changed key whereas processOp() is called 
with previously aggregated key. This will result in processOp() writing the 
last row of previous group as the first row of next group. This happens only 
when used with group by operator.

The fix is to not rely on startGroup() and do the partition directory creation 
in processOp() itself.

  was:
HIVE-6455 added dynamic partition sort optimization. It added startGroup() 
method to FileSink operator to look for changes in reduce key for creating 
partition directories. This method however is reliable as the key called with 
startGroup() is different from the key called with processOp(). startGroup() is 
called with newly changed key whereas processOp() is called with previously 
aggregated key. This will result in processOp() writing the last row of 
previous group as the first row of next group. This happens only when used with 
group by operator.

The fix is to not rely on startGroup() and do the partition directory creation 
in processOp() itself.


> Dynamic partition sort optimization inserts record wrongly to partition when 
> used with GroupBy
> --
>
> Key: HIVE-8151
> URL: https://issues.apache.org/jira/browse/HIVE-8151
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0, 0.13.1
>Reporter: Prasanth J
>Assignee: Prasanth J
>Priority: Critical
> Attachments: HIVE-8151.1.patch
>
>
> HIVE-6455 added dynamic partition sort optimization. It added startGroup() 
> method to FileSink operator to look for changes in reduce key for creating 
> partition directories. This method however is not reliable as the key called 
> with startGroup() is different from the key called with processOp(). 
> startGroup() is called with newly changed key whereas processOp() is called 
> with previously aggregated key. This will result in processOp() writing the 
> last row of previous group as the first row of next group. This happens only 
> when used with group by operator.
> The fix is to not rely on startGroup() and do the partition directory 
> creation in processOp() itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8153) Reduce the verbosity of debug logs in ORC record reader

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-8153:
-
Attachment: HIVE-8153.2.patch

> Reduce the verbosity of debug logs in ORC record reader
> ---
>
> Key: HIVE-8153
> URL: https://issues.apache.org/jira/browse/HIVE-8153
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.14.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>Priority: Trivial
> Attachments: HIVE-8153.1.patch, HIVE-8153.2.patch
>
>
> Following fields are logged for every row.
> {code}
> if (LOG.isDebugEnabled()) {
>   LOG.debug("row from " + reader.path);
>   LOG.debug("orc row = " + result);
> }
> {code}
> This should be moved to trace logging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8153) Reduce the verbosity of debug logs in ORC record reader

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136545#comment-14136545
 ] 

Gopal V commented on HIVE-8153:
---

LGTM - +1

> Reduce the verbosity of debug logs in ORC record reader
> ---
>
> Key: HIVE-8153
> URL: https://issues.apache.org/jira/browse/HIVE-8153
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.14.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>Priority: Trivial
> Attachments: HIVE-8153.1.patch, HIVE-8153.2.patch
>
>
> Following fields are logged for every row.
> {code}
> if (LOG.isDebugEnabled()) {
>   LOG.debug("row from " + reader.path);
>   LOG.debug("orc row = " + result);
> }
> {code}
> This should be moved to trace logging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8140) Remove obsolete code from SparkWork [Spark Branch]

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-8140:
--
   Resolution: Fixed
Fix Version/s: spark-branch
   Status: Resolved  (was: Patch Available)

Patch committed to Spark branch. Thanks to Chao for the contribution.

> Remove obsolete code from SparkWork [Spark Branch]
> --
>
> Key: HIVE-8140
> URL: https://issues.apache.org/jira/browse/HIVE-8140
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Chao
>  Labels: Spark-M1
> Fix For: spark-branch
>
> Attachments: HIVE-8140.1-spark.patch
>
>
> There are old code in SparkWork about get/set map/reduce work. It's from POC 
> code, which isn't applicable any more. We should remove it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8140) Remove obsolete code from SparkWork [Spark Branch]

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136542#comment-14136542
 ] 

Xuefu Zhang commented on HIVE-8140:
---

+1

> Remove obsolete code from SparkWork [Spark Branch]
> --
>
> Key: HIVE-8140
> URL: https://issues.apache.org/jira/browse/HIVE-8140
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Chao
>  Labels: Spark-M1
> Attachments: HIVE-8140.1-spark.patch
>
>
> There are old code in SparkWork about get/set map/reduce work. It's from POC 
> code, which isn't applicable any more. We should remove it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8141) Refactor the GraphTran code by moving union handling logic to UnionTran [Spark Branch]

[Na Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136541#comment-14136541
 ] 

Na Yang commented on HIVE-8141:
---

review board link: https://reviews.apache.org/r/25720/

> Refactor the GraphTran code by moving union handling logic to UnionTran 
> [Spark Branch]
> --
>
> Key: HIVE-8141
> URL: https://issues.apache.org/jira/browse/HIVE-8141
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Na Yang
>Assignee: Na Yang
>  Labels: Spark-M1
> Attachments: HIVE-8141.1-spark.patch
>
>
> In the current hive on spark code, union logic is handled in the GraphTran 
> class. The Union logic could be moved to the UnionTran class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 25720: HIVE-8141:Refactor the GraphTran code by moving union handling logic to UnionTran [Spark Branch]

[Na Yang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25720/
---

Review request for hive, Brock Noland and Xuefu Zhang.


Bugs: HIVE-8141
https://issues.apache.org/jira/browse/HIVE-8141


Repository: hive-git


Description
---

Refactor the GraphTran code by moving union handling logic to UnionTran


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/GraphTran.java 93674c1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/UnionTran.java 40f22a0 

Diff: https://reviews.apache.org/r/25720/diff/


Testing
---


Thanks,

Na Yang

[jira] [Updated] (HIVE-8141) Refactor the GraphTran code by moving union handling logic to UnionTran [Spark Branch]

[Na Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Yang updated HIVE-8141:
--
Status: Patch Available  (was: Open)

> Refactor the GraphTran code by moving union handling logic to UnionTran 
> [Spark Branch]
> --
>
> Key: HIVE-8141
> URL: https://issues.apache.org/jira/browse/HIVE-8141
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Na Yang
>Assignee: Na Yang
>  Labels: Spark-M1
> Attachments: HIVE-8141.1-spark.patch
>
>
> In the current hive on spark code, union logic is handled in the GraphTran 
> class. The Union logic could be moved to the UnionTran class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8141) Refactor the GraphTran code by moving union handling logic to UnionTran [Spark Branch]

[Na Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Na Yang updated HIVE-8141:
--
Attachment: HIVE-8141.1-spark.patch

> Refactor the GraphTran code by moving union handling logic to UnionTran 
> [Spark Branch]
> --
>
> Key: HIVE-8141
> URL: https://issues.apache.org/jira/browse/HIVE-8141
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Na Yang
>Assignee: Na Yang
>  Labels: Spark-M1
> Attachments: HIVE-8141.1-spark.patch
>
>
> In the current hive on spark code, union logic is handled in the GraphTran 
> class. The Union logic could be moved to the UnionTran class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7100) Users of hive should be able to specify skipTrash when dropping tables.

[david serafini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

david serafini updated HIVE-7100:
-
Attachment: HIVE-7100.10.patch

this fixes the last review comment and improves the documentation a bit

> Users of hive should be able to specify skipTrash when dropping tables.
> ---
>
> Key: HIVE-7100
> URL: https://issues.apache.org/jira/browse/HIVE-7100
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 0.13.0
>Reporter: Ravi Prakash
>Assignee: david serafini
> Attachments: HIVE-7100.1.patch, HIVE-7100.10.patch, 
> HIVE-7100.2.patch, HIVE-7100.3.patch, HIVE-7100.4.patch, HIVE-7100.5.patch, 
> HIVE-7100.8.patch, HIVE-7100.9.patch, HIVE-7100.patch
>
>
> Users of our clusters are often running up against their quota limits because 
> of Hive tables. When they drop tables, they have to then manually delete the 
> files from HDFS using skipTrash. This is cumbersome and unnecessary. We 
> should enable users to skipTrash directly when dropping tables.
> We should also be able to provide this functionality without polluting SQL 
> syntax.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8055) Code cleanup after HIVE-8054 [Spark Branch]

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-8055:
--
   Resolution: Fixed
Fix Version/s: spark-branch
   Status: Resolved  (was: Patch Available)

Patch committed to Spark branch. Thanks to Na for the contribution.

> Code cleanup after HIVE-8054 [Spark Branch]
> ---
>
> Key: HIVE-8055
> URL: https://issues.apache.org/jira/browse/HIVE-8055
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Na Yang
>  Labels: Spark-M1
> Fix For: spark-branch
>
> Attachments: HIVE-8055-spark.patch
>
>
> There is quite some code handling union removal optimization in SparkCompiler 
> and related classes. We need to clean this up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7777) Add CSV Serde based on OpenCSV

[Lefty Leverenz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lefty Leverenz updated HIVE-:
-
Labels: TODOC14  (was: )

> Add CSV Serde based on OpenCSV
> --
>
> Key: HIVE-
> URL: https://issues.apache.org/jira/browse/HIVE-
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
>  Labels: TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-.1.patch, HIVE-.2.patch, HIVE-.3.patch, 
> HIVE-.patch, csv-serde-master.zip
>
>
> There is no official support for csvSerde for hive while there is an open 
> source project in github(https://github.com/ogrodnek/csv-serde). CSV is of 
> high frequency in use as a data format.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 25178: HIVE-7100 Add DROP TABLE PURGE

[david seraf]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25178/
---

(Updated Sept. 16, 2014, 11:52 p.m.)


Review request for hive and Xuefu Zhang.


Changes
---

patch 10 fixes the last remaining review comments


Bugs: HIVE-7100
https://issues.apache.org/jira/browse/HIVE-7100


Repository: hive-git


Description
---

Add PURGE option to DROP TABLE command to skip saving table data to the trash


Diffs (updated)
-

  
hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatPartitionPublish.java
 c2a0f5f 
  
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/tool/TestTempletonUtils.java
 af952f2 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
33745e4 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
8765d53 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreFsImpl.java 
cff0718 
  metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
e86a90a 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreFS.java a141793 
  metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java a32507d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 56bcf1c 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java f40f5f7 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
6d18884 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 05cde3e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 25cd3a5 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DropTableDesc.java ba30e1f 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 406aae9 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveRemote.java 1a5ba87 
  ql/src/test/queries/clientpositive/drop_table_purge.q PRE-CREATION 
  ql/src/test/results/clientpositive/drop_table_purge.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/25178/diff/


Testing
---

added code test and added QL test.  Tests passed in CI, but other, unrelated 
tests failed.


Thanks,

david seraf

Re: Review Request 25178: HIVE-7100 Add DROP TABLE PURGE

[david seraf]

> On Sept. 16, 2014, 10:21 p.m., Xuefu Zhang wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, 
> > line 1409
> > 
> >
> > Maybe I'm misunderstanding. the added dropTableData(Path, boolean) is 
> > called by some public API, such as drop_table_with_environment_context(), 
> > but we don't need to change the API. Instead, before calling 
> > dropTableData(), we just need to get a boolean value (ifPurge) from the 
> > context. The signature of the API doesn't need to change.

I see it now.  Sorry.  Fixed in the next patch.


- david


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25178/#review53612
---


On Sept. 12, 2014, 9:51 p.m., david seraf wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25178/
> ---
> 
> (Updated Sept. 12, 2014, 9:51 p.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-7100
> https://issues.apache.org/jira/browse/HIVE-7100
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add PURGE option to DROP TABLE command to skip saving table data to the trash
> 
> 
> Diffs
> -
> 
>   
> hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatPartitionPublish.java
>  be7134f 
>   
> hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/tool/TestTempletonUtils.java
>  af952f2 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHiveServer2.java
>  da51a55 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> 9489949 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
> a94a7a3 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreFsImpl.java 
> cff0718 
>   metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
> cbdba30 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreFS.java 
> a141793 
>   metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java 613b709 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java cd017d8 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java e387b8f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  4cf98d8 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> f31a409 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 32db0c7 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DropTableDesc.java ba30e1f 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 406aae9 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveRemote.java 1a5ba87 
>   ql/src/test/queries/clientpositive/drop_table_purge.q PRE-CREATION 
>   ql/src/test/results/clientpositive/drop_table_purge.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/25178/diff/
> 
> 
> Testing
> ---
> 
> added code test and added QL test.  Tests passed in CI, but other, unrelated 
> tests failed.
> 
> 
> Thanks,
> 
> david seraf
> 
>

[jira] [Commented] (HIVE-8126) Standalone hive-jdbc jar is not packaged in the Hive distribution

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136530#comment-14136530
 ] 

Hive QA commented on HIVE-8126:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12668979/HIVE-8126.1.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6275 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestParse.testParse_union
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/829/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/829/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-829/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12668979

> Standalone hive-jdbc jar is not packaged in the Hive distribution
> -
>
> Key: HIVE-8126
> URL: https://issues.apache.org/jira/browse/HIVE-8126
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.14.0
>Reporter: Deepesh Khandelwal
>Assignee: Deepesh Khandelwal
> Fix For: 0.14.0
>
> Attachments: HIVE-8126.1.patch
>
>
> With HIVE-538 we started creating the hive-jdbc-*-standalone.jar but the 
> packaging/distribution does not contain the standalone jdbc jar. I would have 
> expected it to locate under the lib folder of the distribution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8153) Reduce the verbosity of debug logs in ORC record reader

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-8153:
-
Status: Patch Available  (was: Open)

> Reduce the verbosity of debug logs in ORC record reader
> ---
>
> Key: HIVE-8153
> URL: https://issues.apache.org/jira/browse/HIVE-8153
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.14.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>Priority: Trivial
> Attachments: HIVE-8153.1.patch
>
>
> Following fields are logged for every row.
> {code}
> if (LOG.isDebugEnabled()) {
>   LOG.debug("row from " + reader.path);
>   LOG.debug("orc row = " + result);
> }
> {code}
> This should be moved to trace logging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8153) Reduce the verbosity of debug logs in ORC record reader

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-8153:
-
Attachment: HIVE-8153.1.patch

> Reduce the verbosity of debug logs in ORC record reader
> ---
>
> Key: HIVE-8153
> URL: https://issues.apache.org/jira/browse/HIVE-8153
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.14.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>Priority: Trivial
> Attachments: HIVE-8153.1.patch
>
>
> Following fields are logged for every row.
> {code}
> if (LOG.isDebugEnabled()) {
>   LOG.debug("row from " + reader.path);
>   LOG.debug("orc row = " + result);
> }
> {code}
> This should be moved to trace logging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 25595: HIVE-8083: Authorization DDLs should not enforce hive identifier syntax for user or group namesname that

[Xuefu Zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25595/#review53624
---



ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g


Is this supposed to be just identifier itself?


- Xuefu Zhang


On Sept. 16, 2014, 6:29 p.m., Prasad Mujumdar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25595/
> ---
> 
> (Updated Sept. 16, 2014, 6:29 p.m.)
> 
> 
> Review request for hive and Brock Noland.
> 
> 
> Bugs: HIVE-8083
> https://issues.apache.org/jira/browse/HIVE-8083
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The compiler expects principals (user, group and role) as hive identifiers 
> for authorization DDLs. The user and group are entities that belong to 
> external namespace and we can't expect those to follow hive identifier syntax 
> rules. For example, a userid or group can contain '-' which is not allowed by 
> compiler.
> The patch is to allow string literal for user and group names.
> The quoted identifier support perhaps can be made to work with this. However 
> IMO this syntax should be supported regardless of quoted identifier support 
> (which is an optional configuration)
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 25cd3a5 
>   ql/src/test/queries/clientpositive/authorization_non_id.q PRE-CREATION 
>   ql/src/test/results/clientpositive/authorization_non_id.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/25595/diff/
> 
> 
> Testing
> ---
> 
> Added test case to verify various auth DDLs with new syntax.
> 
> 
> Thanks,
> 
> Prasad Mujumdar
> 
>

[jira] [Commented] (HIVE-7508) Kerberos support for streaming

[Roshan Naik (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136516#comment-14136516
 ] 

Roshan Naik commented on HIVE-7508:
---

[~leftylev] FYI... I have updated the wiki 

> Kerberos support for streaming
> --
>
> Key: HIVE-7508
> URL: https://issues.apache.org/jira/browse/HIVE-7508
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.13.1
>Reporter: Roshan Naik
>Assignee: Roshan Naik
>  Labels: Streaming, TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-7508.patch
>
>
> Add kerberos support for streaming to secure Hive cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8140) Remove obsolete code from SparkWork [Spark Branch]

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136515#comment-14136515
 ] 

Hive QA commented on HIVE-8140:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12669226/HIVE-8140.1-spark.patch

{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 6437 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/132/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/132/console
Test logs: 
http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-132/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12669226

> Remove obsolete code from SparkWork [Spark Branch]
> --
>
> Key: HIVE-8140
> URL: https://issues.apache.org/jira/browse/HIVE-8140
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Chao
>  Labels: Spark-M1
> Attachments: HIVE-8140.1-spark.patch
>
>
> There are old code in SparkWork about get/set map/reduce work. It's from POC 
> code, which isn't applicable any more. We should remove it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7647) Beeline does not honor --headerInterval and --color when executing with "-e"

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136496#comment-14136496
 ] 

Xuefu Zhang commented on HIVE-7647:
---

+1

> Beeline does not honor --headerInterval and --color when executing with "-e"
> 
>
> Key: HIVE-7647
> URL: https://issues.apache.org/jira/browse/HIVE-7647
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 0.14.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Minor
> Fix For: 0.14.0
>
> Attachments: HIVE-7647.1.patch
>
>
> --showHeader is being honored
> [root@localhost ~]# beeline --showHeader=false -u 
> 'jdbc:hive2://localhost:1/default' -n hive -d 
> org.apache.hive.jdbc.HiveDriver -e "select * from sample_07 limit 10;"
> Connecting to jdbc:hive2://localhost:1/default
> Connected to: Apache Hive (version 0.12.0-cdh5.0.1)
> Driver: Hive JDBC (version 0.12.0-cdh5.0.1)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> -hiveconf (No such file or directory)
> +--+--++-+
> | 00-  | All Occupations  | 135185230  | 42270   |
> | 11-  | Management occupations   | 6152650| 100310  |
> | 11-1011  | Chief executives | 301930 | 160440  |
> | 11-1021  | General and operations managers  | 1697690| 107970  |
> | 11-1031  | Legislators  | 64650  | 37980   |
> | 11-2011  | Advertising and promotions managers  | 36100  | 94720   |
> | 11-2021  | Marketing managers   | 166790 | 118160  |
> | 11-2022  | Sales managers   | 333910 | 110390  |
> | 11-2031  | Public relations managers| 51730  | 101220  |
> | 11-3011  | Administrative services managers | 246930 | 79500   |
> +--+--++-+
> 10 rows selected (0.838 seconds)
> Beeline version 0.12.0-cdh5.1.0 by Apache Hive
> Closing: org.apache.hive.jdbc.HiveConnection
> --outputFormat is being honored.
> [root@localhost ~]# beeline --outputFormat=csv -u 
> 'jdbc:hive2://localhost:1/default' -n hive -d 
> org.apache.hive.jdbc.HiveDriver -e "select * from sample_07 limit 10;"
> Connecting to jdbc:hive2://localhost:1/default
> Connected to: Apache Hive (version 0.12.0-cdh5.0.1)
> Driver: Hive JDBC (version 0.12.0-cdh5.0.1)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> 'code','description','total_emp','salary'
> '00-','All Occupations','135185230','42270'
> '11-','Management occupations','6152650','100310'
> '11-1011','Chief executives','301930','160440'
> '11-1021','General and operations managers','1697690','107970'
> '11-1031','Legislators','64650','37980'
> '11-2011','Advertising and promotions managers','36100','94720'
> '11-2021','Marketing managers','166790','118160'
> '11-2022','Sales managers','333910','110390'
> '11-2031','Public relations managers','51730','101220'
> '11-3011','Administrative services managers','246930','79500'
> 10 rows selected (0.664 seconds)
> Beeline version 0.12.0-cdh5.1.0 by Apache Hive
> Closing: org.apache.hive.jdbc.HiveConnection
> both --color & --headerInterval are being honored when executing using "-f" 
> option (reads query from a file rather than the commandline) (cannot really 
> see the color here but use the terminal colors)
> [root@localhost ~]# beeline --showheader=true --color=true --headerInterval=5 
> -u 'jdbc:hive2://localhost:1/default' -n hive -d 
> org.apache.hive.jdbc.HiveDriver -f /tmp/tmp.sql  
> Connecting to jdbc:hive2://localhost:1/default
> Connected to: Apache Hive (version 0.12.0-cdh5.0.1)
> Driver: Hive JDBC (version 0.12.0-cdh5.0.1)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 0.12.0-cdh5.1.0 by Apache Hive
> 0: jdbc:hive2://localhost> select * from sample_07 limit 8;
> +--+--++-+
> |   code   | description  | total_emp  | salary  |
> +--+--++-+
> | 00-  | All Occupations  | 135185230  | 42270   |
> | 11-  | Management occupations   | 6152650| 100310  |
> | 11-1011  | Chief executives | 301930 | 160440  |
> | 11-1021  | General and operations managers  | 1697690| 107970  |
> | 11-1031  | Legislators  | 64650  | 37980   |
> +--+--++-+
> |   code   | description  | total_emp  | salary  |
> +--+--+-

[jira] [Created] (HIVE-8153) Reduce the verbosity of debug logs in ORC record reader

[Prasanth J (JIRA)]

Prasanth J created HIVE-8153:


 Summary: Reduce the verbosity of debug logs in ORC record reader
 Key: HIVE-8153
 URL: https://issues.apache.org/jira/browse/HIVE-8153
 Project: Hive
  Issue Type: Bug
  Components: File Formats
Affects Versions: 0.14.0
Reporter: Prasanth J
Assignee: Prasanth J
Priority: Trivial


Following fields are logged for every row.
{code}
if (LOG.isDebugEnabled()) {
  LOG.debug("row from " + reader.path);
  LOG.debug("orc row = " + result);
}
{code}

This should be moved to trace logging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8148) HDFS Path named with file:// instead of file:/// results in Unit test failures in Windows

[Hari Sankar Sivarama Subramaniyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-8148:

Attachment: HIVE-8148.1.patch

> HDFS Path named with file:// instead of file:/// results in Unit test 
> failures in Windows
> -
>
> Key: HIVE-8148
> URL: https://issues.apache.org/jira/browse/HIVE-8148
> Project: Hive
>  Issue Type: Bug
> Environment: Windows
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-8148.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8148) HDFS Path named with file:// instead of file:/// results in Unit test failures in Windows

[Hari Sankar Sivarama Subramaniyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-8148:

Attachment: (was: HIVE-xxx.1.patch)

> HDFS Path named with file:// instead of file:/// results in Unit test 
> failures in Windows
> -
>
> Key: HIVE-8148
> URL: https://issues.apache.org/jira/browse/HIVE-8148
> Project: Hive
>  Issue Type: Bug
> Environment: Windows
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-8148.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8113) Derby server fails to start on windows

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136420#comment-14136420
 ] 

Sushanth Sowmyan commented on HIVE-8113:


+1, Looks good to me.

> Derby server fails to start on windows
> --
>
> Key: HIVE-8113
> URL: https://issues.apache.org/jira/browse/HIVE-8113
> Project: Hive
>  Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-8113.1.patch
>
>
> %HIVE_HOME%\lib\derby-10.10.1.1.jar
> doesn't contain the main class 
> org.apache.derby.drda.NetworkServerControl
> referenced in
> %HIVE_HOME%\bin\derbyserver.cmd



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8151) Dynamic partition sort optimization inserts record wrongly to partition when used with GroupBy

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-8151:
-
Attachment: HIVE-8151.1.patch

> Dynamic partition sort optimization inserts record wrongly to partition when 
> used with GroupBy
> --
>
> Key: HIVE-8151
> URL: https://issues.apache.org/jira/browse/HIVE-8151
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0, 0.13.1
>Reporter: Prasanth J
>Assignee: Prasanth J
>Priority: Critical
> Attachments: HIVE-8151.1.patch
>
>
> HIVE-6455 added dynamic partition sort optimization. It added startGroup() 
> method to FileSink operator to look for changes in reduce key for creating 
> partition directories. This method however is reliable as the key called with 
> startGroup() is different from the key called with processOp(). startGroup() 
> is called with newly changed key whereas processOp() is called with 
> previously aggregated key. This will result in processOp() writing the last 
> row of previous group as the first row of next group. This happens only when 
> used with group by operator.
> The fix is to not rely on startGroup() and do the partition directory 
> creation in processOp() itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8150) [CBO] Type coercion in union queries

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-8150:
---
Attachment: HIVE-8150.cbo.patch

> [CBO] Type coercion in union queries
> 
>
> Key: HIVE-8150
> URL: https://issues.apache.org/jira/browse/HIVE-8150
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-8150.cbo.patch
>
>
> If we can't get common type from Optiq, bail out for now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8152) Update with expression in set fails

[Alan Gates (JIRA)]

Alan Gates created HIVE-8152:


 Summary: Update with expression in set fails
 Key: HIVE-8152
 URL: https://issues.apache.org/jira/browse/HIVE-8152
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.14.0
Reporter: Alan Gates
Assignee: Eugene Koifman
Priority: Critical


Doing: {{update orctabsmall set age = age + 1;}} results in {{FAILED: 
ParseException line 1:33 missing EOF at '+' near 'age'}}

We should be able to handle simple expressions in set clauses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8151) Dynamic partition sort optimization inserts record wrongly to partition when used with GroupBy

[Prasanth J (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136396#comment-14136396
 ] 

Prasanth J commented on HIVE-8151:
--

The vectorization test case in dynpart_sort_optimization2.q must be revisited 
as HIVE-7557 disabled VectorFileSinkOperator. The proper fix for HIVE-7557 
should make sure that VectorFS should inherit mostly from FS operator. Current 
code in trunk is stale as VectorFS contains old code from FS which went through 
many changes recently. cc/ [~mmccline]

> Dynamic partition sort optimization inserts record wrongly to partition when 
> used with GroupBy
> --
>
> Key: HIVE-8151
> URL: https://issues.apache.org/jira/browse/HIVE-8151
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0, 0.13.1
>Reporter: Prasanth J
>Assignee: Prasanth J
>Priority: Critical
> Attachments: HIVE-8151.1.patch
>
>
> HIVE-6455 added dynamic partition sort optimization. It added startGroup() 
> method to FileSink operator to look for changes in reduce key for creating 
> partition directories. This method however is reliable as the key called with 
> startGroup() is different from the key called with processOp(). startGroup() 
> is called with newly changed key whereas processOp() is called with 
> previously aggregated key. This will result in processOp() writing the last 
> row of previous group as the first row of next group. This happens only when 
> used with group by operator.
> The fix is to not rely on startGroup() and do the partition directory 
> creation in processOp() itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136388#comment-14136388
 ] 

Sergey Shelukhin commented on HIVE-8111:


Tagging doesn't really seem possible unless Rex* stuff is modified (for 
example, to add shim CAST distinguishable from cast, which is very hacky, or 
add some identity to those, as object identity of RexNode-s doesn't appear to 
be preserved). So I am proceeding to hacky territory :)

> CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
> 
>
> Key: HIVE-8111
> URL: https://issues.apache.org/jira/browse/HIVE-8111
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>
> Original test failure: looks like column type changes to different decimals 
> in most cases. In one case it causes the integer part to be too big to fit, 
> so the result becomes null it seems.
> What happens is that CBO adds casts to arithmetic expressions to make them 
> type compatible; these casts become part of new AST, and then Hive adds casts 
> on top of these casts. This (the first part) also causes lots of out file 
> changes. It's not clear how to best fix it so far, in addition to incorrect 
> decimal width and sometimes nulls when width is larger than allowed in Hive.
> Option one - don't add those for numeric ops - cannot be done if numeric op 
> is a part of compare, for which CBO needs correct types.
> Option two - unwrap casts when determining type in Hive - hard or impossible 
> to tell apart CBO-added casts and user casts. 
> Option three - don't change types in Hive if CBO has run - seems hacky and 
> hard to ensure it's applied everywhere.
> Option four - map all expressions precisely between two trees and remove 
> casts again after optimization, will be pretty difficult.
> Option five - somehow mark those casts. Not sure about how yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8151) Dynamic partition sort optimization inserts record wrongly to partition when used with GroupBy

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-8151:
-
Status: Patch Available  (was: Open)

> Dynamic partition sort optimization inserts record wrongly to partition when 
> used with GroupBy
> --
>
> Key: HIVE-8151
> URL: https://issues.apache.org/jira/browse/HIVE-8151
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.1, 0.14.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>Priority: Critical
> Attachments: HIVE-8151.1.patch
>
>
> HIVE-6455 added dynamic partition sort optimization. It added startGroup() 
> method to FileSink operator to look for changes in reduce key for creating 
> partition directories. This method however is reliable as the key called with 
> startGroup() is different from the key called with processOp(). startGroup() 
> is called with newly changed key whereas processOp() is called with 
> previously aggregated key. This will result in processOp() writing the last 
> row of previous group as the first row of next group. This happens only when 
> used with group by operator.
> The fix is to not rely on startGroup() and do the partition directory 
> creation in processOp() itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 25716: Type coercion for union queries.

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25716/#review53614
---

Ship it!


Ship It!

- John Pullokkaran


On Sept. 16, 2014, 10:29 p.m., Ashutosh Chauhan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25716/
> ---
> 
> (Updated Sept. 16, 2014, 10:29 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-8150
> https://issues.apache.org/jira/browse/HIVE-8150
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Type coercion for union queries.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 395b5e6 
> 
> Diff: https://reviews.apache.org/r/25716/diff/
> 
> 
> Testing
> ---
> 
> union32.q
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>

[jira] [Updated] (HIVE-8150) [CBO] Type coercion in union queries

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-8150:
---
Status: Patch Available  (was: Open)

> [CBO] Type coercion in union queries
> 
>
> Key: HIVE-8150
> URL: https://issues.apache.org/jira/browse/HIVE-8150
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-8150.cbo.patch
>
>
> If we can't get common type from Optiq, bail out for now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 25716: Type coercion for union queries.

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25716/
---

Review request for hive and John Pullokkaran.


Bugs: HIVE-8150
https://issues.apache.org/jira/browse/HIVE-8150


Repository: hive-git


Description
---

Type coercion for union queries.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 395b5e6 

Diff: https://reviews.apache.org/r/25716/diff/


Testing
---

union32.q


Thanks,

Ashutosh Chauhan

[jira] [Created] (HIVE-8151) Dynamic partition sort optimization inserts record wrongly to partition when used with GroupBy

[Prasanth J (JIRA)]

Prasanth J created HIVE-8151:


 Summary: Dynamic partition sort optimization inserts record 
wrongly to partition when used with GroupBy
 Key: HIVE-8151
 URL: https://issues.apache.org/jira/browse/HIVE-8151
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.13.1, 0.14.0
Reporter: Prasanth J
Assignee: Prasanth J
Priority: Critical


HIVE-6455 added dynamic partition sort optimization. It added startGroup() 
method to FileSink operator to look for changes in reduce key for creating 
partition directories. This method however is reliable as the key called with 
startGroup() is different from the key called with processOp(). startGroup() is 
called with newly changed key whereas processOp() is called with previously 
aggregated key. This will result in processOp() writing the last row of 
previous group as the first row of next group. This happens only when used with 
group by operator.

The fix is to not rely on startGroup() and do the partition directory creation 
in processOp() itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7777) Add CSV Serde based on OpenCSV

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-:
---
Summary: Add CSV Serde based on OpenCSV  (was: add CSV support for Serde)

Attribution for this fix goes to  Larry Ogrodnek and Ferdinand Xu.

Thank you! I have committed this to trunk!

> Add CSV Serde based on OpenCSV
> --
>
> Key: HIVE-
> URL: https://issues.apache.org/jira/browse/HIVE-
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
> Fix For: 0.14.0
>
> Attachments: HIVE-.1.patch, HIVE-.2.patch, HIVE-.3.patch, 
> HIVE-.patch, csv-serde-master.zip
>
>
> There is no official support for csvSerde for hive while there is an open 
> source project in github(https://github.com/ogrodnek/csv-serde). CSV is of 
> high frequency in use as a data format.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7777) add CSV support for Serde

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-:
---
   Resolution: Fixed
Fix Version/s: 0.14.0
 Release Note: A CSVSerde based on OpenCSV has been added. This Serde works 
for most CSV data, but does not handled embedded newlines.
   Status: Resolved  (was: Patch Available)

> add CSV support for Serde
> -
>
> Key: HIVE-
> URL: https://issues.apache.org/jira/browse/HIVE-
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
> Fix For: 0.14.0
>
> Attachments: HIVE-.1.patch, HIVE-.2.patch, HIVE-.3.patch, 
> HIVE-.patch, csv-serde-master.zip
>
>
> There is no official support for csvSerde for hive while there is an open 
> source project in github(https://github.com/ogrodnek/csv-serde). CSV is of 
> high frequency in use as a data format.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8140) Remove obsolete code from SparkWork [Spark Branch]

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-8140:
--
Status: Patch Available  (was: Open)

> Remove obsolete code from SparkWork [Spark Branch]
> --
>
> Key: HIVE-8140
> URL: https://issues.apache.org/jira/browse/HIVE-8140
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Chao
>  Labels: Spark-M1
> Attachments: HIVE-8140.1-spark.patch
>
>
> There are old code in SparkWork about get/set map/reduce work. It's from POC 
> code, which isn't applicable any more. We should remove it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8150) [CBO] Type coercion in union queries

[Ashutosh Chauhan (JIRA)]

Ashutosh Chauhan created HIVE-8150:
--

 Summary: [CBO] Type coercion in union queries
 Key: HIVE-8150
 URL: https://issues.apache.org/jira/browse/HIVE-8150
 Project: Hive
  Issue Type: Bug
  Components: CBO
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


If we can't get common type from Optiq, bail out for now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7946) CBO: Merge CBO changes to Trunk

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136371#comment-14136371
 ] 

Hive QA commented on HIVE-7946:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12668977/HIVE-7946.9.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/828/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/828/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-828/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-828/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target 
shims/0.20S/target shims/0.23/target shims/aggregator/target 
shims/common/target shims/common-secure/target packaging/target 
hbase-handler/target testutils/target jdbc/target metastore/target 
itests/target itests/hcatalog-unit/target itests/test-serde/target 
itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target 
itests/hive-unit/target itests/custom-serde/target itests/util/target 
hcatalog/target hcatalog/core/target hcatalog/streaming/target 
hcatalog/server-extensions/target hcatalog/hcatalog-pig-adapter/target 
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target 
accumulo-handler/target hwi/target common/target common/src/gen contrib/target 
service/target serde/target 
serde/src/test/org/apache/hadoop/hive/serde2/io/TestDateWritable.java 
beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
+ svn update
Uql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIndex.java
Uql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
Uql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
Uql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
Uql/src/test/org/apache/hadoop/hive/ql/exec/TestFunctionRegistry.java
Aql/src/test/queries/clientpositive/array_map_access_nonconstant.q
Dql/src/test/queries/negative/invalid_list_index2.q
Dql/src/test/queries/negative/invalid_map_index2.q
Dql/src/test/queries/negative/invalid_list_index.q
Dql/src/test/queries/negative/invalid_map_index.q
Aql/src/test/results/clientpositive/array_map_access_nonconstant.q.out
Dql/src/test/results/compiler/errors/invalid_map_index2.q.out
Dql/src/test/results/compiler/errors/invalid_map_index.q.out
Dql/src/test/results/compiler/errors/invalid_list_index2.q.out
Dql/src/test/results/compiler/errors/invalid_list_index.q.out

Fetching external item into 'hcatalog/src/test/e2e/harness'
Updated external to revision 1625420.

Updated to revision 1625420.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is autom

[jira] [Commented] (HIVE-8102) Partitions of type 'date' behave incorrectly with daylight saving time.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136369#comment-14136369
 ] 

Hive QA commented on HIVE-8102:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12669189/HIVE-8102.2.patch

{color:green}SUCCESS:{color} +1 6279 tests passed

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/827/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/827/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-827/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12669189

> Partitions of type 'date' behave incorrectly with daylight saving time.
> ---
>
> Key: HIVE-8102
> URL: https://issues.apache.org/jira/browse/HIVE-8102
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema, Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Eli Acherkan
> Attachments: HIVE-8102.1.patch, HIVE-8102.2.patch
>
>
> On 2AM on March 28th 2014, Israel went from standard time (GMT+2) to daylight 
> saving time (GMT+3).
> The server's timezone is Asia/Jerusalem. When creating a partition whose key 
> is 2014-03-28, Hive creates a partition for 2013-03-27 instead:
> hive (default)> create table test (a int) partitioned by (`b_prt` date);
> OK
> Time taken: 0.092 seconds
> hive (default)> alter table test add partition (b_prt='2014-03-28');
> OK
> Time taken: 0.187 seconds
> hive (default)> show partitions test;   
> OK
> partition
> b_prt=2014-03-27
> Time taken: 0.134 seconds, Fetched: 1 row(s)
> It seems that the root cause is the behavior of 
> DateWritable.daysToMillis/dateToDays.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 25178: HIVE-7100 Add DROP TABLE PURGE

[Xuefu Zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25178/#review53612
---



metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java


Maybe I'm misunderstanding. the added dropTableData(Path, boolean) is 
called by some public API, such as drop_table_with_environment_context(), but 
we don't need to change the API. Instead, before calling dropTableData(), we 
just need to get a boolean value (ifPurge) from the context. The signature of 
the API doesn't need to change.


- Xuefu Zhang


On Sept. 12, 2014, 9:51 p.m., david seraf wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25178/
> ---
> 
> (Updated Sept. 12, 2014, 9:51 p.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-7100
> https://issues.apache.org/jira/browse/HIVE-7100
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add PURGE option to DROP TABLE command to skip saving table data to the trash
> 
> 
> Diffs
> -
> 
>   
> hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatPartitionPublish.java
>  be7134f 
>   
> hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/tool/TestTempletonUtils.java
>  af952f2 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHiveServer2.java
>  da51a55 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> 9489949 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
> a94a7a3 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreFsImpl.java 
> cff0718 
>   metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
> cbdba30 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreFS.java 
> a141793 
>   metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java 613b709 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java cd017d8 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java e387b8f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  4cf98d8 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> f31a409 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 32db0c7 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DropTableDesc.java ba30e1f 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 406aae9 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveRemote.java 1a5ba87 
>   ql/src/test/queries/clientpositive/drop_table_purge.q PRE-CREATION 
>   ql/src/test/results/clientpositive/drop_table_purge.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/25178/diff/
> 
> 
> Testing
> ---
> 
> added code test and added QL test.  Tests passed in CI, but other, unrelated 
> tests failed.
> 
> 
> Thanks,
> 
> david seraf
> 
>

Re: Review Request 25711: Create root scratch dir with 733 instead of 777 perms

[Vaibhav Gumashta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25711/
---

(Updated Sept. 16, 2014, 10:20 p.m.)


Review request for hive, Gopal V and Thejas Nair.


Bugs: HIVE-8143
https://issues.apache.org/jira/browse/HIVE-8143


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-8143


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 185ea95 
  itests/hive-unit/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 
b430cae 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
241d315 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java d7227ed 

Diff: https://reviews.apache.org/r/25711/diff/


Testing
---

Unit + manual


Thanks,

Vaibhav Gumashta

[jira] [Updated] (HIVE-8143) Create root scratch dir with 733 instead of 777 perms

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-8143:
---
Attachment: HIVE-8143.2.patch

> Create root scratch dir with 733 instead of 777 perms
> -
>
> Key: HIVE-8143
> URL: https://issues.apache.org/jira/browse/HIVE-8143
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 0.14.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 0.14.0
>
> Attachments: HIVE-8143.1.patch, HIVE-8143.2.patch
>
>
> hive.exec.scratchdir which is treated as the root scratch directory on hdfs 
> only needs to be writable by all. We can use 733 instead of 777 for doing 
> that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8148) HDFS Path named with file:// instead of file:/// results in Unit test failures in Windows

[Hari Sankar Sivarama Subramaniyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-8148:

Status: Patch Available  (was: Open)

> HDFS Path named with file:// instead of file:/// results in Unit test 
> failures in Windows
> -
>
> Key: HIVE-8148
> URL: https://issues.apache.org/jira/browse/HIVE-8148
> Project: Hive
>  Issue Type: Bug
> Environment: Windows
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-xxx.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8149) hive.optimize.reducededuplication should be set to false for IUD ops

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-8149:


 Summary: hive.optimize.reducededuplication should be set to false 
for IUD ops
 Key: HIVE-8149
 URL: https://issues.apache.org/jira/browse/HIVE-8149
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.14.0
Reporter: Eugene Koifman
Assignee: Alan Gates


this optimizer causes both old and new rows to show up in a select after update 
(for tables involving few rows)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8113) Derby server fails to start on windows

[Hari Sankar Sivarama Subramaniyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-8113:

Attachment: HIVE-8113.1.patch

cc-ing [~sushanth] for review

> Derby server fails to start on windows
> --
>
> Key: HIVE-8113
> URL: https://issues.apache.org/jira/browse/HIVE-8113
> Project: Hive
>  Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-8113.1.patch
>
>
> %HIVE_HOME%\lib\derby-10.10.1.1.jar
> doesn't contain the main class 
> org.apache.derby.drda.NetworkServerControl
> referenced in
> %HIVE_HOME%\bin\derbyserver.cmd



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8113) Derby server fails to start on windows

[Hari Sankar Sivarama Subramaniyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-8113:

Status: Patch Available  (was: Open)

> Derby server fails to start on windows
> --
>
> Key: HIVE-8113
> URL: https://issues.apache.org/jira/browse/HIVE-8113
> Project: Hive
>  Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-8113.1.patch
>
>
> %HIVE_HOME%\lib\derby-10.10.1.1.jar
> doesn't contain the main class 
> org.apache.derby.drda.NetworkServerControl
> referenced in
> %HIVE_HOME%\bin\derbyserver.cmd



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8074) Merge trunk into spark 9/12/2014

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-8074:
---
Summary: Merge trunk into spark 9/12/2014  (was: Merge spark into trunk 
9/12/2014)

I had my language backwards in the title. This was merging trunk into spark, 
not spark into trunk.

> Merge trunk into spark 9/12/2014
> 
>
> Key: HIVE-8074
> URL: https://issues.apache.org/jira/browse/HIVE-8074
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Affects Versions: spark-branch
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: spark-branch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HIVE-8032) Fix TestSparkCliDriver => optimize_nullscan.q

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland resolved HIVE-8032.

Resolution: Duplicate

Fixed in HIVE-8074.

> Fix TestSparkCliDriver => optimize_nullscan.q
> -
>
> Key: HIVE-8032
> URL: https://issues.apache.org/jira/browse/HIVE-8032
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Brock Noland
>Assignee: Brock Noland
>
> It's been failing lately, perhaps since the last merge from trunk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8074) Merge spark into trunk 9/12/2014

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136352#comment-14136352
 ] 

Brock Noland commented on HIVE-8074:


Fixed :)

> Merge spark into trunk 9/12/2014
> 
>
> Key: HIVE-8074
> URL: https://issues.apache.org/jira/browse/HIVE-8074
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Affects Versions: spark-branch
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: spark-branch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7699) HIVE fails to LOAD DATA INTO TABLE from external storage

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HIVE-7699:

Affects Version/s: 0.13.0

> HIVE fails to LOAD DATA  INTO TABLE from external storage
> -
>
> Key: HIVE-7699
> URL: https://issues.apache.org/jira/browse/HIVE-7699
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>
> Hive is supposed to support loading data to table from external storage like 
> Azure blob storage. Here's an exampLe:
> 
>fs.azure.account.key.XXX.blob.core.windows.net
>CREDENTIAL-KEY
>  is configured into hadoop core-site.xml. bin/hadoop fs works, 
> however if run the query like:
> LOAD DATA INPATH 'wasb://t...@account-name.blob.core.windows.net/data.txt' 
> OVERWRITE INTO TABLE hivetable;
> HIVE CLI will complain as
> FAILED: SemanticException [Error 10028]: Line 1:17 Path is not legal 
> ''wasb://t...@account-name.blob.core.windows.net/data.t
> xt'': Move from: wasb://t...@account-name.blob.core.windows.net/data.txt to: 
> hdfs://HDFS-HOST:8020/hivetable is not vali
> d. Please check that values for params "default.fs.name" and 
> "hive.metastore.warehouse.dir" do not conflict.
> The stacktrace is 
>   at 
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:170)
>   at 
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:236)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:427)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:323)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:980)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1045)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:916)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:906)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:272)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:224)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:427)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:363)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:460)
>   at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:489)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:772)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:709)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:648)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:212)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8074) Merge spark into trunk 9/12/2014

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-8074:
---
   Resolution: Fixed
Fix Version/s: spark-branch
   Status: Resolved  (was: Patch Available)

> Merge spark into trunk 9/12/2014
> 
>
> Key: HIVE-8074
> URL: https://issues.apache.org/jira/browse/HIVE-8074
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Affects Versions: spark-branch
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: spark-branch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7790) Update privileges to check for update and delete

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-7790:
-
Status: Patch Available  (was: Open)

> Update privileges to check for update and delete
> 
>
> Key: HIVE-7790
> URL: https://issues.apache.org/jira/browse/HIVE-7790
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-7790.2.patch, HIVE-7790.3.patch, HIVE-7790.patch
>
>
> In the new SQLStdAuth scheme, we need to add UPDATE and DELETE as operations 
> and add ability check for them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7511) Hive: output is incorrect if there are UTF-8 characters in where clause of a hive select query.

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HIVE-7511:

Fix Version/s: (was: 0.14.0)

> Hive: output is incorrect if there are UTF-8 characters in where clause of a 
> hive select query.
> ---
>
> Key: HIVE-7511
> URL: https://issues.apache.org/jira/browse/HIVE-7511
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
> Environment: Windows Server 2008 R2
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>Priority: Critical
> Attachments: HIVE-7511.1.patch
>
>
> When we put UTF-8 characters in where clause of a hive query the results are 
> empty for "where content like '%丄%'" and results contain all rows for "where 
> content not like '%丄%';" even when few rows contain this character.
> Steps to reproduce:
> 1. Save a file called data.txt in the root container. The contents of the 
> files are as follows.
> 190   丄f齄啊c狛䶴h䶴c狝
> 899   d狜狜㐁geg阿狚ea䶴eead狜e
> 137   齄鼾h狝ge㐀狛g狚阿
> 21﨩﨩e㐀c狛鼾d䶴﨨
> 767   﨩c﨩g狜㐁狜狛齄阿﨩狚齄﨨䶵狝﨨
> 281   﨨㐀啊aga啊c狝e鼾鼾
> 573   㐁䶴hc﨨b狝㐁﨩䶴狜丄hc齄
> 966   䶴丄狜﨨e狝eb狜㐁c㐀鼾﨩丄ga狚丄
> 565   䶵㐀﨩㐀bb狛ehd丄ea丄㐀
> 778   﨩㐁阿﨨狚bbea丄䶵丄狚鼾狚a䶵
> 363   gd齄a鼾a䶴b㐁㐁fg鼾
> 822   a阿狜䶵h䶵e狛h﨩gac狜阿㐀啊b
> 338   b齄㐁ff阿e狜e㐀ba齄
> 2. Execute the following queries to setup the table.
> a. CREATE TABLE hivetable(row INT, content STRING) ROW FORMAT DELIMITED 
> FIELDS TERMINATED BY '
> t' LOCATION '/hivetable';
> b. LOAD DATA INPATH 'wasb:///data.txt' OVERWRITE INTO TABLE hivetable;
> 3. create a query file query.hql with following contents
> INSERT OVERWRITE DIRECTORY 'wasb:///hiveoutput'
> select * from hivetable where content like '%丄%';
> 4. even though few rows contains this character the output is empty.
> 5. change the contents of query.hql to 
> INSERT OVERWRITE DIRECTORY 'wasb:///hiveoutput'
> select * from hivetable where content not like '%丄%';
> 6. The output contains all rows including those containing the given 
> character.
> 7. Similar results are observed when using "where content = '丄f齄啊c狛䶴h䶴c狝'; "
> 8. We get expected results when using "where content like '%a%'; "



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8147) Fail to LOAD DATA INTO table after ALTER TABLE..SET LOCATION

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HIVE-8147:

Affects Version/s: (was: 0.13.0)
   0.14.0

> Fail to LOAD DATA INTO table after ALTER TABLE..SET LOCATION
> 
>
> Key: HIVE-8147
> URL: https://issues.apache.org/jira/browse/HIVE-8147
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: table.dat
>
>
> Fail to LOAD DATA INTO table after ALTER TABLE..SET LOCATION. 
> Repro steps:
> {noformat}
> 1. hadoop fs -put table.dat /data/table.dat
> 2. drop table if exists hcat_altertable_16;
> 3. create table hcat_altertable_16(a int, b string) ROW FORMAT DELIMITED 
> FIELDS TERMINATED BY '\t' stored as textfile;
> 4. LOAD DATA INPATH '/data/table.dat' OVERWRITE INTO TABLE hcat_altertable_16;
> 5. hadoop fs -put table.dat /data/table.dat
> 6. hadoop fs -mkdir /tmp/table_has_moved
> 7. alter table hcat_altertable_16 set location 'hdfs:///tmp/table_has_moved/';
> 8. LOAD DATA INPATH '/data/table.dat' OVERWRITE INTO TABLE 
> hcat_altertable_16; 
> {noformat}
> Step 8 will complain:
> {noformat}
> FAILED: SemanticException [Error 10028]: Line 1:17 Path is not legal 
> ''/data/table.dat'': Move from: hdfs://hhuang-clst21:8020/data/table.dat to: 
> hdfs:/tmp/table_has_moved is not valid. Please check that values for params 
> "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> {noformat}
> Stack trace:
> {noformat}
> 2014-09-16 21:37:45,020 ERROR ql.Driver (SessionState.java:printError(824)) - 
> FAILED: SemanticException [Error 10028]: Line 1:17 Path is not legal 
> ''/data/bug-22915'': Move from: hdfs://hhuang-clst21:8020/data/bug-22915 to: 
> hdfs:/tmp/table_has_moved is not valid. Please check that values for params 
> "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:17 Path is not 
> legal ''/data/bug-22915'': Move from: 
> hdfs://hhuang-clst21:8020/data/bug-22915 to: hdfs:/tmp/table_has_moved is not 
> valid. Please check that values for params "default.fs.name" and 
> "hive.metastore.warehouse.dir" do not conflict.
>   at 
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:169)
>   at 
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:235)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:217)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:407)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1058)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1128)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:995)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:985)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:246)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:198)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-5764) Stopping Metastore and HiveServer2 from command line

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HIVE-5764:

Fix Version/s: (was: 0.14.0)

> Stopping Metastore and HiveServer2 from command line
> 
>
> Key: HIVE-5764
> URL: https://issues.apache.org/jira/browse/HIVE-5764
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Metastore
>Reporter: Vaibhav Gumashta
>Assignee: Xiaobing Zhou
>  Labels: patch
> Attachments: HIVE-5764.1.patch
>
>
> Currently a user needs to kill the process. Ideally there should be something 
> like:
> hive --service metastore stop
> hive --service hiveserver2 stop



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7875) Hive cannot load data into partitioned table with Unicode key

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HIVE-7875:

Fix Version/s: (was: 0.14.0)

> Hive cannot load data into partitioned table with Unicode key
> -
>
> Key: HIVE-7875
> URL: https://issues.apache.org/jira/browse/HIVE-7875
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
> Environment: Windows Server 2008
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: CreatePartitionedTable.hql, 
> LoadIntoPartitionedTable.hql, partitioned.txt
>
>
> Steps to reproduce:
> 1) Copy the file partitioned.txt to the root folder HDFS. Copy the two hql 
> files to your local directory.
> 2) Open Hive CLI.
> 3) Run:
> hive> source ;
> 4) Run
> hive> source ;
> The following error will be shown:
> hive> source C:\Scripts\partition\LoadIntoPartitionedTable.hql;
> Loading data to table default.mypartitioned partition (tag=䶵)
> Failed with exception null
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.MoveTask



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)