[jira] [Commented] (HIVE-15056) Support index shifting for struct fields

2016-10-31 Thread Lefty Leverenz (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624476#comment-15624476
 ] 

Lefty Leverenz commented on HIVE-15056:
---

[~csun], this issue has fix version 2.1.1 but the commit went to master 
(2.2.0), not branch-2.1.  Will it also be committed to branch-2.1?

> Support index shifting for struct fields
> 
>
> Key: HIVE-15056
> URL: https://issues.apache.org/jira/browse/HIVE-15056
> Project: Hive
>  Issue Type: Sub-task
>  Components: File Formats, Physical Optimizer, 
> Serializers/Deserializers
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 2.1.1
>
> Attachments: HIVE-15056.1.patch, HIVE-15056.2.patch
>
>
> In HIVE-13873, the following case doesn't work:
> {code}
> select s.c from tbl
> {code}
> where {{tbl}} is of schema:
> {code}
> a int
> s struct
> {code}
> This is because currently we generate a "pruned" schema (in terms of 
> {{GroupType}}) for Parquet reader to scan the data. However, on the Hive side 
> the object inspector still uses the original schema. In particular, in this 
> case for {{s.c}} the data returned by Parquet reader is in index 0, but the 
> object inspector tries to read it in index 1. Therefore, in correct result 
> will be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15096) hplsql registerUDF conflicts with pom.xml

2016-10-31 Thread Fei Hui (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624446#comment-15624446
 ] 

Fei Hui commented on HIVE-15096:


update the patch, remove the hardcode version 2.2.0-SNAPSHOT.
can you review it again? give suggestions

> hplsql registerUDF conflicts with pom.xml
> -
>
> Key: HIVE-15096
> URL: https://issues.apache.org/jira/browse/HIVE-15096
> Project: Hive
>  Issue Type: Bug
>  Components: hpl/sql
>Affects Versions: 2.0.0, 2.1.0, 2.0.1
>Reporter: Fei Hui
> Fix For: 2.2.0
>
> Attachments: HIVE-15096.1.patch, HIVE-15096.patch
>
>
> in hplsql code, registerUDF code is
> sql.add("ADD JAR " + dir + "hplsql.jar");
> sql.add("ADD JAR " + dir + "antlr-runtime-4.5.jar");
> sql.add("ADD FILE " + dir + Conf.SITE_XML);
> but pom configufation is
>   
> org.apache.hive
> hive
> 2.2.0-SNAPSHOT
> ../pom.xml
>   
>   hive-hplsql
>   jar
>   Hive HPL/SQL
> 
>org.antlr
>antlr4-runtime
>4.5
> 
> when run hplsql , errors occur as below
>  Error while processing statement: 
> /opt/apps/apache-hive-2.0.0-bin/lib/hplsql.jar does not exist



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8890) HiveServer2 dynamic service discovery: use persistent ephemeral nodes curator recipe

2016-10-31 Thread Lefty Leverenz (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lefty Leverenz updated HIVE-8890:
-
Labels:   (was: TODOC1.2)

> HiveServer2 dynamic service discovery: use persistent ephemeral nodes curator 
> recipe
> 
>
> Key: HIVE-8890
> URL: https://issues.apache.org/jira/browse/HIVE-8890
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.14.0, 1.0.0, 1.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: HIVE-8890.1.patch, HIVE-8890.2.patch, HIVE-8890.3.patch, 
> HIVE-8890.4.patch
>
>
> Using this recipe gives better reliability.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8890) HiveServer2 dynamic service discovery: use persistent ephemeral nodes curator recipe

2016-10-31 Thread Lefty Leverenz (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624440#comment-15624440
 ] 

Lefty Leverenz commented on HIVE-8890:
--

Doc done, so I'm removing the TODOC1.2 label.

Thanks go to [~sergeant] for updating the parameter description.

> HiveServer2 dynamic service discovery: use persistent ephemeral nodes curator 
> recipe
> 
>
> Key: HIVE-8890
> URL: https://issues.apache.org/jira/browse/HIVE-8890
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.14.0, 1.0.0, 1.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: HIVE-8890.1.patch, HIVE-8890.2.patch, HIVE-8890.3.patch, 
> HIVE-8890.4.patch
>
>
> Using this recipe gives better reliability.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15096) hplsql registerUDF conflicts with pom.xml

2016-10-31 Thread Fei Hui (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fei Hui updated HIVE-15096:
---
Attachment: HIVE-15096.1.patch

> hplsql registerUDF conflicts with pom.xml
> -
>
> Key: HIVE-15096
> URL: https://issues.apache.org/jira/browse/HIVE-15096
> Project: Hive
>  Issue Type: Bug
>  Components: hpl/sql
>Affects Versions: 2.0.0, 2.1.0, 2.0.1
>Reporter: Fei Hui
> Fix For: 2.2.0
>
> Attachments: HIVE-15096.1.patch, HIVE-15096.patch
>
>
> in hplsql code, registerUDF code is
> sql.add("ADD JAR " + dir + "hplsql.jar");
> sql.add("ADD JAR " + dir + "antlr-runtime-4.5.jar");
> sql.add("ADD FILE " + dir + Conf.SITE_XML);
> but pom configufation is
>   
> org.apache.hive
> hive
> 2.2.0-SNAPSHOT
> ../pom.xml
>   
>   hive-hplsql
>   jar
>   Hive HPL/SQL
> 
>org.antlr
>antlr4-runtime
>4.5
> 
> when run hplsql , errors occur as below
>  Error while processing statement: 
> /opt/apps/apache-hive-2.0.0-bin/lib/hplsql.jar does not exist



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.

2016-10-31 Thread liyunzhang_intel (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyunzhang_intel updated HIVE-13517:

Attachment: executor-driver-log.PNG

> Hive logs in Spark Executor and Driver should show thread-id.
> -
>
> Key: HIVE-13517
> URL: https://issues.apache.org/jira/browse/HIVE-13517
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Szehon Ho
>Assignee: liyunzhang_intel
> Attachments: executor-driver-log.PNG
>
>
> In Spark, there might be more than one task running in one executor. 
> Similarly, there may be more than one thread running in Driver.
> This makes debugging through the logs a nightmare. It would be great if there 
> could be thread-ids in the logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.

2016-10-31 Thread liyunzhang_intel (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624357#comment-15624357
 ] 

liyunzhang_intel commented on HIVE-13517:
-

[~szehon]:
 currently i view the driver and executor logs by url like 
"http://xxx:18080/;(see the attached picture) and click the link to view the 
detailed logs. The content of log is like
{noformat}
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/home/hadoop26/nm-local-dir/filecache/11/spark-assembly-1.6.2-SNAPSHOT-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/home/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/10/31 23:16:58 INFO CoarseGrainedExecutorBackend: Registered signal handlers 
for [TERM, HUP, INT]
16/10/31 23:16:59 WARN NativeCodeLoader: Unable to load native-hadoop library 
for your platform... using builtin-java classes where applicable
16/10/31 23:16:59 INFO SecurityManager: Changing view acls to: root
16/10/31 23:16:59 INFO SecurityManager: Changing modify acls to: root
16/10/31 23:16:59 INFO SecurityManager: SecurityManager: authentication 
disabled; ui acls disabled; users with view permissions: Set(root); users with 
modify permissions: Set(root)
16/10/31 23:16:59 INFO SecurityManager: Changing view acls to: root
16/10/31 23:16:59 INFO SecurityManager: Changing modify acls to: root
{noformat}

So what you want is 16/10/31 23:16:59 *Thread-Name* INFO SecurityManager: 
Changing modify acls to: root? is my understanding right?


> Hive logs in Spark Executor and Driver should show thread-id.
> -
>
> Key: HIVE-13517
> URL: https://issues.apache.org/jira/browse/HIVE-13517
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Szehon Ho
>Assignee: liyunzhang_intel
>
> In Spark, there might be more than one task running in one executor. 
> Similarly, there may be more than one thread running in Driver.
> This makes debugging through the logs a nightmare. It would be great if there 
> could be thread-ids in the logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-14943) Base Implementation

2016-10-31 Thread Eugene Koifman (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-14943:
--
Attachment: HIVE-14943.7.patch

> Base Implementation
> ---
>
> Key: HIVE-14943
> URL: https://issues.apache.org/jira/browse/HIVE-14943
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-14943.2.patch, HIVE-14943.3.patch, 
> HIVE-14943.4.patch, HIVE-14943.5.patch, HIVE-14943.6.patch, 
> HIVE-14943.7.patch, HIVE-14943.patch
>
>
> Create the 1st pass functional implementation of MERGE
> This should run e2e and produce correct results.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (HIVE-15096) hplsql registerUDF conflicts with pom.xml

2016-10-31 Thread Fei Hui (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624106#comment-15624106
 ] 

Fei Hui edited comment on HIVE-15096 at 11/1/16 2:49 AM:
-

there is no hplsql.jar with hive release. users don't know how to  change this 
file if they have no source code


was (Author: ferhui):
there is no hplsql.jar with hive release. users do not change this file if they 
have no source code

> hplsql registerUDF conflicts with pom.xml
> -
>
> Key: HIVE-15096
> URL: https://issues.apache.org/jira/browse/HIVE-15096
> Project: Hive
>  Issue Type: Bug
>  Components: hpl/sql
>Affects Versions: 2.0.0, 2.1.0, 2.0.1
>Reporter: Fei Hui
> Fix For: 2.2.0
>
> Attachments: HIVE-15096.patch
>
>
> in hplsql code, registerUDF code is
> sql.add("ADD JAR " + dir + "hplsql.jar");
> sql.add("ADD JAR " + dir + "antlr-runtime-4.5.jar");
> sql.add("ADD FILE " + dir + Conf.SITE_XML);
> but pom configufation is
>   
> org.apache.hive
> hive
> 2.2.0-SNAPSHOT
> ../pom.xml
>   
>   hive-hplsql
>   jar
>   Hive HPL/SQL
> 
>org.antlr
>antlr4-runtime
>4.5
> 
> when run hplsql , errors occur as below
>  Error while processing statement: 
> /opt/apps/apache-hive-2.0.0-bin/lib/hplsql.jar does not exist



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15096) hplsql registerUDF conflicts with pom.xml

2016-10-31 Thread Fei Hui (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624107#comment-15624107
 ] 

Fei Hui commented on HIVE-15096:


there is no hplsql.jar with hive release. users do not change this file if they 
have no source code

> hplsql registerUDF conflicts with pom.xml
> -
>
> Key: HIVE-15096
> URL: https://issues.apache.org/jira/browse/HIVE-15096
> Project: Hive
>  Issue Type: Bug
>  Components: hpl/sql
>Affects Versions: 2.0.0, 2.1.0, 2.0.1
>Reporter: Fei Hui
> Fix For: 2.2.0
>
> Attachments: HIVE-15096.patch
>
>
> in hplsql code, registerUDF code is
> sql.add("ADD JAR " + dir + "hplsql.jar");
> sql.add("ADD JAR " + dir + "antlr-runtime-4.5.jar");
> sql.add("ADD FILE " + dir + Conf.SITE_XML);
> but pom configufation is
>   
> org.apache.hive
> hive
> 2.2.0-SNAPSHOT
> ../pom.xml
>   
>   hive-hplsql
>   jar
>   Hive HPL/SQL
> 
>org.antlr
>antlr4-runtime
>4.5
> 
> when run hplsql , errors occur as below
>  Error while processing statement: 
> /opt/apps/apache-hive-2.0.0-bin/lib/hplsql.jar does not exist



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Issue Comment Deleted] (HIVE-15096) hplsql registerUDF conflicts with pom.xml

2016-10-31 Thread Fei Hui (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fei Hui updated HIVE-15096:
---
Comment: was deleted

(was: there is no hplsql.jar with hive release. users do not change this file 
if they have no source code)

> hplsql registerUDF conflicts with pom.xml
> -
>
> Key: HIVE-15096
> URL: https://issues.apache.org/jira/browse/HIVE-15096
> Project: Hive
>  Issue Type: Bug
>  Components: hpl/sql
>Affects Versions: 2.0.0, 2.1.0, 2.0.1
>Reporter: Fei Hui
> Fix For: 2.2.0
>
> Attachments: HIVE-15096.patch
>
>
> in hplsql code, registerUDF code is
> sql.add("ADD JAR " + dir + "hplsql.jar");
> sql.add("ADD JAR " + dir + "antlr-runtime-4.5.jar");
> sql.add("ADD FILE " + dir + Conf.SITE_XML);
> but pom configufation is
>   
> org.apache.hive
> hive
> 2.2.0-SNAPSHOT
> ../pom.xml
>   
>   hive-hplsql
>   jar
>   Hive HPL/SQL
> 
>org.antlr
>antlr4-runtime
>4.5
> 
> when run hplsql , errors occur as below
>  Error while processing statement: 
> /opt/apps/apache-hive-2.0.0-bin/lib/hplsql.jar does not exist



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15096) hplsql registerUDF conflicts with pom.xml

2016-10-31 Thread Fei Hui (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624106#comment-15624106
 ] 

Fei Hui commented on HIVE-15096:


there is no hplsql.jar with hive release. users do not change this file if they 
have no source code

> hplsql registerUDF conflicts with pom.xml
> -
>
> Key: HIVE-15096
> URL: https://issues.apache.org/jira/browse/HIVE-15096
> Project: Hive
>  Issue Type: Bug
>  Components: hpl/sql
>Affects Versions: 2.0.0, 2.1.0, 2.0.1
>Reporter: Fei Hui
> Fix For: 2.2.0
>
> Attachments: HIVE-15096.patch
>
>
> in hplsql code, registerUDF code is
> sql.add("ADD JAR " + dir + "hplsql.jar");
> sql.add("ADD JAR " + dir + "antlr-runtime-4.5.jar");
> sql.add("ADD FILE " + dir + Conf.SITE_XML);
> but pom configufation is
>   
> org.apache.hive
> hive
> 2.2.0-SNAPSHOT
> ../pom.xml
>   
>   hive-hplsql
>   jar
>   Hive HPL/SQL
> 
>org.antlr
>antlr4-runtime
>4.5
> 
> when run hplsql , errors occur as below
>  Error while processing statement: 
> /opt/apps/apache-hive-2.0.0-bin/lib/hplsql.jar does not exist



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-14943) Base Implementation

2016-10-31 Thread Eugene Koifman (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-14943:
--
Attachment: HIVE-14943.6.patch

patch 6 - support for update and delete in the same merge stmt

> Base Implementation
> ---
>
> Key: HIVE-14943
> URL: https://issues.apache.org/jira/browse/HIVE-14943
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-14943.2.patch, HIVE-14943.3.patch, 
> HIVE-14943.4.patch, HIVE-14943.5.patch, HIVE-14943.6.patch, HIVE-14943.patch
>
>
> Create the 1st pass functional implementation of MERGE
> This should run e2e and produce correct results.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-14945) Add support for multiple WHEN MATCHED clauses

2016-10-31 Thread Eugene Koifman (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-14945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-14945:
--
Attachment: (was: HIVE-14943.6.patch)

> Add support for multiple WHEN MATCHED clauses
> -
>
> Key: HIVE-14945
> URL: https://issues.apache.org/jira/browse/HIVE-14945
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> Different vendors have different rules.
> HIVE-14943 only allows 1 WHEN MATCHED clause
> IBM Allows > 1 WHEN MATCHED clause 
> https://www.ibm.com/support/knowledgecenter/SSEPGG_9.5.0/com.ibm.db2.luw.sql.ref.doc/doc/r0010873.html
> So does SQL Server https://msdn.microsoft.com/en-us/library/bb510625.aspx
> everyone seems to support update/delete/insert - in that order



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-14945) Add support for multiple WHEN MATCHED clauses

2016-10-31 Thread Eugene Koifman (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-14945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-14945:
--
Attachment: HIVE-14943.6.patch

patch 6 - support for update and delete in the same merge stmt

> Add support for multiple WHEN MATCHED clauses
> -
>
> Key: HIVE-14945
> URL: https://issues.apache.org/jira/browse/HIVE-14945
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-14943.6.patch
>
>
> Different vendors have different rules.
> HIVE-14943 only allows 1 WHEN MATCHED clause
> IBM Allows > 1 WHEN MATCHED clause 
> https://www.ibm.com/support/knowledgecenter/SSEPGG_9.5.0/com.ibm.db2.luw.sql.ref.doc/doc/r0010873.html
> So does SQL Server https://msdn.microsoft.com/en-us/library/bb510625.aspx
> everyone seems to support update/delete/insert - in that order



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (HIVE-14945) Add support for multiple WHEN MATCHED clauses

2016-10-31 Thread Eugene Koifman (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-14945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman resolved HIVE-14945.
---
Resolution: Duplicate

Fixed in HIVE-14943

> Add support for multiple WHEN MATCHED clauses
> -
>
> Key: HIVE-14945
> URL: https://issues.apache.org/jira/browse/HIVE-14945
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> Different vendors have different rules.
> HIVE-14943 only allows 1 WHEN MATCHED clause
> IBM Allows > 1 WHEN MATCHED clause 
> https://www.ibm.com/support/knowledgecenter/SSEPGG_9.5.0/com.ibm.db2.luw.sql.ref.doc/doc/r0010873.html
> So does SQL Server https://msdn.microsoft.com/en-us/library/bb510625.aspx
> everyone seems to support update/delete/insert - in that order



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15099) PTFOperator.PTFInvocation didn't properly reset the input partition

2016-10-31 Thread Wei Zheng (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-15099:
-
  Resolution: Fixed
   Fix Version/s: 2.1.1
  2.2.0
  1.3.0
Target Version/s: 1.3.0, 2.2.0, 2.1.1  (was: 1.3.0, 2.2.0)
  Status: Resolved  (was: Patch Available)

Committed to master, branch-2.1 and branch-1. Thanks Ashutosh for the review!

> PTFOperator.PTFInvocation didn't properly reset the input partition
> ---
>
> Key: HIVE-15099
> URL: https://issues.apache.org/jira/browse/HIVE-15099
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, PTF-Windowing
>Affects Versions: 1.3.0, 1.2.1, 2.1.0, 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Fix For: 1.3.0, 2.2.0, 2.1.1
>
> Attachments: HIVE-15099.1.patch
>
>
> There is an issue with PTFOperator.PTFInvocation where the inputPart is not 
> reset properly. The inputPart has been closed and its content (member 
> variables) has been cleaned up, but since itself is not nullified, it's 
> reused in the next round and caused NPE issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15039) A better job monitor console output for HoS

2016-10-31 Thread Ferdinand Xu (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623891#comment-15623891
 ] 

Ferdinand Xu commented on HIVE-15039:
-

Thanks [~sseth]

> A better job monitor console output for HoS
> ---
>
> Key: HIVE-15039
> URL: https://issues.apache.org/jira/browse/HIVE-15039
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-15039.1.patch, HIVE-15039.2.patch, example 
> screenshot.png, with patch.png
>
>
> When there're many stages, it's very difficult to read the console output of 
> job progress of HoS. Attached screenshot is an example.
> We may learn from HoT as it does much better than HoS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15099) PTFOperator.PTFInvocation didn't properly reset the input partition

2016-10-31 Thread Ashutosh Chauhan (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623870#comment-15623870
 ] 

Ashutosh Chauhan commented on HIVE-15099:
-

+1

> PTFOperator.PTFInvocation didn't properly reset the input partition
> ---
>
> Key: HIVE-15099
> URL: https://issues.apache.org/jira/browse/HIVE-15099
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, PTF-Windowing
>Affects Versions: 1.3.0, 1.2.1, 2.1.0, 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15099.1.patch
>
>
> There is an issue with PTFOperator.PTFInvocation where the inputPart is not 
> reset properly. The inputPart has been closed and its content (member 
> variables) has been cleaned up, but since itself is not nullified, it's 
> reused in the next round and caused NPE issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15099) PTFOperator.PTFInvocation didn't properly reset the input partition

2016-10-31 Thread Wei Zheng (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623868#comment-15623868
 ] 

Wei Zheng commented on HIVE-15099:
--

The problematic plan can only be reproduced with specific environment and 
dataset

> PTFOperator.PTFInvocation didn't properly reset the input partition
> ---
>
> Key: HIVE-15099
> URL: https://issues.apache.org/jira/browse/HIVE-15099
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, PTF-Windowing
>Affects Versions: 1.3.0, 1.2.1, 2.1.0, 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15099.1.patch
>
>
> There is an issue with PTFOperator.PTFInvocation where the inputPart is not 
> reset properly. The inputPart has been closed and its content (member 
> variables) has been cleaned up, but since itself is not nullified, it's 
> reused in the next round and caused NPE issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15094) Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types

2016-10-31 Thread Jesus Camacho Rodriguez (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-15094:
---
Attachment: HIVE-15094-branch-2.1.patch

> Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types
> -
>
> Key: HIVE-15094
> URL: https://issues.apache.org/jira/browse/HIVE-15094
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sergio Peña
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-15094-branch-2.1.patch
>
>
> Several tests failures related to schema evolution are happening on 
> branch-2.1 due to a patch reverted in the past.
> {noformat}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> 

[jira] [Updated] (HIVE-15094) Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types

2016-10-31 Thread Jesus Camacho Rodriguez (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-15094:
---
Status: Patch Available  (was: In Progress)

> Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types
> -
>
> Key: HIVE-15094
> URL: https://issues.apache.org/jira/browse/HIVE-15094
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sergio Peña
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-15094-branch-2.1.patch
>
>
> Several tests failures related to schema evolution are happening on 
> branch-2.1 due to a patch reverted in the past.
> {noformat}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> 

[jira] [Work started] (HIVE-15094) Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types

2016-10-31 Thread Jesus Camacho Rodriguez (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-15094 started by Jesus Camacho Rodriguez.
--
> Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types
> -
>
> Key: HIVE-15094
> URL: https://issues.apache.org/jira/browse/HIVE-15094
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sergio Peña
>Assignee: Jesus Camacho Rodriguez
>
> Several tests failures related to schema evolution are happening on 
> branch-2.1 due to a patch reverted in the past.
> {noformat}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> 

[jira] [Commented] (HIVE-15099) PTFOperator.PTFInvocation didn't properly reset the input partition

2016-10-31 Thread Ashutosh Chauhan (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623742#comment-15623742
 ] 

Ashutosh Chauhan commented on HIVE-15099:
-

Patch looks good to me. [~wzheng] Can you add a test case for it as well?

> PTFOperator.PTFInvocation didn't properly reset the input partition
> ---
>
> Key: HIVE-15099
> URL: https://issues.apache.org/jira/browse/HIVE-15099
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, PTF-Windowing
>Affects Versions: 1.3.0, 1.2.1, 2.1.0, 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15099.1.patch
>
>
> There is an issue with PTFOperator.PTFInvocation where the inputPart is not 
> reset properly. The inputPart has been closed and its content (member 
> variables) has been cleaned up, but since itself is not nullified, it's 
> reused in the next round and caused NPE issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15041) Specify GCE network name on Hive ptest

2016-10-31 Thread Siddharth Seth (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623687#comment-15623687
 ] 

Siddharth Seth commented on HIVE-15041:
---

Got it. Was looking at the error.

A post call to projects/gcp-hive-upstream/global/firewalls
complaining about gcp-hive-upstream/global/networks/jclouds-hive-ptest-slaves 
not being available. (Should this have been 
gcp-hive-upstream/global/networks/hive-ptest-slaves). Not sure where the 
jclouds gets inserted.

Anyway - seems like any refresh causes this problem. Till this figured out, one 
possibility would be to try and re-created the ExecutionContextProvider once 
the timeout has expired, instead of attempting to re-use the same 
executionContextProvider to get more nodes.
Couldn't trace down what changes in between which could cause this to fail. The 
config etc could change a bit to make parameters final for thread visibility - 
but I don't think that is the problem.

> Specify GCE network name on Hive ptest
> --
>
> Key: HIVE-15041
> URL: https://issues.apache.org/jira/browse/HIVE-15041
> Project: Hive
>  Issue Type: Task
>  Components: Hive
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-15041.1.patch, HIVE-15041.2.patch
>
>
> NO PRECOMMIT TESTS
> A new option on cloudhost.properties should be added to specify the GCE 
> network name:
> # GCE network option
> network = 
> https://www.googleapis.com/compute/v1/projects//global/networks/default



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15041) Specify GCE network name on Hive ptest

2016-10-31 Thread JIRA

[ 
https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623608#comment-15623608
 ] 

Sergio Peña commented on HIVE-15041:


[~sseth] Seems that running a 'git pull origin' will apply all latest commits 
clean if the changes on the code do not conflict. 

Regarding stricthostkeychecking, the /etc/ssh/ssh_config is configured with the 
following lines to avoid errors with SSH:
StrictHostKeyChecking no
UserKnownHostsFile=/dev/nul

> Specify GCE network name on Hive ptest
> --
>
> Key: HIVE-15041
> URL: https://issues.apache.org/jira/browse/HIVE-15041
> Project: Hive
>  Issue Type: Task
>  Components: Hive
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-15041.1.patch, HIVE-15041.2.patch
>
>
> NO PRECOMMIT TESTS
> A new option on cloudhost.properties should be added to specify the GCE 
> network name:
> # GCE network option
> network = 
> https://www.googleapis.com/compute/v1/projects//global/networks/default



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15041) Specify GCE network name on Hive ptest

2016-10-31 Thread JIRA

 [ 
https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Peña updated HIVE-15041:
---
Attachment: HIVE-15041.2.patch

Here's the latest patch that was running for 1 week in ptest.

> Specify GCE network name on Hive ptest
> --
>
> Key: HIVE-15041
> URL: https://issues.apache.org/jira/browse/HIVE-15041
> Project: Hive
>  Issue Type: Task
>  Components: Hive
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-15041.1.patch, HIVE-15041.2.patch
>
>
> NO PRECOMMIT TESTS
> A new option on cloudhost.properties should be added to specify the GCE 
> network name:
> # GCE network option
> network = 
> https://www.googleapis.com/compute/v1/projects//global/networks/default



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-14990) run all tests for MM tables and fix the issues that are found

2016-10-31 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-14990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623580#comment-15623580
 ] 

Sergey Shelukhin commented on HIVE-14990:
-

HCat tests are expected to fail - no support for any special tables like ACID 
or MM. Added a proper error for HCat output.

> run all tests for MM tables and fix the issues that are found
> -
>
> Key: HIVE-14990
> URL: https://issues.apache.org/jira/browse/HIVE-14990
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14990.01.patch, HIVE-14990.02.patch, 
> HIVE-14990.03.patch, HIVE-14990.04.patch, HIVE-14990.04.patch, 
> HIVE-14990.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15093) For S3-to-S3 renames, files should be moved individually rather than at a directory level

2016-10-31 Thread Sahil Takiar (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623569#comment-15623569
 ] 

Sahil Takiar commented on HIVE-15093:
-

[~spena] - patch is ready for review. I created an RB and linked it to this 
JIRA.

> For S3-to-S3 renames, files should be moved individually rather than at a 
> directory level
> -
>
> Key: HIVE-15093
> URL: https://issues.apache.org/jira/browse/HIVE-15093
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-15093.1.patch, HIVE-15093.2.patch, 
> HIVE-15093.3.patch, HIVE-15093.4.patch
>
>
> Hive's MoveTask uses the Hive.moveFile method to move data within a 
> distributed filesystem as well as blobstore filesystems.
> If the move is done within the same filesystem:
> 1: If the source path is a subdirectory of the destination path, files will 
> be moved one by one using a threapool of workers
> 2: If the source path is not a subdirectory of the destination path, a single 
> rename operation is used to move the entire directory
> The second option may not work well on blobstores such as S3. Renames are not 
> metadata operations and require copying all the data. Client connectors to 
> blobstores may not efficiently rename directories. Worst case, the connector 
> will copy each file one by one, sequentially rather than using a threadpool 
> of workers to copy the data (e.g. HADOOP-13600).
> Hive already has code to rename files using a threadpool of workers, but this 
> only occurs in case number 1.
> This JIRA aims to modify the code so that case 1 is triggered when copying 
> within a blobstore. The focus is on copies within a blobstore because 
> needToCopy will return true if the src and target filesystems are different, 
> in which case a different code path is triggered.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15093) For S3-to-S3 renames, files should be moved individually rather than at a directory level

2016-10-31 Thread Sahil Takiar (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-15093:

Attachment: HIVE-15093.4.patch

> For S3-to-S3 renames, files should be moved individually rather than at a 
> directory level
> -
>
> Key: HIVE-15093
> URL: https://issues.apache.org/jira/browse/HIVE-15093
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-15093.1.patch, HIVE-15093.2.patch, 
> HIVE-15093.3.patch, HIVE-15093.4.patch
>
>
> Hive's MoveTask uses the Hive.moveFile method to move data within a 
> distributed filesystem as well as blobstore filesystems.
> If the move is done within the same filesystem:
> 1: If the source path is a subdirectory of the destination path, files will 
> be moved one by one using a threapool of workers
> 2: If the source path is not a subdirectory of the destination path, a single 
> rename operation is used to move the entire directory
> The second option may not work well on blobstores such as S3. Renames are not 
> metadata operations and require copying all the data. Client connectors to 
> blobstores may not efficiently rename directories. Worst case, the connector 
> will copy each file one by one, sequentially rather than using a threadpool 
> of workers to copy the data (e.g. HADOOP-13600).
> Hive already has code to rename files using a threadpool of workers, but this 
> only occurs in case number 1.
> This JIRA aims to modify the code so that case 1 is triggered when copying 
> within a blobstore. The focus is on copies within a blobstore because 
> needToCopy will return true if the src and target filesystems are different, 
> in which case a different code path is triggered.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15062) create backward compat checking for metastore APIs

2016-10-31 Thread Thejas M Nair (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623517#comment-15623517
 ] 

Thejas M Nair commented on HIVE-15062:
--

[~sershe] Thanks for clarifying (offline) that this config is settable from the 
client because it has been added to HiveConf.metaConfVars.

+1


> create backward compat checking for metastore APIs
> --
>
> Key: HIVE-15062
> URL: https://issues.apache.org/jira/browse/HIVE-15062
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15062.01.nogen.patch, HIVE-15062.01.patch, 
> HIVE-15062.02.nogen.patch, HIVE-15062.02.patch, HIVE-15062.03.nogen.patch, 
> HIVE-15062.03.patch, HIVE-15062.04.nogen.patch, HIVE-15062.04.patch, 
> HIVE-15062.nogen.patch, HIVE-15062.patch
>
>
> This is to add client capability checking to Hive metastore.
> This could have been used, for example, when introducing ACID tables - a 
> client trying to get_table on such a table without specifying that it is 
> aware of ACID tables would get an error by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15057) Support other types of operators (other than SELECT)

2016-10-31 Thread Chao Sun (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Sun updated HIVE-15057:

Attachment: HIVE-15057.1.patch

> Support other types of operators (other than SELECT)
> 
>
> Key: HIVE-15057
> URL: https://issues.apache.org/jira/browse/HIVE-15057
> Project: Hive
>  Issue Type: Sub-task
>  Components: Logical Optimizer, Physical Optimizer
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-15057.1.patch
>
>
> Currently only SELECT operators are supported for nested column pruning. We 
> should add support for other types of operators so the optimization can work 
> for complex queries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15057) Support other types of operators (other than SELECT)

2016-10-31 Thread Chao Sun (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Sun updated HIVE-15057:

Attachment: (was: HIVE-15057.1.patch)

> Support other types of operators (other than SELECT)
> 
>
> Key: HIVE-15057
> URL: https://issues.apache.org/jira/browse/HIVE-15057
> Project: Hive
>  Issue Type: Sub-task
>  Components: Logical Optimizer, Physical Optimizer
>Reporter: Chao Sun
>Assignee: Chao Sun
>
> Currently only SELECT operators are supported for nested column pruning. We 
> should add support for other types of operators so the optimization can work 
> for complex queries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15057) Support other types of operators (other than SELECT)

2016-10-31 Thread Chao Sun (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Sun updated HIVE-15057:

Status: Patch Available  (was: Open)

> Support other types of operators (other than SELECT)
> 
>
> Key: HIVE-15057
> URL: https://issues.apache.org/jira/browse/HIVE-15057
> Project: Hive
>  Issue Type: Sub-task
>  Components: Logical Optimizer, Physical Optimizer
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-15057.1.patch
>
>
> Currently only SELECT operators are supported for nested column pruning. We 
> should add support for other types of operators so the optimization can work 
> for complex queries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15057) Support other types of operators (other than SELECT)

2016-10-31 Thread Chao Sun (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Sun updated HIVE-15057:

Attachment: HIVE-15057.1.patch

Attaching initial patch for testing. Not fully ready for review yet.

> Support other types of operators (other than SELECT)
> 
>
> Key: HIVE-15057
> URL: https://issues.apache.org/jira/browse/HIVE-15057
> Project: Hive
>  Issue Type: Sub-task
>  Components: Logical Optimizer, Physical Optimizer
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-15057.1.patch
>
>
> Currently only SELECT operators are supported for nested column pruning. We 
> should add support for other types of operators so the optimization can work 
> for complex queries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (HIVE-14880) integrate MM tables into ACID: allow insert overwrite and don't require buckets, ORC etc. for the new type; don't run compaction

2016-10-31 Thread Wei Zheng (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-14880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng resolved HIVE-14880.
--
  Resolution: Duplicate
Release Note: This was included in HIVE-15087

> integrate MM tables into ACID: allow insert overwrite and don't require 
> buckets, ORC etc. for the new type; don't run compaction
> 
>
> Key: HIVE-14880
> URL: https://issues.apache.org/jira/browse/HIVE-14880
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Wei Zheng
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Assigned] (HIVE-14880) integrate MM tables into ACID: allow insert overwrite and don't require buckets, ORC etc. for the new type; don't run compaction

2016-10-31 Thread Wei Zheng (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-14880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng reassigned HIVE-14880:


Assignee: Wei Zheng

> integrate MM tables into ACID: allow insert overwrite and don't require 
> buckets, ORC etc. for the new type; don't run compaction
> 
>
> Key: HIVE-14880
> URL: https://issues.apache.org/jira/browse/HIVE-14880
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Wei Zheng
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15041) Specify GCE network name on Hive ptest

2016-10-31 Thread Siddharth Seth (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623465#comment-15623465
 ] 

Siddharth Seth commented on HIVE-15041:
---

Didn't know the patch was already running. Wouldn't it reset each time there's 
an update and restart if it's not committed?

I'll try taking a look as well. In terms of the IP problem - would making use 
of hostnames instead help? If GCE is giving the same IPs - I'm surprised we 
haven't run into problems with entries in .ssh/known_hosts. (Some part of the 
code works with stricthostkeychecking set to no, other parts don't)

> Specify GCE network name on Hive ptest
> --
>
> Key: HIVE-15041
> URL: https://issues.apache.org/jira/browse/HIVE-15041
> Project: Hive
>  Issue Type: Task
>  Components: Hive
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-15041.1.patch
>
>
> NO PRECOMMIT TESTS
> A new option on cloudhost.properties should be added to specify the GCE 
> network name:
> # GCE network option
> network = 
> https://www.googleapis.com/compute/v1/projects//global/networks/default



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15094) Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types

2016-10-31 Thread Aihua Xu (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623431#comment-15623431
 ] 

Aihua Xu commented on HIVE-15094:
-

Sure. Sergio asked me to check this. Didn't notice you already started.

> Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types
> -
>
> Key: HIVE-15094
> URL: https://issues.apache.org/jira/browse/HIVE-15094
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sergio Peña
>Assignee: Jesus Camacho Rodriguez
>
> Several tests failures related to schema evolution are happening on 
> branch-2.1 due to a patch reverted in the past.
> {noformat}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> 

[jira] [Commented] (HIVE-15099) PTFOperator.PTFInvocation didn't properly reset the input partition

2016-10-31 Thread Wei Zheng (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623430#comment-15623430
 ] 

Wei Zheng commented on HIVE-15099:
--

[~ashutoshc] Can you take a look please?

> PTFOperator.PTFInvocation didn't properly reset the input partition
> ---
>
> Key: HIVE-15099
> URL: https://issues.apache.org/jira/browse/HIVE-15099
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, PTF-Windowing
>Affects Versions: 1.3.0, 1.2.1, 2.1.0, 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15099.1.patch
>
>
> There is an issue with PTFOperator.PTFInvocation where the inputPart is not 
> reset properly. The inputPart has been closed and its content (member 
> variables) has been cleaned up, but since itself is not nullified, it's 
> reused in the next round and caused NPE issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15061) Metastore types are sometimes case sensitive

2016-10-31 Thread Chaoyu Tang (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chaoyu Tang updated HIVE-15061:
---
   Resolution: Fixed
Fix Version/s: 2.1.1
   2.2.0
   Status: Resolved  (was: Patch Available)

Committed to 2.2.0 and 2.1.1. Thanks [~mohitsabharwal] for reviewing the patch.

> Metastore types are sometimes case sensitive
> 
>
> Key: HIVE-15061
> URL: https://issues.apache.org/jira/browse/HIVE-15061
> Project: Hive
>  Issue Type: Bug
>  Components: API
>Affects Versions: 1.1.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Chaoyu Tang
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-15061.1.patch, HIVE-15061.1.patch, HIVE-15061.patch
>
>
> Impala recently encountered an issue with the metastore 
> ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column 
> stats would get dropped when adding a column to a table.
> The reason seems to be that Hive does a case sensitive check on the column 
> stats types during an "alter table" and expects the types to be all lower 
> case. This case sensitive check doesn't appear to happen when the stats are 
> set in the first place.
> We're solving this on the Impala end by storing types in the metastore as all 
> lower case, but Hive's behavior here is very confusing. It should either 
> always be case sensitive, so that you can't create column stats with types 
> that Hive considers invalid, or it should never be case sensitive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Assigned] (HIVE-13947) HoS print wrong number for hash table size in map join scenario

2016-10-31 Thread Aihua Xu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-13947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu reassigned HIVE-13947:
---

Assignee: Aihua Xu

> HoS print wrong number for hash table size in map join scenario
> ---
>
> Key: HIVE-13947
> URL: https://issues.apache.org/jira/browse/HIVE-13947
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 1.2.1
>Reporter: wangwenli
>Assignee: Aihua Xu
>Priority: Minor
>
> In *sparkHashTableSinkOperator*, when *flushToFile*, before close output 
> stream, it try to get the file length, and will get 0 for it,  take 
> *hashTableSinkOperator* for ref, it should get length after output stream 
> closed



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15003) Update 'ALTER TABLE...UPDATE STATISTICS FOR COLUMN..' statement to support more data types

2016-10-31 Thread Vineet Garg (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-15003:
---
Status: Open  (was: Patch Available)

> Update 'ALTER TABLE...UPDATE STATISTICS FOR COLUMN..' statement to support 
> more data types
> --
>
> Key: HIVE-15003
> URL: https://issues.apache.org/jira/browse/HIVE-15003
> Project: Hive
>  Issue Type: Task
>  Components: Statistics
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-15003.1.patch
>
>
> Currently ALTER TABLE...UDPATE STATISTICS FOR COLUMN... only support updating 
> statistics for following data types:
> * STRING
> * DOUBLE
> * BOOLEAN
> * BINARY
> * DECIMAL
> * DATE
> We are missing the following data types
> * TINYINT
> * SMALLINT
> * INT
> * BIGINT
> * FLOAT
> * VARCHAR
> * CHAR
> * TIMESTAMP



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15003) Update 'ALTER TABLE...UPDATE STATISTICS FOR COLUMN..' statement to support more data types

2016-10-31 Thread Vineet Garg (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-15003:
---
Status: Patch Available  (was: Open)

Submitting once again to run pre-commit testing. 

> Update 'ALTER TABLE...UPDATE STATISTICS FOR COLUMN..' statement to support 
> more data types
> --
>
> Key: HIVE-15003
> URL: https://issues.apache.org/jira/browse/HIVE-15003
> Project: Hive
>  Issue Type: Task
>  Components: Statistics
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-15003.1.patch
>
>
> Currently ALTER TABLE...UDPATE STATISTICS FOR COLUMN... only support updating 
> statistics for following data types:
> * STRING
> * DOUBLE
> * BOOLEAN
> * BINARY
> * DECIMAL
> * DATE
> We are missing the following data types
> * TINYINT
> * SMALLINT
> * INT
> * BIGINT
> * FLOAT
> * VARCHAR
> * CHAR
> * TIMESTAMP



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15093) For S3-to-S3 renames, files should be moved individually rather than at a directory level

2016-10-31 Thread Sahil Takiar (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623359#comment-15623359
 ] 

Sahil Takiar commented on HIVE-15093:
-

Patch is still a WIP, had to re-attach version 2 again because ptest was 
failing for some reason.

> For S3-to-S3 renames, files should be moved individually rather than at a 
> directory level
> -
>
> Key: HIVE-15093
> URL: https://issues.apache.org/jira/browse/HIVE-15093
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-15093.1.patch, HIVE-15093.2.patch, 
> HIVE-15093.3.patch
>
>
> Hive's MoveTask uses the Hive.moveFile method to move data within a 
> distributed filesystem as well as blobstore filesystems.
> If the move is done within the same filesystem:
> 1: If the source path is a subdirectory of the destination path, files will 
> be moved one by one using a threapool of workers
> 2: If the source path is not a subdirectory of the destination path, a single 
> rename operation is used to move the entire directory
> The second option may not work well on blobstores such as S3. Renames are not 
> metadata operations and require copying all the data. Client connectors to 
> blobstores may not efficiently rename directories. Worst case, the connector 
> will copy each file one by one, sequentially rather than using a threadpool 
> of workers to copy the data (e.g. HADOOP-13600).
> Hive already has code to rename files using a threadpool of workers, but this 
> only occurs in case number 1.
> This JIRA aims to modify the code so that case 1 is triggered when copying 
> within a blobstore. The focus is on copies within a blobstore because 
> needToCopy will return true if the src and target filesystems are different, 
> in which case a different code path is triggered.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15093) For S3-to-S3 renames, files should be moved individually rather than at a directory level

2016-10-31 Thread Sahil Takiar (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-15093:

Attachment: HIVE-15093.3.patch

> For S3-to-S3 renames, files should be moved individually rather than at a 
> directory level
> -
>
> Key: HIVE-15093
> URL: https://issues.apache.org/jira/browse/HIVE-15093
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-15093.1.patch, HIVE-15093.2.patch, 
> HIVE-15093.3.patch
>
>
> Hive's MoveTask uses the Hive.moveFile method to move data within a 
> distributed filesystem as well as blobstore filesystems.
> If the move is done within the same filesystem:
> 1: If the source path is a subdirectory of the destination path, files will 
> be moved one by one using a threapool of workers
> 2: If the source path is not a subdirectory of the destination path, a single 
> rename operation is used to move the entire directory
> The second option may not work well on blobstores such as S3. Renames are not 
> metadata operations and require copying all the data. Client connectors to 
> blobstores may not efficiently rename directories. Worst case, the connector 
> will copy each file one by one, sequentially rather than using a threadpool 
> of workers to copy the data (e.g. HADOOP-13600).
> Hive already has code to rename files using a threadpool of workers, but this 
> only occurs in case number 1.
> This JIRA aims to modify the code so that case 1 is triggered when copying 
> within a blobstore. The focus is on copies within a blobstore because 
> needToCopy will return true if the src and target filesystems are different, 
> in which case a different code path is triggered.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15062) create backward compat checking for metastore APIs

2016-10-31 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-15062:

Attachment: HIVE-15062.04.patch
HIVE-15062.04.nogen.patch

Changed the API layout and error message.

> create backward compat checking for metastore APIs
> --
>
> Key: HIVE-15062
> URL: https://issues.apache.org/jira/browse/HIVE-15062
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15062.01.nogen.patch, HIVE-15062.01.patch, 
> HIVE-15062.02.nogen.patch, HIVE-15062.02.patch, HIVE-15062.03.nogen.patch, 
> HIVE-15062.03.patch, HIVE-15062.04.nogen.patch, HIVE-15062.04.patch, 
> HIVE-15062.nogen.patch, HIVE-15062.patch
>
>
> This is to add client capability checking to Hive metastore.
> This could have been used, for example, when introducing ACID tables - a 
> client trying to get_table on such a table without specifying that it is 
> aware of ACID tables would get an error by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15062) create backward compat checking for metastore APIs

2016-10-31 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623336#comment-15623336
 ] 

Sergey Shelukhin commented on HIVE-15062:
-

Thanks for the feedback.
I have added the usage guidelines in the exception text; the setting is also 
settable from the client (added that to the error message now), so such tools 
should be able to set it per session.
I don't think there's a way around this for external tools - at worst, they 
will have to disable this globally, which is no worse than the current 
situation, or they can disable it per session. However, we would have proper 
handling for different versions of Hive, such as in heterogeneous clusters and 
for rolling upgrade.

> create backward compat checking for metastore APIs
> --
>
> Key: HIVE-15062
> URL: https://issues.apache.org/jira/browse/HIVE-15062
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15062.01.nogen.patch, HIVE-15062.01.patch, 
> HIVE-15062.02.nogen.patch, HIVE-15062.02.patch, HIVE-15062.03.nogen.patch, 
> HIVE-15062.03.patch, HIVE-15062.nogen.patch, HIVE-15062.patch
>
>
> This is to add client capability checking to Hive metastore.
> This could have been used, for example, when introducing ACID tables - a 
> client trying to get_table on such a table without specifying that it is 
> aware of ACID tables would get an error by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.

2016-10-31 Thread Szehon Ho (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623323#comment-15623323
 ] 

Szehon Ho commented on HIVE-13517:
--

Yea if the thread name is there, that is great.  

I thought last time when I checked the Spark Executor and Driver logs that they 
were mixed, and there was no indication about the thread.  I don't have an 
environment right now to check that, do you see the thread name now in those 
logs?

> Hive logs in Spark Executor and Driver should show thread-id.
> -
>
> Key: HIVE-13517
> URL: https://issues.apache.org/jira/browse/HIVE-13517
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Szehon Ho
>Assignee: liyunzhang_intel
>
> In Spark, there might be more than one task running in one executor. 
> Similarly, there may be more than one thread running in Driver.
> This makes debugging through the logs a nightmare. It would be great if there 
> could be thread-ids in the logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15094) Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types

2016-10-31 Thread Jesus Camacho Rodriguez (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623313#comment-15623313
 ] 

Jesus Camacho Rodriguez commented on HIVE-15094:


Sure. In case you want to help, there are still a few issues (some of them look 
like just regenerating q files) in last ptest run in HIVE-14884.

> Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types
> -
>
> Key: HIVE-15094
> URL: https://issues.apache.org/jira/browse/HIVE-15094
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sergio Peña
>Assignee: Jesus Camacho Rodriguez
>
> Several tests failures related to schema evolution are happening on 
> branch-2.1 due to a patch reverted in the past.
> {noformat}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> 

[jira] [Assigned] (HIVE-15094) Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types

2016-10-31 Thread Jesus Camacho Rodriguez (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-15094:
--

Assignee: Jesus Camacho Rodriguez  (was: Aihua Xu)

> Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types
> -
>
> Key: HIVE-15094
> URL: https://issues.apache.org/jira/browse/HIVE-15094
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sergio Peña
>Assignee: Jesus Camacho Rodriguez
>
> Several tests failures related to schema evolution are happening on 
> branch-2.1 due to a patch reverted in the past.
> {noformat}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> 

[jira] [Commented] (HIVE-15094) Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types

2016-10-31 Thread Aihua Xu (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623307#comment-15623307
 ] 

Aihua Xu commented on HIVE-15094:
-

Cool. Can you take it then? 

> Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types
> -
>
> Key: HIVE-15094
> URL: https://issues.apache.org/jira/browse/HIVE-15094
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sergio Peña
>Assignee: Aihua Xu
>
> Several tests failures related to schema evolution are happening on 
> branch-2.1 due to a patch reverted in the past.
> {noformat}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> 

[jira] [Comment Edited] (HIVE-15094) Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types

2016-10-31 Thread Jesus Camacho Rodriguez (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623304#comment-15623304
 ] 

Jesus Camacho Rodriguez edited comment on HIVE-15094 at 10/31/16 8:24 PM:
--

I almost have the fix for this one [~aihuaxu] (changing tests and regenerating 
q files).


was (Author: jcamachorodriguez):
I almost have the fix for this one [~aihuaxu].

> Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types
> -
>
> Key: HIVE-15094
> URL: https://issues.apache.org/jira/browse/HIVE-15094
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sergio Peña
>Assignee: Aihua Xu
>
> Several tests failures related to schema evolution are happening on 
> branch-2.1 due to a patch reverted in the past.
> {noformat}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> 

[jira] [Commented] (HIVE-15094) Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types

2016-10-31 Thread Jesus Camacho Rodriguez (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623304#comment-15623304
 ] 

Jesus Camacho Rodriguez commented on HIVE-15094:


I almost have the fix for this one [~aihuaxu].

> Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types
> -
>
> Key: HIVE-15094
> URL: https://issues.apache.org/jira/browse/HIVE-15094
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sergio Peña
>Assignee: Aihua Xu
>
> Several tests failures related to schema evolution are happening on 
> branch-2.1 due to a patch reverted in the past.
> {noformat}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> 

[jira] [Assigned] (HIVE-15094) Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types

2016-10-31 Thread Aihua Xu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu reassigned HIVE-15094:
---

Assignee: Aihua Xu  (was: Jesus Camacho Rodriguez)

> Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types
> -
>
> Key: HIVE-15094
> URL: https://issues.apache.org/jira/browse/HIVE-15094
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sergio Peña
>Assignee: Aihua Xu
>
> Several tests failures related to schema evolution are happening on 
> branch-2.1 due to a patch reverted in the past.
> {noformat}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> 

[jira] [Commented] (HIVE-14990) run all tests for MM tables and fix the issues that are found

2016-10-31 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-14990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623279#comment-15623279
 ] 

Sergey Shelukhin commented on HIVE-14990:
-

Any tests with merge concatenate are expected to fail.
Many diffs because MoveTask for merge is gone for MM tables for the 
merge-or-move path, since there can be no partial merges.
Lots of stats diffs - probably broken stat collection, need to check...

There are some failures with bad results or exceptions, looking at those and at 
unit tests.


> run all tests for MM tables and fix the issues that are found
> -
>
> Key: HIVE-14990
> URL: https://issues.apache.org/jira/browse/HIVE-14990
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14990.01.patch, HIVE-14990.02.patch, 
> HIVE-14990.03.patch, HIVE-14990.04.patch, HIVE-14990.04.patch, 
> HIVE-14990.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-13989) Extended ACLs are not handled according to specification

2016-10-31 Thread Alan Gates (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-13989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-13989:
--
Target Version/s: 2.0.0, 1.2.1  (was: 1.2.1, 2.0.0)
  Status: Open  (was: Patch Available)

Cancelling patch as it looks like further investigation is required.

> Extended ACLs are not handled according to specification
> 
>
> Key: HIVE-13989
> URL: https://issues.apache.org/jira/browse/HIVE-13989
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 2.0.0, 1.2.1
>Reporter: Chris Drome
>Assignee: Chris Drome
> Attachments: HIVE-13989-branch-1.patch, HIVE-13989.1-branch-1.patch, 
> HIVE-13989.1.patch
>
>
> Hive takes two approaches to working with extended ACLs depending on whether 
> data is being produced via a Hive query or HCatalog APIs. A Hive query will 
> run an FsShell command to recursively set the extended ACLs for a directory 
> sub-tree. HCatalog APIs will attempt to build up the directory sub-tree 
> programmatically and runs some code to set the ACLs to match the parent 
> directory.
> Some incorrect assumptions were made when implementing the extended ACLs 
> support. Refer to https://issues.apache.org/jira/browse/HDFS-4685 for the 
> design documents of extended ACLs in HDFS. These documents model the 
> implementation after the POSIX implementation on Linux, which can be found at 
> http://www.vanemery.com/Linux/ACL/POSIX_ACL_on_Linux.html.
> The code for setting extended ACLs via HCatalog APIs is found in 
> HdfsUtils.java:
> {code}
> if (aclEnabled) {
>   aclStatus =  sourceStatus.getAclStatus();
>   if (aclStatus != null) {
> LOG.trace(aclStatus.toString());
> aclEntries = aclStatus.getEntries();
> removeBaseAclEntries(aclEntries);
> //the ACL api's also expect the tradition user/group/other permission 
> in the form of ACL
> aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.USER, 
> sourcePerm.getUserAction()));
> aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.GROUP, 
> sourcePerm.getGroupAction()));
> aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.OTHER, 
> sourcePerm.getOtherAction()));
>   }
> }
> {code}
> We found that DEFAULT extended ACL rules were not being inherited properly by 
> the directory sub-tree, so the above code is incomplete because it 
> effectively drops the DEFAULT rules. The second problem is with the call to 
> {{sourcePerm.getGroupAction()}}, which is incorrect in the case of extended 
> ACLs. When extended ACLs are used the GROUP permission is replaced with the 
> extended ACL mask. So the above code will apply the wrong permissions to the 
> GROUP. Instead the correct GROUP permissions now need to be pulled from the 
> AclEntry as returned by {{getAclStatus().getEntries()}}. See the 
> implementation of the new method {{getDefaultAclEntries}} for details.
> Similar issues exist with the HCatalog API. None of the API accounts for 
> setting extended ACLs on the directory sub-tree. The changes to the HCatalog 
> API allow the extended ACLs to be passed into the required methods similar to 
> how basic permissions are passed in. When building the directory sub-tree the 
> extended ACLs of the table directory are inherited by all sub-directories, 
> including the DEFAULT rules.
> Replicating the problem:
> Create a table to write data into (I will use acl_test as the destination and 
> words_text as the source) and set the ACLs as follows:
> {noformat}
> $ hdfs dfs -setfacl -m 
> default:user::rwx,default:group::r-x,default:mask::rwx,default:user:hdfs:rwx,group::r-x,user:hdfs:rwx
>  /user/cdrome/hive/acl_test
> $ hdfs dfs -ls -d /user/cdrome/hive/acl_test
> drwxrwx---+  - cdrome hdfs  0 2016-07-13 20:36 
> /user/cdrome/hive/acl_test
> $ hdfs dfs -getfacl -R /user/cdrome/hive/acl_test
> # file: /user/cdrome/hive/acl_test
> # owner: cdrome
> # group: hdfs
> user::rwx
> user:hdfs:rwx
> group::r-x
> mask::rwx
> other::---
> default:user::rwx
> default:user:hdfs:rwx
> default:group::r-x
> default:mask::rwx
> default:other::---
> {noformat}
> Note that the basic GROUP permission is set to {{rwx}} after setting the 
> ACLs. The ACLs explicitly set the DEFAULT rules and a rule specifically for 
> the {{hdfs}} user.
> Run the following query to populate the table:
> {noformat}
> insert into acl_test partition (dt='a', ds='b') select a, b from words_text 
> where dt = 'c';
> {noformat}
> Note that words_text only has a single partition key.
> Now examine the ACLs for the resulting directories:
> {noformat}
> $ hdfs dfs -getfacl -R /user/cdrome/hive/acl_test
> # file: /user/cdrome/hive/acl_test
> # owner: cdrome
> # group: hdfs
> user::rwx
> user:hdfs:rwx
> group::r-x
> 

[jira] [Commented] (HIVE-15099) PTFOperator.PTFInvocation didn't properly reset the input partition

2016-10-31 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623153#comment-15623153
 ] 

Hive QA commented on HIVE-15099:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12836191/HIVE-15099.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 10627 tests passed

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1888/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1888/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1888/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12836191 - PreCommit-HIVE-Build

> PTFOperator.PTFInvocation didn't properly reset the input partition
> ---
>
> Key: HIVE-15099
> URL: https://issues.apache.org/jira/browse/HIVE-15099
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, PTF-Windowing
>Affects Versions: 1.3.0, 1.2.1, 2.1.0, 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15099.1.patch
>
>
> There is an issue with PTFOperator.PTFInvocation where the inputPart is not 
> reset properly. The inputPart has been closed and its content (member 
> variables) has been cleaned up, but since itself is not nullified, it's 
> reused in the next round and caused NPE issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15041) Specify GCE network name on Hive ptest

2016-10-31 Thread JIRA

[ 
https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623096#comment-15623096
 ] 

Sergio Peña commented on HIVE-15041:


The patch has been running on ptest since I created this jira, but I don't see 
any difference on fixing the issues. There was one thing I added to the patch 
that I haven't updated, but it will not add any improvement. I will include the 
line, and commit it just to keep it in the branch for future issues with the 
network.

Another thing while debugging is that seems there are issues with IPs already 
used before. Ptest saves a list of Hosts IPs that are destroyed when they are 
not used (normal operation), but then for the next batch of Hosts created, GCE 
is giving some IPs that we already saved in a terminated list. So far is good, 
but the problem comes when a ptest background check runs (every 1h) that looks 
into those "already" terminated hosts, and it destroys them again. I'll dig 
more into this issue and create a jira. I haven't had too much time to work on 
this though.

> Specify GCE network name on Hive ptest
> --
>
> Key: HIVE-15041
> URL: https://issues.apache.org/jira/browse/HIVE-15041
> Project: Hive
>  Issue Type: Task
>  Components: Hive
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-15041.1.patch
>
>
> NO PRECOMMIT TESTS
> A new option on cloudhost.properties should be added to specify the GCE 
> network name:
> # GCE network option
> network = 
> https://www.googleapis.com/compute/v1/projects//global/networks/default



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-14933) include argparse with LLAP scripts to support antique Python versions

2016-10-31 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-14933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-14933:

   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Committed to master

> include argparse with LLAP scripts to support antique Python versions
> -
>
> Key: HIVE-14933
> URL: https://issues.apache.org/jira/browse/HIVE-14933
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 2.2.0
>
> Attachments: HIVE-14933.01.patch, HIVE-14933.02.patch, 
> HIVE-14933.patch
>
>
> The module is a standalone file, and it's under Python license that is 
> compatible with Apache. In the long term we should probably just move 
> LlapServiceDriver code entirely to Java, as right now it's a combination of 
> part-py, part-java.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-14946) Optimizations

2016-10-31 Thread Eugene Koifman (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-14946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-14946:
--
Description: 
For example, if there is only WHEN NOT MATCHED clause, the base generating 
expression can be an INNER JOIN

Generally, we should make sure the optimizer is able to work with the plan for 
Merge statement

Various WHEN clauses can have "extra" predicates.  In some cases they may be 
pushable.

The "source" can be an arbitrary expression - in particular it may include 
joins which together with the join introduced by Merge itself may need to be 
reordered.

  was:
For example, if there is only WHEN NOT MATCHED clause, the base generating 
expression can be an INNER JOIN

Generally, we should make sure the optimizer is able to work with the plan for 
Merge statement


> Optimizations
> -
>
> Key: HIVE-14946
> URL: https://issues.apache.org/jira/browse/HIVE-14946
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Planning, Transactions
>Reporter: Eugene Koifman
>
> For example, if there is only WHEN NOT MATCHED clause, the base generating 
> expression can be an INNER JOIN
> Generally, we should make sure the optimizer is able to work with the plan 
> for Merge statement
> Various WHEN clauses can have "extra" predicates.  In some cases they may be 
> pushable.
> The "source" can be an arbitrary expression - in particular it may include 
> joins which together with the join introduced by Merge itself may need to be 
> reordered.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15085) Reduce the memory used by unit tests, MiniCliDriver, MiniLlapLocal, MiniSpark

2016-10-31 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623046#comment-15623046
 ] 

Hive QA commented on HIVE-15085:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12835680/HIVE-15085.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10627 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic] 
(batchId=131)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] 
(batchId=91)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1887/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1887/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1887/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12835680 - PreCommit-HIVE-Build

> Reduce the memory used by unit tests, MiniCliDriver, MiniLlapLocal, MiniSpark
> -
>
> Key: HIVE-15085
> URL: https://issues.apache.org/jira/browse/HIVE-15085
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-15085.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15041) Specify GCE network name on Hive ptest

2016-10-31 Thread Siddharth Seth (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622987#comment-15622987
 ] 

Siddharth Seth commented on HIVE-15041:
---

[~spena] - should this be committed. Looks like ptest needs a restart fairly 
often rightnow. (HIVE-15041)

> Specify GCE network name on Hive ptest
> --
>
> Key: HIVE-15041
> URL: https://issues.apache.org/jira/browse/HIVE-15041
> Project: Hive
>  Issue Type: Task
>  Components: Hive
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-15041.1.patch
>
>
> NO PRECOMMIT TESTS
> A new option on cloudhost.properties should be added to specify the GCE 
> network name:
> # GCE network option
> network = 
> https://www.googleapis.com/compute/v1/projects//global/networks/default



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15039) A better job monitor console output for HoS

2016-10-31 Thread Siddharth Seth (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622986#comment-15622986
 ] 

Siddharth Seth commented on HIVE-15039:
---

Restarted. HIVE-15041 will likely fix this.

> A better job monitor console output for HoS
> ---
>
> Key: HIVE-15039
> URL: https://issues.apache.org/jira/browse/HIVE-15039
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-15039.1.patch, HIVE-15039.2.patch, example 
> screenshot.png, with patch.png
>
>
> When there're many stages, it's very difficult to read the console output of 
> job progress of HoS. Attached screenshot is an example.
> We may learn from HoT as it does much better than HoS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15084) Flaky test: TestMiniTezCliDriver:explainanalyze_2, 3, 4

2016-10-31 Thread Siddharth Seth (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth updated HIVE-15084:
--
Summary: Flaky test: TestMiniTezCliDriver:explainanalyze_2, 3, 4  (was: 
Flaky test: TestMiniTezCliDriver:explainanalyze_2)

> Flaky test: TestMiniTezCliDriver:explainanalyze_2, 3, 4
> ---
>
> Key: HIVE-15084
> URL: https://issues.apache.org/jira/browse/HIVE-15084
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15094) Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types

2016-10-31 Thread JIRA

[ 
https://issues.apache.org/jira/browse/HIVE-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622943#comment-15622943
 ] 

Sergio Peña commented on HIVE-15094:


[~jcamachorodriguez] Sorry, my mistake. The commit is not reverted from master, 
so that's why the tests are not failing there.

> Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types
> -
>
> Key: HIVE-15094
> URL: https://issues.apache.org/jira/browse/HIVE-15094
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sergio Peña
>Assignee: Jesus Camacho Rodriguez
>
> Several tests failures related to schema evolution are happening on 
> branch-2.1 due to a patch reverted in the past.
> {noformat}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> 

[jira] [Updated] (HIVE-15099) PTFOperator.PTFInvocation didn't properly reset the input partition

2016-10-31 Thread Wei Zheng (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-15099:
-
Attachment: HIVE-15099.1.patch

> PTFOperator.PTFInvocation didn't properly reset the input partition
> ---
>
> Key: HIVE-15099
> URL: https://issues.apache.org/jira/browse/HIVE-15099
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, PTF-Windowing
>Affects Versions: 1.3.0, 1.2.1, 2.1.0, 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15099.1.patch
>
>
> There is an issue with PTFOperator.PTFInvocation where the inputPart is not 
> reset properly. The inputPart has been closed and its content (member 
> variables) has been cleaned up, but since itself is not nullified, it's 
> reused in the next round and caused NPE issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15099) PTFOperator.PTFInvocation didn't properly reset the input partition

2016-10-31 Thread Wei Zheng (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-15099:
-
Status: Patch Available  (was: Open)

> PTFOperator.PTFInvocation didn't properly reset the input partition
> ---
>
> Key: HIVE-15099
> URL: https://issues.apache.org/jira/browse/HIVE-15099
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, PTF-Windowing
>Affects Versions: 2.1.0, 1.2.1, 1.3.0, 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15099.1.patch
>
>
> There is an issue with PTFOperator.PTFInvocation where the inputPart is not 
> reset properly. The inputPart has been closed and its content (member 
> variables) has been cleaned up, but since itself is not nullified, it's 
> reused in the next round and caused NPE issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15099) PTFOperator.PTFInvocation didn't properly reset the input partition

2016-10-31 Thread Wei Zheng (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622899#comment-15622899
 ] 

Wei Zheng commented on HIVE-15099:
--

Sample query that can reproduce the issue
{code}
SELECT 1
FROM ( 
  SELECT   a,
   b, 
   row_number() OVER ( partition BY a ORDER BY c DESC ) AS rank
  FROM t1) x 
JOIN 
 ( 
  SELECT   a, 
   row_number() OVER ( partition BY a ORDER BY c DESC ) AS rank
  FROM t2) y 
ON y.a=x.a
LEFT OUTER JOIN 
 ( 
  SELECT   a, 
   row_number() OVER ( partition BY a ORDER BY c DESC ) AS rank
  FROM t3) z
ON y.a=z.a
WHERE x.rank = 1 
AND y.rank=1 
{code}

> PTFOperator.PTFInvocation didn't properly reset the input partition
> ---
>
> Key: HIVE-15099
> URL: https://issues.apache.org/jira/browse/HIVE-15099
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, PTF-Windowing
>Affects Versions: 1.3.0, 1.2.1, 2.1.0, 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>
> There is an issue with PTFOperator.PTFInvocation where the inputPart is not 
> reset properly. The inputPart has been closed and its content (member 
> variables) has been cleaned up, but since itself is not nullified, it's 
> reused in the next round and caused NPE issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15098) SELECT DISTINCT w/o CBO produces incorrect results

2016-10-31 Thread Chao Sun (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Sun updated HIVE-15098:

Description: 
For the following simple (yet nonsense) query:
{code}
SELECT  DISTINCT 
UNIX_TIMESTAMP('2016-10-02','-mm-dd') AS col1,
UNIX_TIMESTAMP('2016-10-02','-MM-dd') AS col2
FROM 
default.dummy;
{code}

With CBO, this generate correct results:
{code}
+-+-+
|col1 |col2 |
+-+-+
| 1451722200  | 1475391600  |
+-+-+
{code}

However, if CBO is turned off, the result no longer make sense:
{code}
+---+---+
| col1  | col2  |
+---+---+
| NULL  | NULL  |
+---+---+
{code}

  was:
For the following simple query:
{code}
SELECT  DISTINCT 
UNIX_TIMESTAMP('2016-10-02','-mm-dd') AS col1,
UNIX_TIMESTAMP('2016-10-02','-MM-dd') AS col2
FROM 
default.dummy;
{code}

With CBO, this generate correct results:
{code}
+-+-+
|col1 |col2 |
+-+-+
| 1451722200  | 1475391600  |
+-+-+
{code}

However, if CBO is turned off, the result no longer make sense:
{code}
+---+---+
| col1  | col2  |
+---+---+
| NULL  | NULL  |
+---+---+
{code}


> SELECT DISTINCT w/o CBO produces incorrect results
> --
>
> Key: HIVE-15098
> URL: https://issues.apache.org/jira/browse/HIVE-15098
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Reporter: Chao Sun
>
> For the following simple (yet nonsense) query:
> {code}
> SELECT  DISTINCT 
> UNIX_TIMESTAMP('2016-10-02','-mm-dd') AS col1,
> UNIX_TIMESTAMP('2016-10-02','-MM-dd') AS col2
> FROM 
> default.dummy;
> {code}
> With CBO, this generate correct results:
> {code}
> +-+-+
> |col1 |col2 |
> +-+-+
> | 1451722200  | 1475391600  |
> +-+-+
> {code}
> However, if CBO is turned off, the result no longer make sense:
> {code}
> +---+---+
> | col1  | col2  |
> +---+---+
> | NULL  | NULL  |
> +---+---+
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15094) Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types

2016-10-31 Thread Jesus Camacho Rodriguez (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622818#comment-15622818
 ] 

Jesus Camacho Rodriguez commented on HIVE-15094:


[~spena], HIVE-13380 does not seem to be reverted from Master, can you point me 
to the commit?

> Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types
> -
>
> Key: HIVE-15094
> URL: https://issues.apache.org/jira/browse/HIVE-15094
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sergio Peña
>Assignee: Jesus Camacho Rodriguez
>
> Several tests failures related to schema evolution are happening on 
> branch-2.1 due to a patch reverted in the past.
> {noformat}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> 

[jira] [Assigned] (HIVE-15094) Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types

2016-10-31 Thread Jesus Camacho Rodriguez (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-15094:
--

Assignee: Jesus Camacho Rodriguez

> Fix test failures for 2.1.1 regarding schema evolution with DECIMAL types
> -
>
> Key: HIVE-15094
> URL: https://issues.apache.org/jira/browse/HIVE-15094
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sergio Peña
>Assignee: Jesus Camacho Rodriguez
>
> Several tests failures related to schema evolution are happening on 
> branch-2.1 due to a patch reverted in the past.
> {noformat}
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_part
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_text_vec_mapwork_table
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
> 

[jira] [Updated] (HIVE-15093) For S3-to-S3 renames, files should be moved individually rather than at a directory level

2016-10-31 Thread Sahil Takiar (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-15093:

Attachment: HIVE-15093.2.patch

> For S3-to-S3 renames, files should be moved individually rather than at a 
> directory level
> -
>
> Key: HIVE-15093
> URL: https://issues.apache.org/jira/browse/HIVE-15093
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.1.0
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-15093.1.patch, HIVE-15093.2.patch
>
>
> Hive's MoveTask uses the Hive.moveFile method to move data within a 
> distributed filesystem as well as blobstore filesystems.
> If the move is done within the same filesystem:
> 1: If the source path is a subdirectory of the destination path, files will 
> be moved one by one using a threapool of workers
> 2: If the source path is not a subdirectory of the destination path, a single 
> rename operation is used to move the entire directory
> The second option may not work well on blobstores such as S3. Renames are not 
> metadata operations and require copying all the data. Client connectors to 
> blobstores may not efficiently rename directories. Worst case, the connector 
> will copy each file one by one, sequentially rather than using a threadpool 
> of workers to copy the data (e.g. HADOOP-13600).
> Hive already has code to rename files using a threadpool of workers, but this 
> only occurs in case number 1.
> This JIRA aims to modify the code so that case 1 is triggered when copying 
> within a blobstore. The focus is on copies within a blobstore because 
> needToCopy will return true if the src and target filesystems are different, 
> in which case a different code path is triggered.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.

2016-10-31 Thread liyunzhang_intel (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622634#comment-15622634
 ] 

liyunzhang_intel edited comment on HIVE-13517 at 10/31/16 4:38 PM:
---

[~szehon]:  
{quote}
 It would be great if there could be thread-ids in the logs.

{quote}
  currently hive log4j pattern is like {noformat} 
appender.DRFA.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n{noformat}  Here 
%t means thread's name.  What you want is thread-id, is my understanding right? 
 I'm confused that why thread-name is not more suitable than thread-id as log4j 
only support thread'name not thread id.
 


was (Author: kellyzly):
[~szehon]:  
{quote}
 It would be great if there could be thread-ids in the logs.

{quote}
  currently hive log4j pattern is like {noformat} 
appender.DRFA.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n{noformat}  Here 
%t means thread's name.  What you want is thread-id, is my understanding right?
 

> Hive logs in Spark Executor and Driver should show thread-id.
> -
>
> Key: HIVE-13517
> URL: https://issues.apache.org/jira/browse/HIVE-13517
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Szehon Ho
>Assignee: liyunzhang_intel
>
> In Spark, there might be more than one task running in one executor. 
> Similarly, there may be more than one thread running in Driver.
> This makes debugging through the logs a nightmare. It would be great if there 
> could be thread-ids in the logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.

2016-10-31 Thread liyunzhang_intel (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622634#comment-15622634
 ] 

liyunzhang_intel commented on HIVE-13517:
-

[~szehon]:  
{quote}
 It would be great if there could be thread-ids in the logs.

{quote}
  currently hive log4j pattern is like {noformat} 
appender.DRFA.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n{noformat}  Here 
%t means thread's name.  What you want is thread-id, is my understanding right?
 

> Hive logs in Spark Executor and Driver should show thread-id.
> -
>
> Key: HIVE-13517
> URL: https://issues.apache.org/jira/browse/HIVE-13517
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Szehon Ho
>Assignee: liyunzhang_intel
>
> In Spark, there might be more than one task running in one executor. 
> Similarly, there may be more than one thread running in Driver.
> This makes debugging through the logs a nightmare. It would be great if there 
> could be thread-ids in the logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15056) Support index shifting for struct fields

2016-10-31 Thread Chao Sun (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Sun updated HIVE-15056:

   Resolution: Fixed
Fix Version/s: 2.1.1
   Status: Resolved  (was: Patch Available)

> Support index shifting for struct fields
> 
>
> Key: HIVE-15056
> URL: https://issues.apache.org/jira/browse/HIVE-15056
> Project: Hive
>  Issue Type: Sub-task
>  Components: File Formats, Physical Optimizer, 
> Serializers/Deserializers
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 2.1.1
>
> Attachments: HIVE-15056.1.patch, HIVE-15056.2.patch
>
>
> In HIVE-13873, the following case doesn't work:
> {code}
> select s.c from tbl
> {code}
> where {{tbl}} is of schema:
> {code}
> a int
> s struct
> {code}
> This is because currently we generate a "pruned" schema (in terms of 
> {{GroupType}}) for Parquet reader to scan the data. However, on the Hive side 
> the object inspector still uses the original schema. In particular, in this 
> case for {{s.c}} the data returned by Parquet reader is in index 0, but the 
> object inspector tries to read it in index 1. Therefore, in correct result 
> will be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15056) Support index shifting for struct fields

2016-10-31 Thread Chao Sun (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622616#comment-15622616
 ] 

Chao Sun commented on HIVE-15056:
-

Committed to the master branch. Thanks [~Ferd] for the review!

> Support index shifting for struct fields
> 
>
> Key: HIVE-15056
> URL: https://issues.apache.org/jira/browse/HIVE-15056
> Project: Hive
>  Issue Type: Sub-task
>  Components: File Formats, Physical Optimizer, 
> Serializers/Deserializers
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 2.1.1
>
> Attachments: HIVE-15056.1.patch, HIVE-15056.2.patch
>
>
> In HIVE-13873, the following case doesn't work:
> {code}
> select s.c from tbl
> {code}
> where {{tbl}} is of schema:
> {code}
> a int
> s struct
> {code}
> This is because currently we generate a "pruned" schema (in terms of 
> {{GroupType}}) for Parquet reader to scan the data. However, on the Hive side 
> the object inspector still uses the original schema. In particular, in this 
> case for {{s.c}} the data returned by Parquet reader is in index 0, but the 
> object inspector tries to read it in index 1. Therefore, in correct result 
> will be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15054) Hive insertion query execution fails on Hive on Spark

2016-10-31 Thread Aihua Xu (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622519#comment-15622519
 ] 

Aihua Xu commented on HIVE-15054:
-

[~lirui] I think what you provided is better. Just updated the comments in the 
patch.

> Hive insertion query execution fails on Hive on Spark
> -
>
> Key: HIVE-15054
> URL: https://issues.apache.org/jira/browse/HIVE-15054
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15054.1.patch, HIVE-15054.2.patch, 
> HIVE-15054.3.patch, HIVE-15054.4.patch
>
>
> The query of {{insert overwrite table tbl1}} sometimes will fail with the 
> following errors. Seems we are constructing taskAttemptId with partitionId 
> which is not unique if there are multiple attempts.
> {noformat}
> ava.lang.IllegalStateException: Hit error while closing operators - failing 
> tree: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename 
> output from: 
> hdfs://table1/.hive-staging_hive_2016-06-14_01-53-17_386_3231646810118049146-9/_task_tmp.-ext-10002/_tmp.002148_0
>  to: 
> hdfs://table1/.hive-staging_hive_2016-06-14_01-53-17_386_3231646810118049146-9/_tmp.-ext-10002/002148_0
> at 
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.close(SparkMapRecordHandler.java:202)
> at 
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.closeRecordProcessor(HiveMapFunctionResultList.java:58)
> at 
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:106)
> at 
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at 
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15054) Hive insertion query execution fails on Hive on Spark

2016-10-31 Thread Aihua Xu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-15054:

Attachment: HIVE-15054.4.patch

> Hive insertion query execution fails on Hive on Spark
> -
>
> Key: HIVE-15054
> URL: https://issues.apache.org/jira/browse/HIVE-15054
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15054.1.patch, HIVE-15054.2.patch, 
> HIVE-15054.3.patch, HIVE-15054.4.patch
>
>
> The query of {{insert overwrite table tbl1}} sometimes will fail with the 
> following errors. Seems we are constructing taskAttemptId with partitionId 
> which is not unique if there are multiple attempts.
> {noformat}
> ava.lang.IllegalStateException: Hit error while closing operators - failing 
> tree: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename 
> output from: 
> hdfs://table1/.hive-staging_hive_2016-06-14_01-53-17_386_3231646810118049146-9/_task_tmp.-ext-10002/_tmp.002148_0
>  to: 
> hdfs://table1/.hive-staging_hive_2016-06-14_01-53-17_386_3231646810118049146-9/_tmp.-ext-10002/002148_0
> at 
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.close(SparkMapRecordHandler.java:202)
> at 
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.closeRecordProcessor(HiveMapFunctionResultList.java:58)
> at 
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:106)
> at 
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at 
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15054) Hive insertion query execution fails on Hive on Spark

2016-10-31 Thread Aihua Xu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-15054:

Status: Patch Available  (was: In Progress)

patch-4: update the comment.

> Hive insertion query execution fails on Hive on Spark
> -
>
> Key: HIVE-15054
> URL: https://issues.apache.org/jira/browse/HIVE-15054
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15054.1.patch, HIVE-15054.2.patch, 
> HIVE-15054.3.patch, HIVE-15054.4.patch
>
>
> The query of {{insert overwrite table tbl1}} sometimes will fail with the 
> following errors. Seems we are constructing taskAttemptId with partitionId 
> which is not unique if there are multiple attempts.
> {noformat}
> ava.lang.IllegalStateException: Hit error while closing operators - failing 
> tree: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename 
> output from: 
> hdfs://table1/.hive-staging_hive_2016-06-14_01-53-17_386_3231646810118049146-9/_task_tmp.-ext-10002/_tmp.002148_0
>  to: 
> hdfs://table1/.hive-staging_hive_2016-06-14_01-53-17_386_3231646810118049146-9/_tmp.-ext-10002/002148_0
> at 
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.close(SparkMapRecordHandler.java:202)
> at 
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.closeRecordProcessor(HiveMapFunctionResultList.java:58)
> at 
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:106)
> at 
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at 
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15054) Hive insertion query execution fails on Hive on Spark

2016-10-31 Thread Aihua Xu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-15054:

Status: In Progress  (was: Patch Available)

> Hive insertion query execution fails on Hive on Spark
> -
>
> Key: HIVE-15054
> URL: https://issues.apache.org/jira/browse/HIVE-15054
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15054.1.patch, HIVE-15054.2.patch, 
> HIVE-15054.3.patch, HIVE-15054.4.patch
>
>
> The query of {{insert overwrite table tbl1}} sometimes will fail with the 
> following errors. Seems we are constructing taskAttemptId with partitionId 
> which is not unique if there are multiple attempts.
> {noformat}
> ava.lang.IllegalStateException: Hit error while closing operators - failing 
> tree: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename 
> output from: 
> hdfs://table1/.hive-staging_hive_2016-06-14_01-53-17_386_3231646810118049146-9/_task_tmp.-ext-10002/_tmp.002148_0
>  to: 
> hdfs://table1/.hive-staging_hive_2016-06-14_01-53-17_386_3231646810118049146-9/_tmp.-ext-10002/002148_0
> at 
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.close(SparkMapRecordHandler.java:202)
> at 
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.closeRecordProcessor(HiveMapFunctionResultList.java:58)
> at 
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:106)
> at 
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at 
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-alpha1

2016-10-31 Thread JIRA

[ 
https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622460#comment-15622460
 ] 

Sergio Peña commented on HIVE-15016:


Update:

Hive cannot compile with Hadoop 3.0.0-alpha1 due to some API incompatibilities. 
These are already fixed on 3.0.0-alpha2, so I'll wait until alpha2 is released, 
and then continue with this testing.

> Run tests with Hadoop 3.0.0-alpha1
> --
>
> Key: HIVE-15016
> URL: https://issues.apache.org/jira/browse/HIVE-15016
> Project: Hive
>  Issue Type: Task
>  Components: Hive
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>
> Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run 
> tests against this new version before GA.
> We should start running tests with Hive to validate compatibility against 
> Hadoop 3.0.
> NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 
> GA is released.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-14476) Fix logging issue for branch-1

2016-10-31 Thread JIRA

[ 
https://issues.apache.org/jira/browse/HIVE-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622424#comment-15622424
 ] 

Sergio Peña commented on HIVE-14476:


These 3 tests may be fixed with HIVE-13587
{noformat}
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore
 (batchId=426)
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore
 (batchId=426)
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin (batchId=423)
{noformat}

These 3 tests are also failing on master
{noformat}
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0]
 (batchId=250)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] (batchId=250)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] (batchId=250
{noformat}

I don't know what could be causing the failures on the other tests. Are they 
also failing locally?
If they do fail locally, I usually try to search for the commit that is causing 
those failures. That commit can give you a better idea how to fix it. I use 
'git bisect' to investigate the commit.

> Fix logging issue for branch-1
> --
>
> Key: HIVE-14476
> URL: https://issues.apache.org/jira/browse/HIVE-14476
> Project: Hive
>  Issue Type: Bug
>Reporter: Tao Li
>Assignee: Tao Li
> Attachments: HIVE-14476.1-branch-1.2.patch
>
>
> This issue is from branch-1 code when we decide if a log entry is an 
> operational log or not (the operational logs are visible to the client). The 
> problem is that the code is checking the logging mode at the beginning of the 
> decide() method, while the logging mode is updated after that check. Due to 
> this issue, we ran into an issue that an operational log could be filtered 
> out if it's the very first log being checked from the this method. As a 
> result, that particular log is not showing up for the end user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15039) A better job monitor console output for HoS

2016-10-31 Thread Ferdinand Xu (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622429#comment-15622429
 ] 

Ferdinand Xu commented on HIVE-15039:
-

Hi [~sseth], looks like precommit is still falling as the error: {noformat}> 
Exception in thread "main" java.lang.RuntimeException: 404 Not Found
> at org.apache.hive.ptest.api.client.PTestClient.
{noformat} Do you know how to restart the ptest server to fix it? Thank you! 

> A better job monitor console output for HoS
> ---
>
> Key: HIVE-15039
> URL: https://issues.apache.org/jira/browse/HIVE-15039
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-15039.1.patch, HIVE-15039.2.patch, example 
> screenshot.png, with patch.png
>
>
> When there're many stages, it's very difficult to read the console output of 
> job progress of HoS. Attached screenshot is an example.
> We may learn from HoT as it does much better than HoS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15061) Metastore types are sometimes case sensitive

2016-10-31 Thread Mohit Sabharwal (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622361#comment-15622361
 ] 

Mohit Sabharwal commented on HIVE-15061:


LGTM +1

(I double-checked other places where coltypes are set
like StatObjectConverter - found that we are converting
to lower case there already)

> Metastore types are sometimes case sensitive
> 
>
> Key: HIVE-15061
> URL: https://issues.apache.org/jira/browse/HIVE-15061
> Project: Hive
>  Issue Type: Bug
>  Components: API
>Affects Versions: 1.1.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Chaoyu Tang
> Attachments: HIVE-15061.1.patch, HIVE-15061.1.patch, HIVE-15061.patch
>
>
> Impala recently encountered an issue with the metastore 
> ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column 
> stats would get dropped when adding a column to a table.
> The reason seems to be that Hive does a case sensitive check on the column 
> stats types during an "alter table" and expects the types to be all lower 
> case. This case sensitive check doesn't appear to happen when the stats are 
> set in the first place.
> We're solving this on the Impala end by storing types in the metastore as all 
> lower case, but Hive's behavior here is very confusing. It should either 
> always be case sensitive, so that you can't create column stats with types 
> that Hive considers invalid, or it should never be case sensitive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15039) A better job monitor console output for HoS

2016-10-31 Thread Rui Li (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622333#comment-15622333
 ] 

Rui Li commented on HIVE-15039:
---

Thanks [~Ferd]. Do you know how to restart it?

> A better job monitor console output for HoS
> ---
>
> Key: HIVE-15039
> URL: https://issues.apache.org/jira/browse/HIVE-15039
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-15039.1.patch, HIVE-15039.2.patch, example 
> screenshot.png, with patch.png
>
>
> When there're many stages, it's very difficult to read the console output of 
> job progress of HoS. Attached screenshot is an example.
> We may learn from HoT as it does much better than HoS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-14910) Flaky test: TestSparkClient.testJobSubmission

2016-10-31 Thread Rui Li (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622330#comment-15622330
 ] 

Rui Li commented on HIVE-14910:
---

It makes sense to me. Have a question: is the Listener only used for tests?

> Flaky test: TestSparkClient.testJobSubmission
> -
>
> Key: HIVE-14910
> URL: https://issues.apache.org/jira/browse/HIVE-14910
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-14910.1.patch, HIVE-14910.patch
>
>
> Have seen this fail in multiple runs (not consistently)
> e.g. https://builds.apache.org/job/PreCommit-HIVE-Build/1426/testReport/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15097) Error: java.lang.RuntimeException: Error in configuring object Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

2016-10-31 Thread chen cong (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chen cong updated HIVE-15097:
-
Description: 
hive> CREATE TABLE user_tag_pre AS 
> SELECT A.userid , A.tag , A.single_tag_sumcount/B.sum_tag_sumcount_avg as 
pre
> FROM  user_tag_sumcount A , user_tag_avg B
> WHERE A.userid = B.userid;
Query ID = ubuntu_20161031134731_bdbfeaa6-5687-460c-b331-86ce3237aa29
Total jobs = 2
Stage-7 is selected by condition resolver.
Stage-1 is filtered out by condition resolver.
Execution log at: 
/tmp/ubuntu/ubuntu_20161031134731_bdbfeaa6-5687-460c-b331-86ce3237aa29.log
2016-10-31 13:47:34 Starting to launch local task to process map join;  
maximum memory = 477626368
2016-10-31 13:47:37 Processing rows:20  Hashtable size: 19  
Memory usage:   134844656   percentage: 0.282
2016-10-31 13:47:38 Processing rows:30  Hashtable size: 29  
Memory usage:   121492112   percentage: 0.254
2016-10-31 13:47:38 Processing rows:40  Hashtable size: 39  
Memory usage:   122277256   percentage: 0.256
2016-10-31 13:47:38 Processing rows:50  Hashtable size: 49  
Memory usage:   151827872   percentage: 0.318
2016-10-31 13:47:40 Processing rows:60  Hashtable size: 59  
Memory usage:   179637408   percentage: 0.376
2016-10-31 13:47:40 Dump the side-table for tag: 1 with group count: 675226 
into file: 
file:/tmp/ubuntu/193dc11b-1e25-46a0-b124-210a7ffde96c/hive_2016-10-31_13-47-31_406_4438237799451830529-1/-local-10003/HashTable-Stage-4/MapJoin-mapfile01--.hashtable
2016-10-31 13:47:41 Uploaded 1 File to: 
file:/tmp/ubuntu/193dc11b-1e25-46a0-b124-210a7ffde96c/hive_2016-10-31_13-47-31_406_4438237799451830529-1/-local-10003/HashTable-Stage-4/MapJoin-mapfile01--.hashtable
 (22302233 bytes)
2016-10-31 13:47:41 End of local task; Time Taken: 6.116 sec.
Execution completed successfully
MapredLocal task succeeded
Launching Job 2 out of 2
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1477875480008_0012, Tracking URL = 
http://master:8088/proxy/application_1477875480008_0012/
Kill Command = /home/ubuntu/cloud/hadoop-2.7.2/bin/hadoop job  -kill 
job_1477875480008_0012
Hadoop job information for Stage-4: number of mappers: 1; number of reducers: 0
2016-10-31 13:47:49,787 Stage-4 map = 0%,  reduce = 0%
2016-10-31 13:48:50,370 Stage-4 map = 0%,  reduce = 0%
2016-10-31 13:49:50,768 Stage-4 map = 0%,  reduce = 0%
2016-10-31 13:50:35,748 Stage-4 map = 100%,  reduce = 0%
Ended Job = job_1477875480008_0012 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1477875480008_0012_m_00 (and more) from job 
job_1477875480008_0012

Task with the most failures(4):
-
Task ID:
  task_1477875480008_0012_m_00

URL:
  
http://master:8088/taskdetails.jsp?jobid=job_1477875480008_0012=task_1477875480008_0012_m_00
-
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: Error in configuring object
at 
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at 
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at 
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:449)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at 
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at 
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at 
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 

[jira] [Commented] (HIVE-14909) Preserve the location of table created with the location clause in table rename

2016-10-31 Thread Chaoyu Tang (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622212#comment-15622212
 ] 

Chaoyu Tang commented on HIVE-14909:


Thanks, [~leftylev]

> Preserve the location of table created with the location clause in table 
> rename
> ---
>
> Key: HIVE-14909
> URL: https://issues.apache.org/jira/browse/HIVE-14909
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Affects Versions: 1.1.0
>Reporter: Adriano
>Assignee: Chaoyu Tang
> Fix For: 2.2.0
>
> Attachments: HIVE-14909.1.patch, HIVE-14909.1.patch, 
> HIVE-14909.2.patch, HIVE-14909.3.patch, HIVE-14909.patch, HIVE-14909.patch
>
>
> Alter Table operation for db_enc.rename_test failed to move data due to: 
> '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption 
> zone.'
> When Hive renames a managed table, it always creates the new renamed table 
> directory under its database directory in order to keep a db/table hierarchy. 
> In this case, the renamed table directory is created under "default db" 
> directory "hive/warehouse/". When Hive renames a managed table, it always 
> creates the new renamed table directory under its database directory in order 
> to keep a db/table hierarchy. In this case, the renamed table directory is 
> created under "default' db directory typically set as /hive/warehouse/ . 
> This error doesn't appear if first create a database which points to a 
> directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't 
> have this problem. For example, 
> create database db_enc location '/hdfs/encrypted_path/db_enc; 
> use db_enc; 
> create table rename_test (...) location 
> '/hdfs/encrypted_path/db_enc/rename_test'; 
> alter table rename_test rename to test_rename; 
> The renamed test_rename directory is created under 
> /hdfs/encrypted_path/db_enc. 
> Considering that the encryption of a filesystem is part of the evolution 
> hardening of a system (where the system and the data contained can already 
> exists) and a db can be already created without location set (because it is 
> not strictly required)and the default db is outside the same encryption zone 
> (or in a no-encryption zone) the alter table rename operation will fail.
> Improvement:
> Preserve the "parent location" of the table when an "alter table  
> rename to " is submitted (the case when the db location is not 
> specified and the Hive defult db is outside the same encrypted zone).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15061) Metastore types are sometimes case sensitive

2016-10-31 Thread Chaoyu Tang (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622176#comment-15622176
 ] 

Chaoyu Tang commented on HIVE-15061:


The failed test seems not related to the patch. [~aihuaxu], [~mohitsabharwal] 
could you review it, thanks.

> Metastore types are sometimes case sensitive
> 
>
> Key: HIVE-15061
> URL: https://issues.apache.org/jira/browse/HIVE-15061
> Project: Hive
>  Issue Type: Bug
>  Components: API
>Affects Versions: 1.1.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Chaoyu Tang
> Attachments: HIVE-15061.1.patch, HIVE-15061.1.patch, HIVE-15061.patch
>
>
> Impala recently encountered an issue with the metastore 
> ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column 
> stats would get dropped when adding a column to a table.
> The reason seems to be that Hive does a case sensitive check on the column 
> stats types during an "alter table" and expects the types to be all lower 
> case. This case sensitive check doesn't appear to happen when the stats are 
> set in the first place.
> We're solving this on the Impala end by storing types in the metastore as all 
> lower case, but Hive's behavior here is very confusing. It should either 
> always be case sensitive, so that you can't create column stats with types 
> that Hive considers invalid, or it should never be case sensitive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (HIVE-15061) Metastore types are sometimes case sensitive

2016-10-31 Thread Chaoyu Tang (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609998#comment-15609998
 ] 

Chaoyu Tang edited comment on HIVE-15061 at 10/31/16 1:37 PM:
--

This patch is to fix the following issues:
1. alterTableUpdateTableColumnStats and updatePartColumnStatsForAlterColumns 
did not ignore the case when they compare the columns to determine if their 
stats should be updated or deleted
2. update_table_column_statistics/update_partition_column_statistics allow the 
uppercase column type to be populated to HMS backend DB


was (Author: ctang.ma):
This patch is to fix the following issues:
1. alterTableUpdateTableColumnStats and updatePartColumnStatsForAlterColumns 
did not ignore the case when they compare the columns to determine if their 
stats should be updated or deleted
2. HMS alterTable allows the uppercase column name and type to be populated to 
HMS backend DB
3. update_table_column_statistics/update_partition_column_statistics allow the 
uppercase column type to be populated to HMS backend DB

> Metastore types are sometimes case sensitive
> 
>
> Key: HIVE-15061
> URL: https://issues.apache.org/jira/browse/HIVE-15061
> Project: Hive
>  Issue Type: Bug
>  Components: API
>Affects Versions: 1.1.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Chaoyu Tang
> Attachments: HIVE-15061.1.patch, HIVE-15061.1.patch, HIVE-15061.patch
>
>
> Impala recently encountered an issue with the metastore 
> ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column 
> stats would get dropped when adding a column to a table.
> The reason seems to be that Hive does a case sensitive check on the column 
> stats types during an "alter table" and expects the types to be all lower 
> case. This case sensitive check doesn't appear to happen when the stats are 
> set in the first place.
> We're solving this on the Impala end by storing types in the metastore as all 
> lower case, but Hive's behavior here is very confusing. It should either 
> always be case sensitive, so that you can't create column stats with types 
> that Hive considers invalid, or it should never be case sensitive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15096) hplsql registerUDF conflicts with pom.xml

2016-10-31 Thread Dmitry Tolpeko (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15621622#comment-15621622
 ] 

Dmitry Tolpeko commented on HIVE-15096:
---

-1 why to hardcode the version hive-hplsql-2.2.0-SNAPSHOT.jar 
you will need to change this file with every Hive realease.

> hplsql registerUDF conflicts with pom.xml
> -
>
> Key: HIVE-15096
> URL: https://issues.apache.org/jira/browse/HIVE-15096
> Project: Hive
>  Issue Type: Bug
>  Components: hpl/sql
>Affects Versions: 2.0.0, 2.1.0, 2.0.1
>Reporter: Fei Hui
> Fix For: 2.2.0
>
> Attachments: HIVE-15096.patch
>
>
> in hplsql code, registerUDF code is
> sql.add("ADD JAR " + dir + "hplsql.jar");
> sql.add("ADD JAR " + dir + "antlr-runtime-4.5.jar");
> sql.add("ADD FILE " + dir + Conf.SITE_XML);
> but pom configufation is
>   
> org.apache.hive
> hive
> 2.2.0-SNAPSHOT
> ../pom.xml
>   
>   hive-hplsql
>   jar
>   Hive HPL/SQL
> 
>org.antlr
>antlr4-runtime
>4.5
> 
> when run hplsql , errors occur as below
>  Error while processing statement: 
> /opt/apps/apache-hive-2.0.0-bin/lib/hplsql.jar does not exist



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15039) A better job monitor console output for HoS

2016-10-31 Thread Ferdinand Xu (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15621528#comment-15621528
 ] 

Ferdinand Xu commented on HIVE-15039:
-

Ptest server needs to be restarted for precommit.

> A better job monitor console output for HoS
> ---
>
> Key: HIVE-15039
> URL: https://issues.apache.org/jira/browse/HIVE-15039
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-15039.1.patch, HIVE-15039.2.patch, example 
> screenshot.png, with patch.png
>
>
> When there're many stages, it's very difficult to read the console output of 
> job progress of HoS. Attached screenshot is an example.
> We may learn from HoT as it does much better than HoS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-10-31 Thread Rui Li (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-15039:
--
Attachment: (was: HIVE-15039.2.patch)

> A better job monitor console output for HoS
> ---
>
> Key: HIVE-15039
> URL: https://issues.apache.org/jira/browse/HIVE-15039
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-15039.1.patch, HIVE-15039.2.patch, example 
> screenshot.png, with patch.png
>
>
> When there're many stages, it's very difficult to read the console output of 
> job progress of HoS. Attached screenshot is an example.
> We may learn from HoT as it does much better than HoS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-10-31 Thread Rui Li (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-15039:
--
Attachment: HIVE-15039.2.patch

> A better job monitor console output for HoS
> ---
>
> Key: HIVE-15039
> URL: https://issues.apache.org/jira/browse/HIVE-15039
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-15039.1.patch, HIVE-15039.2.patch, 
> HIVE-15039.2.patch, example screenshot.png, with patch.png
>
>
> When there're many stages, it's very difficult to read the console output of 
> job progress of HoS. Attached screenshot is an example.
> We may learn from HoT as it does much better than HoS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-15081) RetryingMetaStoreClient.getProxy(HiveConf, Boolean) doesn't match constructor of HiveMetaStoreClient

2016-10-31 Thread Rui Li (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-15081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-15081:
--
   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Committed to master. Thanks [~sershe] for the review.

> RetryingMetaStoreClient.getProxy(HiveConf, Boolean) doesn't match constructor 
> of HiveMetaStoreClient
> 
>
> Key: HIVE-15081
> URL: https://issues.apache.org/jira/browse/HIVE-15081
> Project: Hive
>  Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Fix For: 2.2.0
>
> Attachments: HIVE-15081.1.patch
>
>
> Calling RetryingMetaStoreClient.getProxy(HiveConf, Boolean) will result in 
> error
> {noformat}
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1661)
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:81)
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:131)
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:87)
> Caused by: java.lang.NoSuchMethodException: 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(org.apache.hadoop.hive.conf.HiveConf,
>  java.lang.Boolean)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-14950) Support integer data type

2016-10-31 Thread Lefty Leverenz (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-14950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15621493#comment-15621493
 ] 

Lefty Leverenz commented on HIVE-14950:
---

Thanks [~alangates], I agree about the conversions table.  But version 
information was needed so I added it in the Column Types section.

> Support integer data type
> -
>
> Key: HIVE-14950
> URL: https://issues.apache.org/jira/browse/HIVE-14950
> Project: Hive
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-14950.1.patch, HIVE-14950.2.patch
>
>
> maybe its just me bumping into this difference again and again...
> but it's in the sql2011 standard... 
> adding an alias for int would be easy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15054) Hive insertion query execution fails on Hive on Spark

2016-10-31 Thread Rui Li (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15621481#comment-15621481
 ] 

Rui Li commented on HIVE-15054:
---

Thanks [~aihuaxu] for the investigation and update!
The patch looks good. But I find the comments a little bit confusing. How about 
something like this
{code}
// Hive requires this TaskAttemptId to be unique. MR's TaskAttemptId is 
composed of "attempt_timestamp_jobNum_m/r_taskNum_attemptNum". The counterpart 
for Spark should be "attempt_timestamp_stageNum_m/r_partitionId_attemptNum". 
When there're multiple attempts for a task, Hive will rely on the partitionId 
to figure out if the data are duplicate or not (see 
org.apache.hadoop.hive.ql.exec.Utils.removeTempOrDuplicateFiles)  when 
collecting the final outputs
{code}

> Hive insertion query execution fails on Hive on Spark
> -
>
> Key: HIVE-15054
> URL: https://issues.apache.org/jira/browse/HIVE-15054
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15054.1.patch, HIVE-15054.2.patch, 
> HIVE-15054.3.patch
>
>
> The query of {{insert overwrite table tbl1}} sometimes will fail with the 
> following errors. Seems we are constructing taskAttemptId with partitionId 
> which is not unique if there are multiple attempts.
> {noformat}
> ava.lang.IllegalStateException: Hit error while closing operators - failing 
> tree: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename 
> output from: 
> hdfs://table1/.hive-staging_hive_2016-06-14_01-53-17_386_3231646810118049146-9/_task_tmp.-ext-10002/_tmp.002148_0
>  to: 
> hdfs://table1/.hive-staging_hive_2016-06-14_01-53-17_386_3231646810118049146-9/_tmp.-ext-10002/002148_0
> at 
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.close(SparkMapRecordHandler.java:202)
> at 
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.closeRecordProcessor(HiveMapFunctionResultList.java:58)
> at 
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:106)
> at 
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at 
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1$$anonfun$apply$15.apply(AsyncRDDActions.scala:120)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Assigned] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.

2016-10-31 Thread liyunzhang_intel (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyunzhang_intel reassigned HIVE-13517:
---

Assignee: liyunzhang_intel  (was: Szehon Ho)

> Hive logs in Spark Executor and Driver should show thread-id.
> -
>
> Key: HIVE-13517
> URL: https://issues.apache.org/jira/browse/HIVE-13517
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Szehon Ho
>Assignee: liyunzhang_intel
>
> In Spark, there might be more than one task running in one executor. 
> Similarly, there may be more than one thread running in Driver.
> This makes debugging through the logs a nightmare. It would be great if there 
> could be thread-ids in the logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-15096) hplsql registerUDF conflicts with pom.xml

2016-10-31 Thread Ferdinand Xu (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15621397#comment-15621397
 ] 

Ferdinand Xu commented on HIVE-15096:
-

LGTM +1 pending to the test.

> hplsql registerUDF conflicts with pom.xml
> -
>
> Key: HIVE-15096
> URL: https://issues.apache.org/jira/browse/HIVE-15096
> Project: Hive
>  Issue Type: Bug
>  Components: hpl/sql
>Affects Versions: 2.0.0, 2.1.0, 2.0.1
>Reporter: Fei Hui
> Fix For: 2.2.0
>
> Attachments: HIVE-15096.patch
>
>
> in hplsql code, registerUDF code is
> sql.add("ADD JAR " + dir + "hplsql.jar");
> sql.add("ADD JAR " + dir + "antlr-runtime-4.5.jar");
> sql.add("ADD FILE " + dir + Conf.SITE_XML);
> but pom configufation is
>   
> org.apache.hive
> hive
> 2.2.0-SNAPSHOT
> ../pom.xml
>   
>   hive-hplsql
>   jar
>   Hive HPL/SQL
> 
>org.antlr
>antlr4-runtime
>4.5
> 
> when run hplsql , errors occur as below
>  Error while processing statement: 
> /opt/apps/apache-hive-2.0.0-bin/lib/hplsql.jar does not exist



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


  1   2   >