[jira] [Commented] (HIVE-12812) Enable mapred.input.dir.recursive by default to support union with aggregate function
[ https://issues.apache.org/jira/browse/HIVE-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16624059#comment-16624059 ] Chaoyu Tang commented on HIVE-12812: I could not remember the exact reason why this patch was not be committed years ago. It might probably be that we were going to decommission the MR soon and there was a regression in one test case. But as a workaround, you can always set the property to true on the command (not necessary in hive-site.xml): set mapred.input.dir.recursive=true > Enable mapred.input.dir.recursive by default to support union with aggregate > function > - > > Key: HIVE-12812 > URL: https://issues.apache.org/jira/browse/HIVE-12812 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.1, 2.1.0 >Reporter: Chaoyu Tang >Priority: Major > Attachments: HIVE-12812.patch, HIVE-12812.patch, HIVE-12812.patch > > > When union remove optimization is enabled, union query with aggregate > function writes its subquery intermediate results to subdirs which needs > mapred.input.dir.recursive to be enabled in order to be fetched. This > property is not defined by default in Hive and often ignored by user, which > causes the query failure and is hard to be debugged. > So we need set mapred.input.dir.recursive to true whenever union remove > optimization is enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19117) hiveserver2 org.apache.thrift.transport.TTransportException error when running 2nd query after minute of inactivity
[
https://issues.apache.org/jira/browse/HIVE-19117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16774340#comment-16774340
]
Chaoyu Tang edited comment on HIVE-19117 at 2/21/19 5:32 PM:
-
Could it be the HS2 session was timed out? Double check property
hive.server2.idle.session.timeout, it is default set to 7 days and should not
be the problem. In the mean time, check hs2 log file to see what happened
during that period.
was (Author: ctang.ma):
Could it be the HS2 session was timed out, double check property
hive.server2.idle.session.timeout, it is default set to 7 days and should not
be the problem. In the mean time, check hs2 log file to see what happened
during that period.
> hiveserver2 org.apache.thrift.transport.TTransportException error when
> running 2nd query after minute of inactivity
> ---
>
> Key: HIVE-19117
> URL: https://issues.apache.org/jira/browse/HIVE-19117
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Metastore, Thrift API
>Affects Versions: 2.1.1
> Environment: * Hive 2.1.1 with hive.server2.transport.mode set to
> binary (sample JDBC string is jdbc:hive2://remotehost:1/default)
> * Hadoop 2.8.3
> * Metastore using MySQL
> * Java 8
>Reporter: t oo
>Priority: Blocker
>
> I make a JDBC connection from my SQL tool (ie Squirrel SQL, Oracle SQL
> Developer) to HiveServer2 (running on remote server) with port 1.
> I am able to run some queries successfully. I then do something else (not in
> the SQL tool) for 1-2minutes and then return to my SQL tool and attempt to
> run a query but I get this error:
> {code:java}
> org.apache.thrift.transport.TTransportException: java.net.SocketException:
> Software caused connection abort: socket write error{code}
> If I now disconnect and reconnect in my SQL tool I can run queries again. But
> does anyone know what HiveServer2 settings I should change to prevent the
> error? I assume something in hive-site.xml
> From the hiveserver2 logs below, can see an exact 1 minute gap from 30th min
> to 31stmin where the disconnect happens.
> {code:java}
> 2018-04-05T03:30:41,706 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Resetting thread name to HiveServer2-Handler-Pool:
> Thread-36
> 2018-04-05T03:30:41,712 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Updating thread name to
> c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
> 2018-04-05T03:30:41,712 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Resetting thread name to HiveServer2-Handler-Pool:
> Thread-36
> 2018-04-05T03:30:41,718 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Updating thread name to
> c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
> 2018-04-05T03:30:41,719 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Resetting thread name to HiveServer2-Handler-Pool:
> Thread-36
> 2018-04-05T03:31:41,232 INFO [HiveServer2-Handler-Pool: Thread-36]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [c81ec0f9-7a9d-46b6-9708-e7d78520a48a]
> 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36]
> service.CompositeService: Session closed, SessionHandle
> [c81ec0f9-7a9d-46b6-9708-e7d78520a48a], current sessions:0
> 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Updating thread name to
> c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
> 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Resetting thread name to HiveServer2-Handler-Pool:
> Thread-36
> 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Updating thread name to
> c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
> 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.HiveSessionImpl: Operation log session directory is deleted:
> /var/hive/hs2log/tmp/c81ec0f9-7a9d-46b6-9708-e7d78520a48a
> 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Resetting thread name to HiveServer2-Handler-Pool:
> Thread-36
> 2018-04-05T03:31:41,236 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Deleted directory:
> /var/hive/scratch/tmp/anonymous/c81ec0f9-7a9d-46b6-9708-e7d78520a48a on fs
> with scheme file
> 2018-04-05T03:31:41,236 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState:
[jira] [Commented] (HIVE-19117) hiveserver2 org.apache.thrift.transport.TTransportException error when running 2nd query after minute of inactivity
[
https://issues.apache.org/jira/browse/HIVE-19117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16774340#comment-16774340
]
Chaoyu Tang commented on HIVE-19117:
Could it be the HS2 session was timed out, double check property
hive.server2.idle.session.timeout, it is default set to 7 days and should not
be the problem. In the mean time, check hs2 log file to see what happened
during that period.
> hiveserver2 org.apache.thrift.transport.TTransportException error when
> running 2nd query after minute of inactivity
> ---
>
> Key: HIVE-19117
> URL: https://issues.apache.org/jira/browse/HIVE-19117
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Metastore, Thrift API
>Affects Versions: 2.1.1
> Environment: * Hive 2.1.1 with hive.server2.transport.mode set to
> binary (sample JDBC string is jdbc:hive2://remotehost:1/default)
> * Hadoop 2.8.3
> * Metastore using MySQL
> * Java 8
>Reporter: t oo
>Priority: Blocker
>
> I make a JDBC connection from my SQL tool (ie Squirrel SQL, Oracle SQL
> Developer) to HiveServer2 (running on remote server) with port 1.
> I am able to run some queries successfully. I then do something else (not in
> the SQL tool) for 1-2minutes and then return to my SQL tool and attempt to
> run a query but I get this error:
> {code:java}
> org.apache.thrift.transport.TTransportException: java.net.SocketException:
> Software caused connection abort: socket write error{code}
> If I now disconnect and reconnect in my SQL tool I can run queries again. But
> does anyone know what HiveServer2 settings I should change to prevent the
> error? I assume something in hive-site.xml
> From the hiveserver2 logs below, can see an exact 1 minute gap from 30th min
> to 31stmin where the disconnect happens.
> {code:java}
> 2018-04-05T03:30:41,706 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Resetting thread name to HiveServer2-Handler-Pool:
> Thread-36
> 2018-04-05T03:30:41,712 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Updating thread name to
> c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
> 2018-04-05T03:30:41,712 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Resetting thread name to HiveServer2-Handler-Pool:
> Thread-36
> 2018-04-05T03:30:41,718 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Updating thread name to
> c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
> 2018-04-05T03:30:41,719 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Resetting thread name to HiveServer2-Handler-Pool:
> Thread-36
> 2018-04-05T03:31:41,232 INFO [HiveServer2-Handler-Pool: Thread-36]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [c81ec0f9-7a9d-46b6-9708-e7d78520a48a]
> 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36]
> service.CompositeService: Session closed, SessionHandle
> [c81ec0f9-7a9d-46b6-9708-e7d78520a48a], current sessions:0
> 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Updating thread name to
> c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
> 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Resetting thread name to HiveServer2-Handler-Pool:
> Thread-36
> 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Updating thread name to
> c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
> 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.HiveSessionImpl: Operation log session directory is deleted:
> /var/hive/hs2log/tmp/c81ec0f9-7a9d-46b6-9708-e7d78520a48a
> 2018-04-05T03:31:41,233 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Resetting thread name to HiveServer2-Handler-Pool:
> Thread-36
> 2018-04-05T03:31:41,236 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Deleted directory:
> /var/hive/scratch/tmp/anonymous/c81ec0f9-7a9d-46b6-9708-e7d78520a48a on fs
> with scheme file
> 2018-04-05T03:31:41,236 INFO [HiveServer2-Handler-Pool: Thread-36]
> session.SessionState: Deleted directory:
> /var/hive/ec2-user/c81ec0f9-7a9d-46b6-9708-e7d78520a48a on fs with scheme file
> 2018-04-05T03:31:41,236 INFO [HiveServer2-Handler-Pool: Thread-36]
> hive.metastore: Closed a connection to metastore, current connections: 1{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-10574) Metastore to handle expired tokens inline
[ https://issues.apache.org/jira/browse/HIVE-10574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-10574: --- Assignee: (was: Chaoyu Tang) > Metastore to handle expired tokens inline > - > > Key: HIVE-10574 > URL: https://issues.apache.org/jira/browse/HIVE-10574 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Xuefu Zhang > > This is a followup for HIVE-9625. > Metastore has a garbage collection thread that removes expired tokens. > However that still leaves a window (1 hour by default) where clients could > retrieve a token that's expired or about to expire. An option is for > metastore handle expired tokens inline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-10732) Hive JDBC driver does not close operation for metadata queries
[ https://issues.apache.org/jira/browse/HIVE-10732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang reassigned HIVE-10732: -- Assignee: Chaoyu Tang > Hive JDBC driver does not close operation for metadata queries > -- > > Key: HIVE-10732 > URL: https://issues.apache.org/jira/browse/HIVE-10732 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Mala Chikka Kempanna >Assignee: Chaoyu Tang > > In following file > http://github.mtv.cloudera.com/CDH/hive/blob/cdh5-0.14.1/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java > Line 315 implemented the ResultSet.close() method. Because "DatabaseMetadata" > operation doesn't have a statement, it doesn't "close" the operation. > However, regardless whether it has a statement or not, it should close the > operation through the "stmtHandle". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10732) Hive JDBC driver does not close operation for metadata queries
[ https://issues.apache.org/jira/browse/HIVE-10732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-10732: --- Attachment: HIVE-10732.patch Promptly close the OperationHandle passed from HiveDatabaseMetaData in HiveQueryResultSet when DatabaseMetaData ResultSet is closed. It can help reduce HS2 memory usage, though the OperationHandle is eventually removed when it expires or the jdbc conection is closed at client site. > Hive JDBC driver does not close operation for metadata queries > -- > > Key: HIVE-10732 > URL: https://issues.apache.org/jira/browse/HIVE-10732 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Mala Chikka Kempanna >Assignee: Chaoyu Tang > Attachments: HIVE-10732.patch > > > In following file > http://github.mtv.cloudera.com/CDH/hive/blob/cdh5-0.14.1/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java > Line 315 implemented the ResultSet.close() method. Because "DatabaseMetadata" > operation doesn't have a statement, it doesn't "close" the operation. > However, regardless whether it has a statement or not, it should close the > operation through the "stmtHandle". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10732) Hive JDBC driver does not close operation for metadata queries
[ https://issues.apache.org/jira/browse/HIVE-10732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-10732: --- Attachment: HIVE-10732.1.patch Fixed the test failures. > Hive JDBC driver does not close operation for metadata queries > -- > > Key: HIVE-10732 > URL: https://issues.apache.org/jira/browse/HIVE-10732 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Mala Chikka Kempanna >Assignee: Chaoyu Tang > Attachments: HIVE-10732.1.patch, HIVE-10732.patch > > > In following file > http://github.mtv.cloudera.com/CDH/hive/blob/cdh5-0.14.1/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java > Line 315 implemented the ResultSet.close() method. Because "DatabaseMetadata" > operation doesn't have a statement, it doesn't "close" the operation. > However, regardless whether it has a statement or not, it should close the > operation through the "stmtHandle". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10732) Hive JDBC driver does not close operation for metadata queries
[ https://issues.apache.org/jira/browse/HIVE-10732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14551083#comment-14551083 ] Chaoyu Tang commented on HIVE-10732: The one failed test seems not related to this patch. > Hive JDBC driver does not close operation for metadata queries > -- > > Key: HIVE-10732 > URL: https://issues.apache.org/jira/browse/HIVE-10732 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Mala Chikka Kempanna >Assignee: Chaoyu Tang > Attachments: HIVE-10732.1.patch, HIVE-10732.patch > > > In following file > http://github.mtv.cloudera.com/CDH/hive/blob/cdh5-0.14.1/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java > Line 315 implemented the ResultSet.close() method. Because "DatabaseMetadata" > operation doesn't have a statement, it doesn't "close" the operation. > However, regardless whether it has a statement or not, it should close the > operation through the "stmtHandle". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10732) Hive JDBC driver does not close operation for metadata queries
[ https://issues.apache.org/jira/browse/HIVE-10732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14551621#comment-14551621 ] Chaoyu Tang commented on HIVE-10732: [~xuefuz] The stmtHandle in HiveQueryResultSet can be set from HiveStatement or HiveDatabaseMetaData. If the variable "statement" in HiveQueryResultSet is not null, the stmtHandle was from HiveStatement, otherwise from HiveDatabaseMetaData. In method close(), patch#0 closes HiveStatement stmtHandle(TOperationHandle) using s.closeClientOperation(), and close the already closed TOperationHandle again using closeOperationHandle(), so we see such exceptions in the failed tests. But patch#1 closes HiveStatement stmtHandle once by s.closeClientOperation(), and closes HiveDatabaseMetaData stmtHandle by closeOperationHandle(). > Hive JDBC driver does not close operation for metadata queries > -- > > Key: HIVE-10732 > URL: https://issues.apache.org/jira/browse/HIVE-10732 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Mala Chikka Kempanna >Assignee: Chaoyu Tang > Attachments: HIVE-10732.1.patch, HIVE-10732.patch > > > In following file > http://github.mtv.cloudera.com/CDH/hive/blob/cdh5-0.14.1/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java > Line 315 implemented the ResultSet.close() method. Because "DatabaseMetadata" > operation doesn't have a statement, it doesn't "close" the operation. > However, regardless whether it has a statement or not, it should close the > operation through the "stmtHandle". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10784) Beeline requires new line (EOL) at the end of an Hive SQL script (NullPointerException)
[
https://issues.apache.org/jira/browse/HIVE-10784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14554596#comment-14554596
]
Chaoyu Tang commented on HIVE-10784:
I wonder the code changes both from HIVE-9877 and HIVE-10541 might have already
addressed your observed issue.
> Beeline requires new line (EOL) at the end of an Hive SQL script
> (NullPointerException)
> ---
>
> Key: HIVE-10784
> URL: https://issues.apache.org/jira/browse/HIVE-10784
> Project: Hive
> Issue Type: Bug
> Components: Beeline, CLI
>Affects Versions: 0.13.1
> Environment: Linux 2.6.32 (Red Hat 4.4.7)
>Reporter: Andrey Dmitriev
>Assignee: Chinna Rao Lalam
>Priority: Minor
> Attachments: HIVE-10784.patch
>
>
> Beeline tool requires to have "new line" at the end of a Hive/Impala SQL
> script otherwise the last statement will be not executed or
> NullPointerException will be thrown.
> # If a statement ends without end of line AND semicolon is on the same line
> then the statement will be ignored; i.e.
> {code}select * from TABLE;{code} will be *not* executed
> # If a statement ends without end of line BUT semicolon is on the next line
> then the statement will be executed, but
> {color:red};java.lang.NullPointerException{color} will be thrown; i.e.
> {code}select * from TABLE
> ;{code} will be executed, but print
> {color:red};java.lang.NullPointerException{color}
> # If a statement ends with end of line regardless where semicolon is then the
> statement will be executed; i.e.
> {code}select * from TABLE;
> {code}
> or
> {code}select * from TABLE
> ;{code}
> will be executed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10835) Concurrency issues in JDBC driver
[ https://issues.apache.org/jira/browse/HIVE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-10835: --- Attachment: HIVE-10835.patch [~xuefuz], [~szehon] & [~thejas], could you review the patch to see if it make sense. Thanks > Concurrency issues in JDBC driver > - > > Key: HIVE-10835 > URL: https://issues.apache.org/jira/browse/HIVE-10835 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.2.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-10835.patch > > > Though JDBC specification specifies that "Each Connection object can create > multiple Statement objects that may be used concurrently by the program", but > that does not work in current Hive JDBC driver. In addition, there also exist > race conditions between DatabaseMetaData, Statement and ResultSet as long as > they make RPC calls to HS2 using same Thrift transport, which happens within > a connection. > So we need a connection level lock to serialize all these RPC calls in a > connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10835) Concurrency issues in JDBC driver
[ https://issues.apache.org/jira/browse/HIVE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561514#comment-14561514 ] Chaoyu Tang commented on HIVE-10835: Patch has been posted to RB https://reviews.apache.org/r/34727/ requesting for a review. > Concurrency issues in JDBC driver > - > > Key: HIVE-10835 > URL: https://issues.apache.org/jira/browse/HIVE-10835 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.2.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-10835.patch > > > Though JDBC specification specifies that "Each Connection object can create > multiple Statement objects that may be used concurrently by the program", but > that does not work in current Hive JDBC driver. In addition, there also exist > race conditions between DatabaseMetaData, Statement and ResultSet as long as > they make RPC calls to HS2 using same Thrift transport, which happens within > a connection. > So we need a connection level lock to serialize all these RPC calls in a > connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10835) Concurrency issues in JDBC driver
[ https://issues.apache.org/jira/browse/HIVE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561593#comment-14561593 ] Chaoyu Tang commented on HIVE-10835: Thanks [~thejas] for comments. I will look into what you suggested. > Concurrency issues in JDBC driver > - > > Key: HIVE-10835 > URL: https://issues.apache.org/jira/browse/HIVE-10835 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.2.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-10835.patch > > > Though JDBC specification specifies that "Each Connection object can create > multiple Statement objects that may be used concurrently by the program", but > that does not work in current Hive JDBC driver. In addition, there also exist > race conditions between DatabaseMetaData, Statement and ResultSet as long as > they make RPC calls to HS2 using same Thrift transport, which happens within > a connection. > So we need a connection level lock to serialize all these RPC calls in a > connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10835) Concurrency issues in JDBC driver
[ https://issues.apache.org/jira/browse/HIVE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-10835: --- Attachment: HIVE-10835.1.patch Wrap the client in a thread-safe proxy to serialize all its RPC call. [~thejas] Could you review it? Thanks. > Concurrency issues in JDBC driver > - > > Key: HIVE-10835 > URL: https://issues.apache.org/jira/browse/HIVE-10835 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.2.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-10835.1.patch, HIVE-10835.patch > > > Though JDBC specification specifies that "Each Connection object can create > multiple Statement objects that may be used concurrently by the program", but > that does not work in current Hive JDBC driver. In addition, there also exist > race conditions between DatabaseMetaData, Statement and ResultSet as long as > they make RPC calls to HS2 using same Thrift transport, which happens within > a connection. > So we need a connection level lock to serialize all these RPC calls in a > connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10835) Concurrency issues in JDBC driver
[ https://issues.apache.org/jira/browse/HIVE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-10835: --- Attachment: HIVE-10835.1.patch Wrap CLIService client in a proxy to serialize all it RPC calls. [~thejas], could you review it? Thanks > Concurrency issues in JDBC driver > - > > Key: HIVE-10835 > URL: https://issues.apache.org/jira/browse/HIVE-10835 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.2.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-10835.1.patch, HIVE-10835.1.patch, HIVE-10835.patch > > > Though JDBC specification specifies that "Each Connection object can create > multiple Statement objects that may be used concurrently by the program", but > that does not work in current Hive JDBC driver. In addition, there also exist > race conditions between DatabaseMetaData, Statement and ResultSet as long as > they make RPC calls to HS2 using same Thrift transport, which happens within > a connection. > So we need a connection level lock to serialize all these RPC calls in a > connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10752) Revert HIVE-5193
[ https://issues.apache.org/jira/browse/HIVE-10752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14563472#comment-14563472 ] Chaoyu Tang commented on HIVE-10752: Given that HIVE-5193 broke some functionality and it was just for columnar table performance improvement, in addition that patch provided in HIVE-10720 did still not solve the issue. IMO, reverting HIVE-5193 at this moment to restore the functionality is a viable solution. I wonder if any one has other better solutions. Thanks. > Revert HIVE-5193 > > > Key: HIVE-10752 > URL: https://issues.apache.org/jira/browse/HIVE-10752 > Project: Hive > Issue Type: Sub-task > Components: HCatalog >Affects Versions: 1.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-10752.patch > > > Revert HIVE-5193 since it causes pig+hcatalog not working. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10835) Concurrency issues in JDBC driver
[ https://issues.apache.org/jira/browse/HIVE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-10835: --- Attachment: (was: HIVE-10835.1.patch) > Concurrency issues in JDBC driver > - > > Key: HIVE-10835 > URL: https://issues.apache.org/jira/browse/HIVE-10835 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.2.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-10835.1.patch, HIVE-10835.patch > > > Though JDBC specification specifies that "Each Connection object can create > multiple Statement objects that may be used concurrently by the program", but > that does not work in current Hive JDBC driver. In addition, there also exist > race conditions between DatabaseMetaData, Statement and ResultSet as long as > they make RPC calls to HS2 using same Thrift transport, which happens within > a connection. > So we need a connection level lock to serialize all these RPC calls in a > connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10835) Concurrency issues in JDBC driver
[ https://issues.apache.org/jira/browse/HIVE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-10835: --- Attachment: HIVE-10835.2.patch Updated the patch: 1. Add a unit test TestJdbcWithMiniHS2#testConcurrentStatements (300 tasks using 100 threads, take around 1.5 -2 min in my local machine) 2. use the client object itself as the lock [~thejas], [~xuefuz] and [~szehon], could you review it? thanks. > Concurrency issues in JDBC driver > - > > Key: HIVE-10835 > URL: https://issues.apache.org/jira/browse/HIVE-10835 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.2.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-10835.1.patch, HIVE-10835.2.patch, HIVE-10835.patch > > > Though JDBC specification specifies that "Each Connection object can create > multiple Statement objects that may be used concurrently by the program", but > that does not work in current Hive JDBC driver. In addition, there also exist > race conditions between DatabaseMetaData, Statement and ResultSet as long as > they make RPC calls to HS2 using same Thrift transport, which happens within > a connection. > So we need a connection level lock to serialize all these RPC calls in a > connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10835) Concurrency issues in JDBC driver
[ https://issues.apache.org/jira/browse/HIVE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565124#comment-14565124 ] Chaoyu Tang commented on HIVE-10835: Thanks [~sershe] for the information and noticed that you are already working on HIVE-4239. This JIRA mainly addresses the concurrency issue at client site (JDBC driver) while HIVE-4239 may focus on server site as I understand, so I think they could be considered as separate issues. We are working together to address the races in the system but from different areas. Could you take a look at the patch and comment, thanks. > Concurrency issues in JDBC driver > - > > Key: HIVE-10835 > URL: https://issues.apache.org/jira/browse/HIVE-10835 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.2.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-10835.1.patch, HIVE-10835.2.patch, HIVE-10835.patch > > > Though JDBC specification specifies that "Each Connection object can create > multiple Statement objects that may be used concurrently by the program", but > that does not work in current Hive JDBC driver. In addition, there also exist > race conditions between DatabaseMetaData, Statement and ResultSet as long as > they make RPC calls to HS2 using same Thrift transport, which happens within > a connection. > So we need a connection level lock to serialize all these RPC calls in a > connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10835) Concurrency issues in JDBC driver
[ https://issues.apache.org/jira/browse/HIVE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565241#comment-14565241 ] Chaoyu Tang commented on HIVE-10835: I think resolving this JIRA might be helpful since it can be used as one of test cases to HIVE-4239. [~thejas], [~xuefuz], [~szehon] and [~sershe], could you review the patch? Thanks > Concurrency issues in JDBC driver > - > > Key: HIVE-10835 > URL: https://issues.apache.org/jira/browse/HIVE-10835 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.2.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-10835.1.patch, HIVE-10835.2.patch, HIVE-10835.patch > > > Though JDBC specification specifies that "Each Connection object can create > multiple Statement objects that may be used concurrently by the program", but > that does not work in current Hive JDBC driver. In addition, there also exist > race conditions between DatabaseMetaData, Statement and ResultSet as long as > they make RPC calls to HS2 using same Thrift transport, which happens within > a connection. > So we need a connection level lock to serialize all these RPC calls in a > connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10752) Revert HIVE-5193
[ https://issues.apache.org/jira/browse/HIVE-10752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565339#comment-14565339 ] Chaoyu Tang commented on HIVE-10752: +1 > Revert HIVE-5193 > > > Key: HIVE-10752 > URL: https://issues.apache.org/jira/browse/HIVE-10752 > Project: Hive > Issue Type: Sub-task > Components: HCatalog >Affects Versions: 1.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-10752.patch > > > Revert HIVE-5193 since it causes pig+hcatalog not working. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10835) Concurrency issues in JDBC driver
[ https://issues.apache.org/jira/browse/HIVE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-10835: --- Attachment: HIVE-10835.3.patch [~vgumashta] Thanks for the review and I updated the new patch here and rb. > Concurrency issues in JDBC driver > - > > Key: HIVE-10835 > URL: https://issues.apache.org/jira/browse/HIVE-10835 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.2.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-10835.1.patch, HIVE-10835.2.patch, > HIVE-10835.3.patch, HIVE-10835.patch > > > Though JDBC specification specifies that "Each Connection object can create > multiple Statement objects that may be used concurrently by the program", but > that does not work in current Hive JDBC driver. In addition, there also exist > race conditions between DatabaseMetaData, Statement and ResultSet as long as > they make RPC calls to HS2 using same Thrift transport, which happens within > a connection. > So we need a connection level lock to serialize all these RPC calls in a > connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10835) Concurrency issues in JDBC driver
[ https://issues.apache.org/jira/browse/HIVE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566028#comment-14566028 ] Chaoyu Tang commented on HIVE-10835: The 3 failed tests seems not related to this patch. > Concurrency issues in JDBC driver > - > > Key: HIVE-10835 > URL: https://issues.apache.org/jira/browse/HIVE-10835 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 1.2.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-10835.1.patch, HIVE-10835.2.patch, > HIVE-10835.3.patch, HIVE-10835.patch > > > Though JDBC specification specifies that "Each Connection object can create > multiple Statement objects that may be used concurrently by the program", but > that does not work in current Hive JDBC driver. In addition, there also exist > race conditions between DatabaseMetaData, Statement and ResultSet as long as > they make RPC calls to HS2 using same Thrift transport, which happens within > a connection. > So we need a connection level lock to serialize all these RPC calls in a > connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10835) Concurrency issues in JDBC driver
[ https://issues.apache.org/jira/browse/HIVE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566205#comment-14566205 ] Chaoyu Tang commented on HIVE-10835: Thanks [~vgumashta] for reviewing and committing the patch. > Concurrency issues in JDBC driver > - > > Key: HIVE-10835 > URL: https://issues.apache.org/jira/browse/HIVE-10835 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.15.0, 0.14.1, 1.0.0, 1.2.0, > 1.1.0, 1.0.1, 1.1.1, 0.13 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Fix For: 1.3.0 > > Attachments: HIVE-10835.1.patch, HIVE-10835.2.patch, > HIVE-10835.3.patch, HIVE-10835.patch > > > Though JDBC specification specifies that "Each Connection object can create > multiple Statement objects that may be used concurrently by the program", but > that does not work in current Hive JDBC driver. In addition, there also exist > race conditions between DatabaseMetaData, Statement and ResultSet as long as > they make RPC calls to HS2 using same Thrift transport, which happens within > a connection. > So we need a connection level lock to serialize all these RPC calls in a > connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10410) Apparent race condition in HiveServer2 causing intermittent query failures
[
https://issues.apache.org/jira/browse/HIVE-10410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14570898#comment-14570898
]
Chaoyu Tang commented on HIVE-10410:
[~Richard Williams] We ran into the same problem and I also think that the
sharing Hive object (MetaStoreClient) of parent session in child threads can
cause the observed issue. What is your use scenario? Do you have multiple
concurrent statements running with same connection?
> Apparent race condition in HiveServer2 causing intermittent query failures
> --
>
> Key: HIVE-10410
> URL: https://issues.apache.org/jira/browse/HIVE-10410
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.13.1
> Environment: CDH 5.3.3
> CentOS 6.4
>Reporter: Richard Williams
>
> On our secure Hadoop cluster, queries submitted to HiveServer2 through JDBC
> occasionally trigger odd Thrift exceptions with messages such as "Read a
> negative frame size (-2147418110)!" or "out of sequence response" in
> HiveServer2's connections to the metastore. For certain metastore calls (for
> example, showDatabases), these Thrift exceptions are converted to
> MetaExceptions in HiveMetaStoreClient, which prevents RetryingMetaStoreClient
> from retrying these calls and thus causes the failure to bubble out to the
> JDBC client.
> Note that as far as we can tell, this issue appears to only affect queries
> that are submitted with the runAsync flag on TExecuteStatementReq set to true
> (which, in practice, seems to mean all JDBC queries), and it appears to only
> manifest when HiveServer2 is using the new HTTP transport mechanism. When
> both these conditions hold, we are able to fairly reliably reproduce the
> issue by spawning about 100 simple, concurrent hive queries (we have been
> using "show databases"), two or three of which typically fail. However, when
> either of these conditions do not hold, we are no longer able to reproduce
> the issue.
> Some example stack traces from the HiveServer2 logs:
> {noformat}
> 2015-04-16 13:54:55,486 ERROR hive.log: Got exception:
> org.apache.thrift.transport.TTransportException Read a negative frame size
> (-2147418110)!
> org.apache.thrift.transport.TTransportException: Read a negative frame size
> (-2147418110)!
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:435)
> at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:414)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.hadoop.hive.thrift.TFilterTransport.readAll(TFilterTransport.java:62)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:600)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:587)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:837)
> at
> org.apache.sentry.binding.metastore.SentryHiveMetaStoreClient.getDatabases(SentryHiveMetaStoreClient.java:60)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
> at com.sun.proxy.$Proxy6.getDatabases(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getDatabasesByPattern(Hive.java:1139)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.showDatabases(DDLTask.java:2445)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:364)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1554)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1321)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1139)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:962)
> at org.apache.hadoop.hi
[jira] [Commented] (HIVE-10410) Apparent race condition in HiveServer2 causing intermittent query failures
[
https://issues.apache.org/jira/browse/HIVE-10410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14571864#comment-14571864
]
Chaoyu Tang commented on HIVE-10410:
[~Richard Williams] are you still working on the patch?
> Apparent race condition in HiveServer2 causing intermittent query failures
> --
>
> Key: HIVE-10410
> URL: https://issues.apache.org/jira/browse/HIVE-10410
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.13.1
> Environment: CDH 5.3.3
> CentOS 6.4
>Reporter: Richard Williams
>
> On our secure Hadoop cluster, queries submitted to HiveServer2 through JDBC
> occasionally trigger odd Thrift exceptions with messages such as "Read a
> negative frame size (-2147418110)!" or "out of sequence response" in
> HiveServer2's connections to the metastore. For certain metastore calls (for
> example, showDatabases), these Thrift exceptions are converted to
> MetaExceptions in HiveMetaStoreClient, which prevents RetryingMetaStoreClient
> from retrying these calls and thus causes the failure to bubble out to the
> JDBC client.
> Note that as far as we can tell, this issue appears to only affect queries
> that are submitted with the runAsync flag on TExecuteStatementReq set to true
> (which, in practice, seems to mean all JDBC queries), and it appears to only
> manifest when HiveServer2 is using the new HTTP transport mechanism. When
> both these conditions hold, we are able to fairly reliably reproduce the
> issue by spawning about 100 simple, concurrent hive queries (we have been
> using "show databases"), two or three of which typically fail. However, when
> either of these conditions do not hold, we are no longer able to reproduce
> the issue.
> Some example stack traces from the HiveServer2 logs:
> {noformat}
> 2015-04-16 13:54:55,486 ERROR hive.log: Got exception:
> org.apache.thrift.transport.TTransportException Read a negative frame size
> (-2147418110)!
> org.apache.thrift.transport.TTransportException: Read a negative frame size
> (-2147418110)!
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:435)
> at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:414)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.hadoop.hive.thrift.TFilterTransport.readAll(TFilterTransport.java:62)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:600)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:587)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:837)
> at
> org.apache.sentry.binding.metastore.SentryHiveMetaStoreClient.getDatabases(SentryHiveMetaStoreClient.java:60)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
> at com.sun.proxy.$Proxy6.getDatabases(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getDatabasesByPattern(Hive.java:1139)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.showDatabases(DDLTask.java:2445)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:364)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1554)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1321)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1139)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:962)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:957)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:145)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$000(SQLOperatio
[jira] [Commented] (HIVE-10410) Apparent race condition in HiveServer2 causing intermittent query failures
[
https://issues.apache.org/jira/browse/HIVE-10410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572981#comment-14572981
]
Chaoyu Tang commented on HIVE-10410:
I think not only the Hive object, but also sessionState and HiveConf shared in
child threads may also cause the race issue.
> Apparent race condition in HiveServer2 causing intermittent query failures
> --
>
> Key: HIVE-10410
> URL: https://issues.apache.org/jira/browse/HIVE-10410
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.13.1
> Environment: CDH 5.3.3
> CentOS 6.4
>Reporter: Richard Williams
>
> On our secure Hadoop cluster, queries submitted to HiveServer2 through JDBC
> occasionally trigger odd Thrift exceptions with messages such as "Read a
> negative frame size (-2147418110)!" or "out of sequence response" in
> HiveServer2's connections to the metastore. For certain metastore calls (for
> example, showDatabases), these Thrift exceptions are converted to
> MetaExceptions in HiveMetaStoreClient, which prevents RetryingMetaStoreClient
> from retrying these calls and thus causes the failure to bubble out to the
> JDBC client.
> Note that as far as we can tell, this issue appears to only affect queries
> that are submitted with the runAsync flag on TExecuteStatementReq set to true
> (which, in practice, seems to mean all JDBC queries), and it appears to only
> manifest when HiveServer2 is using the new HTTP transport mechanism. When
> both these conditions hold, we are able to fairly reliably reproduce the
> issue by spawning about 100 simple, concurrent hive queries (we have been
> using "show databases"), two or three of which typically fail. However, when
> either of these conditions do not hold, we are no longer able to reproduce
> the issue.
> Some example stack traces from the HiveServer2 logs:
> {noformat}
> 2015-04-16 13:54:55,486 ERROR hive.log: Got exception:
> org.apache.thrift.transport.TTransportException Read a negative frame size
> (-2147418110)!
> org.apache.thrift.transport.TTransportException: Read a negative frame size
> (-2147418110)!
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:435)
> at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:414)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.hadoop.hive.thrift.TFilterTransport.readAll(TFilterTransport.java:62)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:600)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:587)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:837)
> at
> org.apache.sentry.binding.metastore.SentryHiveMetaStoreClient.getDatabases(SentryHiveMetaStoreClient.java:60)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
> at com.sun.proxy.$Proxy6.getDatabases(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getDatabasesByPattern(Hive.java:1139)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.showDatabases(DDLTask.java:2445)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:364)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1554)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1321)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1139)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:962)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:957)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:145)
> at
> org.
[jira] [Assigned] (HIVE-10933) Hive 0.13 returns precision 0 for varchar(32) from DatabaseMetadata.getColumns()
[
https://issues.apache.org/jira/browse/HIVE-10933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang reassigned HIVE-10933:
--
Assignee: Chaoyu Tang
> Hive 0.13 returns precision 0 for varchar(32) from
> DatabaseMetadata.getColumns()
>
>
> Key: HIVE-10933
> URL: https://issues.apache.org/jira/browse/HIVE-10933
> Project: Hive
> Issue Type: Bug
> Components: API
>Affects Versions: 0.13.0
>Reporter: Son Nguyen
>Assignee: Chaoyu Tang
>
> DatabaseMetadata.getColumns() returns "COLUMN_SIZE" as 0 for a column defined
> as varchar(32), or char(32). While ResultSetMetaData.getPrecision() returns
> correct value 32.
> Here is the segment program that reproduces the issue.
> try {
> statement = connection.createStatement();
>
> statement.execute("drop table if exists son_table");
>
> statement.execute("create table son_table( col1 varchar(32) )");
>
> statement.close();
>
> } catch ( Exception e) {
> return;
> }
>
> // get column info using metadata
> try {
> DatabaseMetaData dmd = null;
> ResultSet resultSet = null;
>
> dmd = connection.getMetaData();
>
> resultSet = dmd.getColumns(null, null, "son_table", "col1");
>
> if ( resultSet.next() ) {
> String tabName = resultSet.getString("TABLE_NAME");
> String colName = resultSet.getString("COLUMN_NAME");
> String dataType = resultSet.getString("DATA_TYPE");
> String typeName = resultSet.getString("TYPE_NAME");
> int precision = resultSet.getInt("COLUMN_SIZE");
>
> // output is: colName = col1, dataType = 12, typeName =
> VARCHAR, precision = 0.
> System.out.format("colName = %s, dataType = %s, typeName = %s,
> precision = %d.",
> colName, dataType, typeName, precision);
> }
> } catch ( Exception e) {
> return;
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10933) Hive 0.13 returns precision 0 for varchar(32) from DatabaseMetadata.getColumns()
[
https://issues.apache.org/jira/browse/HIVE-10933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573956#comment-14573956
]
Chaoyu Tang commented on HIVE-10933:
I could not reproduce your issue in trunk and believe it has been resolved by
HIVE-5847 since Hive 0.14. Could you try again in Hive 1.2?
> Hive 0.13 returns precision 0 for varchar(32) from
> DatabaseMetadata.getColumns()
>
>
> Key: HIVE-10933
> URL: https://issues.apache.org/jira/browse/HIVE-10933
> Project: Hive
> Issue Type: Bug
> Components: API
>Affects Versions: 0.13.0
>Reporter: Son Nguyen
>Assignee: Chaoyu Tang
>
> DatabaseMetadata.getColumns() returns "COLUMN_SIZE" as 0 for a column defined
> as varchar(32), or char(32). While ResultSetMetaData.getPrecision() returns
> correct value 32.
> Here is the segment program that reproduces the issue.
> try {
> statement = connection.createStatement();
>
> statement.execute("drop table if exists son_table");
>
> statement.execute("create table son_table( col1 varchar(32) )");
>
> statement.close();
>
> } catch ( Exception e) {
> return;
> }
>
> // get column info using metadata
> try {
> DatabaseMetaData dmd = null;
> ResultSet resultSet = null;
>
> dmd = connection.getMetaData();
>
> resultSet = dmd.getColumns(null, null, "son_table", "col1");
>
> if ( resultSet.next() ) {
> String tabName = resultSet.getString("TABLE_NAME");
> String colName = resultSet.getString("COLUMN_NAME");
> String dataType = resultSet.getString("DATA_TYPE");
> String typeName = resultSet.getString("TYPE_NAME");
> int precision = resultSet.getInt("COLUMN_SIZE");
>
> // output is: colName = col1, dataType = 12, typeName =
> VARCHAR, precision = 0.
> System.out.format("colName = %s, dataType = %s, typeName = %s,
> precision = %d.",
> colName, dataType, typeName, precision);
> }
> } catch ( Exception e) {
> return;
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7193) Hive should support additional LDAP authentication parameters
[
https://issues.apache.org/jira/browse/HIVE-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14575310#comment-14575310
]
Chaoyu Tang commented on HIVE-7193:
---
[~ngangam] Overall, the patch looks good to me. I still have some questions,
could you help clarify them? Thanks
> Hive should support additional LDAP authentication parameters
> -
>
> Key: HIVE-7193
> URL: https://issues.apache.org/jira/browse/HIVE-7193
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Mala Chikka Kempanna
>Assignee: Naveen Gangam
> Attachments: HIVE-7193.2.patch, HIVE-7193.3.patch, HIVE-7193.patch,
> LDAPAuthentication_Design_Doc.docx, LDAPAuthentication_Design_Doc_V2.docx
>
>
> Currently hive has only following authenticator parameters for LDAP
> authentication for hiveserver2.
>
> hive.server2.authentication
> LDAP
>
>
> hive.server2.authentication.ldap.url
> ldap://our_ldap_address
>
> We need to include other LDAP properties as part of hive-LDAP authentication
> like below
> a group search base -> dc=domain,dc=com
> a group search filter -> member={0}
> a user search base -> dc=domain,dc=com
> a user search filter -> sAMAAccountName={0}
> a list of valid user groups -> group1,group2,group3
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-10933) Hive 0.13 returns precision 0 for varchar(32) from DatabaseMetadata.getColumns()
[
https://issues.apache.org/jira/browse/HIVE-10933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang resolved HIVE-10933.
Resolution: Cannot Reproduce
As I understand it has been resolved by HIVE-5847. If you see any related issue
in future, please feel free to reopen this JIRA or open a new one.
> Hive 0.13 returns precision 0 for varchar(32) from
> DatabaseMetadata.getColumns()
>
>
> Key: HIVE-10933
> URL: https://issues.apache.org/jira/browse/HIVE-10933
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 0.13.0
>Reporter: Son Nguyen
>Assignee: Chaoyu Tang
>
> DatabaseMetadata.getColumns() returns "COLUMN_SIZE" as 0 for a column defined
> as varchar(32), or char(32). While ResultSetMetaData.getPrecision() returns
> correct value 32.
> Here is the segment program that reproduces the issue.
> {code}
> try {
> statement = connection.createStatement();
>
> statement.execute("drop table if exists son_table");
>
> statement.execute("create table son_table( col1 varchar(32) )");
>
> statement.close();
>
> } catch ( Exception e) {
> return;
> }
>
> // get column info using metadata
> try {
> DatabaseMetaData dmd = null;
> ResultSet resultSet = null;
>
> dmd = connection.getMetaData();
>
> resultSet = dmd.getColumns(null, null, "son_table", "col1");
>
> if ( resultSet.next() ) {
> String tabName = resultSet.getString("TABLE_NAME");
> String colName = resultSet.getString("COLUMN_NAME");
> String dataType = resultSet.getString("DATA_TYPE");
> String typeName = resultSet.getString("TYPE_NAME");
> int precision = resultSet.getInt("COLUMN_SIZE");
>
> // output is: colName = col1, dataType = 12, typeName =
> VARCHAR, precision = 0.
> System.out.format("colName = %s, dataType = %s, typeName = %s,
> precision = %d.",
> colName, dataType, typeName, precision);
> }
> } catch ( Exception e) {
> return;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10976) Redundant HiveMetaStore connect check in HS2 CLIService start
[ https://issues.apache.org/jira/browse/HIVE-10976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-10976: --- Attachment: HIVE-10976.patch Remove the redundant HMS connection check. [~xuefuz] & [~jxiang] could you review it? Thanks. > Redundant HiveMetaStore connect check in HS2 CLIService start > - > > Key: HIVE-10976 > URL: https://issues.apache.org/jira/browse/HIVE-10976 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Trivial > Attachments: HIVE-10976.patch > > > During HS2 startup, CLIService start() does a HMS connection test to HMS. > It is redundant, since in its init stage, CLIService calls > applyAuthorizationConfigPolicy where it starts a sessionState and establishes > a connection to HMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10976) Redundant HiveMetaStore connect check in HS2 CLIService start
[ https://issues.apache.org/jira/browse/HIVE-10976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14580717#comment-14580717 ] Chaoyu Tang commented on HIVE-10976: These failed tests seem not relevant to this patch. > Redundant HiveMetaStore connect check in HS2 CLIService start > - > > Key: HIVE-10976 > URL: https://issues.apache.org/jira/browse/HIVE-10976 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Trivial > Attachments: HIVE-10976.patch > > > During HS2 startup, CLIService start() does a HMS connection test to HMS. > It is redundant, since in its init stage, CLIService calls > applyAuthorizationConfigPolicy where it starts a sessionState and establishes > a connection to HMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10977) No need to instantiate MetaStoreDirectSql when HMS DirectSql is disabled
[ https://issues.apache.org/jira/browse/HIVE-10977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-10977: --- Attachment: HIVE-10977.patch Instantiate MetaStoreDirectSql in ObjectStore only when HMS DirectSql is enabled. [~sershe] and [~xuefuz], could you review to see if it makes sense? > No need to instantiate MetaStoreDirectSql when HMS DirectSql is disabled > > > Key: HIVE-10977 > URL: https://issues.apache.org/jira/browse/HIVE-10977 > Project: Hive > Issue Type: Bug >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > Attachments: HIVE-10977.patch > > > When hive.metastore.try.direct.sql is set to false, HMS will use JDO to > retrieve data, therefor it is not necessary to instantiate an expensive > MetaStoreDirectSql during ObjectStore initialization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10977) No need to instantiate MetaStoreDirectSql when HMS DirectSql is disabled
[ https://issues.apache.org/jira/browse/HIVE-10977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581887#comment-14581887 ] Chaoyu Tang commented on HIVE-10977: The two failed tests seem not relevant to this patch. > No need to instantiate MetaStoreDirectSql when HMS DirectSql is disabled > > > Key: HIVE-10977 > URL: https://issues.apache.org/jira/browse/HIVE-10977 > Project: Hive > Issue Type: Bug >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > Attachments: HIVE-10977.patch > > > When hive.metastore.try.direct.sql is set to false, HMS will use JDO to > retrieve data, therefor it is not necessary to instantiate an expensive > MetaStoreDirectSql during ObjectStore initialization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10976) Redundant HiveMetaStore connect check in HS2 CLIService start
[ https://issues.apache.org/jira/browse/HIVE-10976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582182#comment-14582182 ] Chaoyu Tang commented on HIVE-10976: [~jxiang] It is possible since it only calls the upperclass start(), but I thought that but still left it there because 1) to be consistent with the stop method; 2) in case in future CLIService needs add something especially in start, whose method already exists. > Redundant HiveMetaStore connect check in HS2 CLIService start > - > > Key: HIVE-10976 > URL: https://issues.apache.org/jira/browse/HIVE-10976 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Trivial > Attachments: HIVE-10976.patch > > > During HS2 startup, CLIService start() does a HMS connection test to HMS. > It is redundant, since in its init stage, CLIService calls > applyAuthorizationConfigPolicy where it starts a sessionState and establishes > a connection to HMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7018) Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but not others
[ https://issues.apache.org/jira/browse/HIVE-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582188#comment-14582188 ] Chaoyu Tang commented on HIVE-7018: --- [~ychena] have you verified that it works with SchemaTool with [~hsubramaniyan] HIVE-10659 fix? > Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but > not others > - > > Key: HIVE-7018 > URL: https://issues.apache.org/jira/browse/HIVE-7018 > Project: Hive > Issue Type: Bug >Reporter: Brock Noland >Assignee: Yongzhi Chen > Attachments: HIVE-7018.1.patch, HIVE-7018.2.patch, HIVE-7018.3.patch > > > It appears that at least postgres and oracle do not have the LINK_TARGET_ID > column while mysql does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10976) Redundant HiveMetaStore connect check in HS2 CLIService start
[ https://issues.apache.org/jira/browse/HIVE-10976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582207#comment-14582207 ] Chaoyu Tang commented on HIVE-10976: Thanks, [~jxiang]. Could you help to commit it? > Redundant HiveMetaStore connect check in HS2 CLIService start > - > > Key: HIVE-10976 > URL: https://issues.apache.org/jira/browse/HIVE-10976 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Trivial > Attachments: HIVE-10976.patch > > > During HS2 startup, CLIService start() does a HMS connection test to HMS. > It is redundant, since in its init stage, CLIService calls > applyAuthorizationConfigPolicy where it starts a sessionState and establishes > a connection to HMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10972) DummyTxnManager always locks the current database in shared mode, which is incorrect.
[ https://issues.apache.org/jira/browse/HIVE-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582615#comment-14582615 ] Chaoyu Tang commented on HIVE-10972: [~aihuaxu] I left some comments and questions on the RB. > DummyTxnManager always locks the current database in shared mode, which is > incorrect. > - > > Key: HIVE-10972 > URL: https://issues.apache.org/jira/browse/HIVE-10972 > Project: Hive > Issue Type: Bug > Components: Locking >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-10972.patch > > > In DummyTxnManager [line 163 | > http://grepcode.com/file/repo1.maven.org/maven2/co.cask.cdap/hive-exec/0.13.0/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java#163], > it always locks the current database. > That is not correct since the current database can be "db1", and the query > can be "select * from db2.tb1", which will lock db1 unnecessarily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7018) Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but not others
[ https://issues.apache.org/jira/browse/HIVE-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14583347#comment-14583347 ] Chaoyu Tang commented on HIVE-7018: --- +1 > Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but > not others > - > > Key: HIVE-7018 > URL: https://issues.apache.org/jira/browse/HIVE-7018 > Project: Hive > Issue Type: Bug >Reporter: Brock Noland >Assignee: Yongzhi Chen > Attachments: HIVE-7018.1.patch, HIVE-7018.2.patch, HIVE-7018.3.patch > > > It appears that at least postgres and oracle do not have the LINK_TARGET_ID > column while mysql does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10972) DummyTxnManager always locks the current database in shared mode, which is incorrect.
[ https://issues.apache.org/jira/browse/HIVE-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14583921#comment-14583921 ] Chaoyu Tang commented on HIVE-10972: [~aihuaxu] Could you have a look at and take care of the failed test lockneg_try_lock_db_in_use? I think the test is valid. > DummyTxnManager always locks the current database in shared mode, which is > incorrect. > - > > Key: HIVE-10972 > URL: https://issues.apache.org/jira/browse/HIVE-10972 > Project: Hive > Issue Type: Bug > Components: Locking >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-10972.patch > > > In DummyTxnManager [line 163 | > http://grepcode.com/file/repo1.maven.org/maven2/co.cask.cdap/hive-exec/0.13.0/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java#163], > it always locks the current database. > That is not correct since the current database can be "db1", and the query > can be "select * from db2.tb1", which will lock db1 unnecessarily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7018) Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but not others
[ https://issues.apache.org/jira/browse/HIVE-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14584598#comment-14584598 ] Chaoyu Tang commented on HIVE-7018: --- Thanks [~ychena] for the patch, it has been committed to 1.3.0 (branch-1). But I think you also need create a separate patch for 2.0.0 (master branch). > Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but > not others > - > > Key: HIVE-7018 > URL: https://issues.apache.org/jira/browse/HIVE-7018 > Project: Hive > Issue Type: Bug >Reporter: Brock Noland >Assignee: Yongzhi Chen > Attachments: HIVE-7018.1.patch, HIVE-7018.2.patch, HIVE-7018.3.patch > > > It appears that at least postgres and oracle do not have the LINK_TARGET_ID > column while mysql does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7018) Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but not others
[ https://issues.apache.org/jira/browse/HIVE-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586098#comment-14586098 ] Chaoyu Tang commented on HIVE-7018: --- [~ychena] Looks like the HMS upgrade test failed, do you know the reason? > Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but > not others > - > > Key: HIVE-7018 > URL: https://issues.apache.org/jira/browse/HIVE-7018 > Project: Hive > Issue Type: Bug >Reporter: Brock Noland >Assignee: Yongzhi Chen > Attachments: HIVE-7018.1.patch, HIVE-7018.2.patch, HIVE-7018.3.patch, > HIVE-7018.4.patch > > > It appears that at least postgres and oracle do not have the LINK_TARGET_ID > column while mysql does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7193) Hive should support additional LDAP authentication parameters
[
https://issues.apache.org/jira/browse/HIVE-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589224#comment-14589224
]
Chaoyu Tang commented on HIVE-7193:
---
Thanks [~ngangam] for the patch. It looks good to me. Regarding to the concern
you had whether the AtnProvider should be changed to be implemented as a
singleton, I agree with you that you would not address it in this patch for
following reasons:
1. The existing code does not implement AtnProvider as a singleton. Making such
change might have some backward compatibility issue. For example, what if a
user has already implemented and used a CustomAuthenticationProvider which is
not for a singleton?
2. The patch only adds several additional read and processing of HiveConf
properties in LdapAuthenticationProviderImpl constructor. Compared to LDAP
authentication itself, its overhead should be trivial and it should not be a
performance bottleneck.
3. In case it turns out the performance is not desirable due to AtnProvider
instantiation, we might consider moving some static logic from constructor to a
static block to improve runtime performance. Or open a separate JIRA to
initiate the investigation to performance implementation (including singleton
etc). But this patch will mainly focuses on the LDAP enhancement.
4. As for your concern "dont know what the user-coded
CustomAuthenticationProvider could do", even if you change the
AuthenticationProviderFactory and allow it to be implemented as a singleton,
but like you said, we still have no control how he implements the singleton.
In addition, the enhancement including its new configuration properties should
be properly documented.
> Hive should support additional LDAP authentication parameters
> -
>
> Key: HIVE-7193
> URL: https://issues.apache.org/jira/browse/HIVE-7193
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Mala Chikka Kempanna
>Assignee: Naveen Gangam
> Attachments: HIVE-7193.2.patch, HIVE-7193.3.patch, HIVE-7193.5.patch,
> HIVE-7193.patch, LDAPAuthentication_Design_Doc.docx,
> LDAPAuthentication_Design_Doc_V2.docx
>
>
> Currently hive has only following authenticator parameters for LDAP
> authentication for hiveserver2:
> {code:xml}
>
> hive.server2.authentication
> LDAP
>
>
> hive.server2.authentication.ldap.url
> ldap://our_ldap_address
>
> {code}
> We need to include other LDAP properties as part of hive-LDAP authentication
> like below:
> {noformat}
> a group search base -> dc=domain,dc=com
> a group search filter -> member={0}
> a user search base -> dc=domain,dc=com
> a user search filter -> sAMAAccountName={0}
> a list of valid user groups -> group1,group2,group3
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7018) Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but not others
[ https://issues.apache.org/jira/browse/HIVE-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590197#comment-14590197 ] Chaoyu Tang commented on HIVE-7018: --- [~ychena] Actually the HMS precommit tests passed, see http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-METASTORE-Test/54/ (Build #54 (Jun 17, 2015 8:59:00 AM) ) +1 > Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but > not others > - > > Key: HIVE-7018 > URL: https://issues.apache.org/jira/browse/HIVE-7018 > Project: Hive > Issue Type: Bug >Reporter: Brock Noland >Assignee: Yongzhi Chen > Attachments: HIVE-7018.1.patch, HIVE-7018.2.patch, HIVE-7018.3.patch, > HIVE-7018.4.patch, HIVE-7018.5.patch > > > It appears that at least postgres and oracle do not have the LINK_TARGET_ID > column while mysql does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7193) Hive should support additional LDAP authentication parameters
[
https://issues.apache.org/jira/browse/HIVE-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14591109#comment-14591109
]
Chaoyu Tang commented on HIVE-7193:
---
+1
> Hive should support additional LDAP authentication parameters
> -
>
> Key: HIVE-7193
> URL: https://issues.apache.org/jira/browse/HIVE-7193
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Mala Chikka Kempanna
>Assignee: Naveen Gangam
> Attachments: HIVE-7193.2.patch, HIVE-7193.3.patch, HIVE-7193.4.patch,
> HIVE-7193.patch, LDAPAuthentication_Design_Doc.docx,
> LDAPAuthentication_Design_Doc_V2.docx
>
>
> Currently hive has only following authenticator parameters for LDAP
> authentication for hiveserver2:
> {code:xml}
>
> hive.server2.authentication
> LDAP
>
>
> hive.server2.authentication.ldap.url
> ldap://our_ldap_address
>
> {code}
> We need to include other LDAP properties as part of hive-LDAP authentication
> like below:
> {noformat}
> a group search base -> dc=domain,dc=com
> a group search filter -> member={0}
> a user search base -> dc=domain,dc=com
> a user search filter -> sAMAAccountName={0}
> a list of valid user groups -> group1,group2,group3
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-10972) DummyTxnManager always locks the current database in shared mode, which is incorrect.
[ https://issues.apache.org/jira/browse/HIVE-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang resolved HIVE-10972. Resolution: Fixed Fix Version/s: 2.0.0 1.3.0 Committed to Hive 2.0.0 and 1.3.0. Thanks [~aihuaxu] for the patch and [~alangates] for review. > DummyTxnManager always locks the current database in shared mode, which is > incorrect. > - > > Key: HIVE-10972 > URL: https://issues.apache.org/jira/browse/HIVE-10972 > Project: Hive > Issue Type: Bug > Components: Locking >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-10972.2.patch, HIVE-10972.patch > > > In DummyTxnManager [line 163 | > http://grepcode.com/file/repo1.maven.org/maven2/co.cask.cdap/hive-exec/0.13.0/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java#163], > it always locks the current database. > That is not correct since the current database can be "db1", and the query > can be "select * from db2.tb1", which will lock db1 unnecessarily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7193) Hive should support additional LDAP authentication parameters
[
https://issues.apache.org/jira/browse/HIVE-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-7193:
--
Labels: TODOC1.3 TODOC2.0 (was: )
> Hive should support additional LDAP authentication parameters
> -
>
> Key: HIVE-7193
> URL: https://issues.apache.org/jira/browse/HIVE-7193
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Mala Chikka Kempanna
>Assignee: Naveen Gangam
> Labels: TODOC1.3, TODOC2.0
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-7193.2.patch, HIVE-7193.3.patch, HIVE-7193.4.patch,
> HIVE-7193.5.patch, HIVE-7193.6.patch, HIVE-7193.patch,
> LDAPAuthentication_Design_Doc.docx, LDAPAuthentication_Design_Doc_V2.docx
>
>
> Currently hive has only following authenticator parameters for LDAP
> authentication for hiveserver2:
> {code:xml}
>
> hive.server2.authentication
> LDAP
>
>
> hive.server2.authentication.ldap.url
> ldap://our_ldap_address
>
> {code}
> We need to include other LDAP properties as part of hive-LDAP authentication
> like below:
> {noformat}
> a group search base -> dc=domain,dc=com
> a group search filter -> member={0}
> a user search base -> dc=domain,dc=com
> a user search filter -> sAMAAccountName={0}
> a list of valid user groups -> group1,group2,group3
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11100) Beeline should escape semi-colon in queries
[ https://issues.apache.org/jira/browse/HIVE-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14602327#comment-14602327 ] Chaoyu Tang commented on HIVE-11100: Since Hive 0.14, Beeline supports multiple commands separated by ";" in a single line passed in via -f or -e, which break the case with ";" in sql query. This JIRA is to provide an escape "\" to the ";' in query, so that it will behave same as CLI. > Beeline should escape semi-colon in queries > --- > > Key: HIVE-11100 > URL: https://issues.apache.org/jira/browse/HIVE-11100 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.2.0, 1.1.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > > Beeline should escape the semicolon in queries. for example, the query like > followings: > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY ';' LINES TERMINATED BY '\n'; > or > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY '\;' LINES TERMINATED BY '\n'; > both failed. > But the 2nd query with semicolon escaped with "\" works in CLI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11100) Beeline should escape semi-colon in queries
[ https://issues.apache.org/jira/browse/HIVE-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-11100: --- Attachment: HIVE-11100.patch Since the ";" is being used to delimit commands in a single line in Beeline, it should be able to be escaped when it is not used as a delimiter. This patch provides "\" as the escape to it like that in CLI. > Beeline should escape semi-colon in queries > --- > > Key: HIVE-11100 > URL: https://issues.apache.org/jira/browse/HIVE-11100 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.2.0, 1.1.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > Attachments: HIVE-11100.patch > > > Beeline should escape the semicolon in queries. for example, the query like > followings: > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY ';' LINES TERMINATED BY '\n'; > or > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY '\;' LINES TERMINATED BY '\n'; > both failed. > But the 2nd query with semicolon escaped with "\" works in CLI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11100) Beeline should escape semi-colon in queries
[ https://issues.apache.org/jira/browse/HIVE-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14602982#comment-14602982 ] Chaoyu Tang commented on HIVE-11100: Patch was uploaded to https://reviews.apache.org/r/35907/ and requested for review. Thanks in advance. > Beeline should escape semi-colon in queries > --- > > Key: HIVE-11100 > URL: https://issues.apache.org/jira/browse/HIVE-11100 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.2.0, 1.1.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > Attachments: HIVE-11100.patch > > > Beeline should escape the semicolon in queries. for example, the query like > followings: > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY ';' LINES TERMINATED BY '\n'; > or > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY '\;' LINES TERMINATED BY '\n'; > both failed. > But the 2nd query with semicolon escaped with "\" works in CLI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11100) Beeline should escape semi-colon in queries
[ https://issues.apache.org/jira/browse/HIVE-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-11100: --- Attachment: (was: HIVE-11100.patch) > Beeline should escape semi-colon in queries > --- > > Key: HIVE-11100 > URL: https://issues.apache.org/jira/browse/HIVE-11100 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.2.0, 1.1.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > > Beeline should escape the semicolon in queries. for example, the query like > followings: > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY ';' LINES TERMINATED BY '\n'; > or > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY '\;' LINES TERMINATED BY '\n'; > both failed. > But the 2nd query with semicolon escaped with "\" works in CLI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11100) Beeline should escape semi-colon in queries
[ https://issues.apache.org/jira/browse/HIVE-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-11100: --- Attachment: HIVE-11100.patch For unknown reason, the precommit test did not run. Reattach the patch to kick off the build. > Beeline should escape semi-colon in queries > --- > > Key: HIVE-11100 > URL: https://issues.apache.org/jira/browse/HIVE-11100 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.2.0, 1.1.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > Attachments: HIVE-11100.patch > > > Beeline should escape the semicolon in queries. for example, the query like > followings: > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY ';' LINES TERMINATED BY '\n'; > or > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY '\;' LINES TERMINATED BY '\n'; > both failed. > But the 2nd query with semicolon escaped with "\" works in CLI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14799) Query operation are not thread safe during its cancellation
[ https://issues.apache.org/jira/browse/HIVE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14799: --- Attachment: HIVE-14799.6.patch > Query operation are not thread safe during its cancellation > --- > > Key: HIVE-14799 > URL: https://issues.apache.org/jira/browse/HIVE-14799 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14799.1.patch, HIVE-14799.2.patch, > HIVE-14799.3.patch, HIVE-14799.4.patch, HIVE-14799.5.patch, > HIVE-14799.5.patch, HIVE-14799.6.patch, HIVE-14799.6.patch, HIVE-14799.patch > > > When a query is cancelled either via Beeline (Ctrl-C) or API call > TCLIService.Client.CancelOperation, SQLOperation.cancel is invoked in a > different thread from that running the query to close/destroy its > encapsulated Driver object. Both SQLOperation and Driver are not thread-safe > which could sometimes result in Runtime exceptions like NPE. The errors from > the running query are not handled properly therefore probably causing some > stuffs (files, locks etc) not being cleaned after the query termination. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14927) Remove code duplication from tests in TestLdapAtnProviderWithMiniDS
[ https://issues.apache.org/jira/browse/HIVE-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572022#comment-15572022 ] Chaoyu Tang commented on HIVE-14927: LGTM, +1 pending precommit tests > Remove code duplication from tests in TestLdapAtnProviderWithMiniDS > --- > > Key: HIVE-14927 > URL: https://issues.apache.org/jira/browse/HIVE-14927 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: HIVE-14927.1.patch > > > * Extract inner class User and implement a proper builder for it. > * Extract all common code to LdapAuthenticationTestCase class > ** setting up the test case >** executing test case >** result validation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14799) Query operation are not thread safe during its cancellation
[ https://issues.apache.org/jira/browse/HIVE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572287#comment-15572287 ] Chaoyu Tang commented on HIVE-14799: The failed tests should not be related to this patch and I am not able to reproduce it in my local env. > Query operation are not thread safe during its cancellation > --- > > Key: HIVE-14799 > URL: https://issues.apache.org/jira/browse/HIVE-14799 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14799.1.patch, HIVE-14799.2.patch, > HIVE-14799.3.patch, HIVE-14799.4.patch, HIVE-14799.5.patch, > HIVE-14799.5.patch, HIVE-14799.6.patch, HIVE-14799.6.patch, HIVE-14799.patch > > > When a query is cancelled either via Beeline (Ctrl-C) or API call > TCLIService.Client.CancelOperation, SQLOperation.cancel is invoked in a > different thread from that running the query to close/destroy its > encapsulated Driver object. Both SQLOperation and Driver are not thread-safe > which could sometimes result in Runtime exceptions like NPE. The errors from > the running query are not handled properly therefore probably causing some > stuffs (files, locks etc) not being cleaned after the query termination. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14799) Query operation are not thread safe during its cancellation
[ https://issues.apache.org/jira/browse/HIVE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572287#comment-15572287 ] Chaoyu Tang edited comment on HIVE-14799 at 10/13/16 5:12 PM: -- The failed tests should not be related to this patch and I am not able to reproduce it in my local env. [~sershe] Could you review the revised patch based on your last review comments? Thanks was (Author: ctang.ma): The failed tests should not be related to this patch and I am not able to reproduce it in my local env. > Query operation are not thread safe during its cancellation > --- > > Key: HIVE-14799 > URL: https://issues.apache.org/jira/browse/HIVE-14799 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14799.1.patch, HIVE-14799.2.patch, > HIVE-14799.3.patch, HIVE-14799.4.patch, HIVE-14799.5.patch, > HIVE-14799.5.patch, HIVE-14799.6.patch, HIVE-14799.6.patch, HIVE-14799.patch > > > When a query is cancelled either via Beeline (Ctrl-C) or API call > TCLIService.Client.CancelOperation, SQLOperation.cancel is invoked in a > different thread from that running the query to close/destroy its > encapsulated Driver object. Both SQLOperation and Driver are not thread-safe > which could sometimes result in Runtime exceptions like NPE. The errors from > the running query are not handled properly therefore probably causing some > stuffs (files, locks etc) not being cleaned after the query termination. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14799) Query operation are not thread safe during its cancellation
[ https://issues.apache.org/jira/browse/HIVE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14799: --- Attachment: HIVE-14799.7.patch revised the patch to address an issue raised by [~ychena]. [~sershe] [~ychena], could you review it. > Query operation are not thread safe during its cancellation > --- > > Key: HIVE-14799 > URL: https://issues.apache.org/jira/browse/HIVE-14799 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14799.1.patch, HIVE-14799.2.patch, > HIVE-14799.3.patch, HIVE-14799.4.patch, HIVE-14799.5.patch, > HIVE-14799.5.patch, HIVE-14799.6.patch, HIVE-14799.6.patch, > HIVE-14799.7.patch, HIVE-14799.patch > > > When a query is cancelled either via Beeline (Ctrl-C) or API call > TCLIService.Client.CancelOperation, SQLOperation.cancel is invoked in a > different thread from that running the query to close/destroy its > encapsulated Driver object. Both SQLOperation and Driver are not thread-safe > which could sometimes result in Runtime exceptions like NPE. The errors from > the running query are not handled properly therefore probably causing some > stuffs (files, locks etc) not being cleaned after the query termination. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14799) Query operation are not thread safe during its cancellation
[ https://issues.apache.org/jira/browse/HIVE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15575355#comment-15575355 ] Chaoyu Tang commented on HIVE-14799: The failed tests are aged or not related. > Query operation are not thread safe during its cancellation > --- > > Key: HIVE-14799 > URL: https://issues.apache.org/jira/browse/HIVE-14799 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14799.1.patch, HIVE-14799.2.patch, > HIVE-14799.3.patch, HIVE-14799.4.patch, HIVE-14799.5.patch, > HIVE-14799.5.patch, HIVE-14799.6.patch, HIVE-14799.6.patch, > HIVE-14799.7.patch, HIVE-14799.patch > > > When a query is cancelled either via Beeline (Ctrl-C) or API call > TCLIService.Client.CancelOperation, SQLOperation.cancel is invoked in a > different thread from that running the query to close/destroy its > encapsulated Driver object. Both SQLOperation and Driver are not thread-safe > which could sometimes result in Runtime exceptions like NPE. The errors from > the running query are not handled properly therefore probably causing some > stuffs (files, locks etc) not being cleaned after the query termination. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14799) Query operation are not thread safe during its cancellation
[ https://issues.apache.org/jira/browse/HIVE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15575461#comment-15575461 ] Chaoyu Tang commented on HIVE-14799: Thanks [~ychena]. [~sershe] Please feel free to let me know if you see any further problems, otherwise I am going to commit it tomorrow. Thanks. > Query operation are not thread safe during its cancellation > --- > > Key: HIVE-14799 > URL: https://issues.apache.org/jira/browse/HIVE-14799 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14799.1.patch, HIVE-14799.2.patch, > HIVE-14799.3.patch, HIVE-14799.4.patch, HIVE-14799.5.patch, > HIVE-14799.5.patch, HIVE-14799.6.patch, HIVE-14799.6.patch, > HIVE-14799.7.patch, HIVE-14799.patch > > > When a query is cancelled either via Beeline (Ctrl-C) or API call > TCLIService.Client.CancelOperation, SQLOperation.cancel is invoked in a > different thread from that running the query to close/destroy its > encapsulated Driver object. Both SQLOperation and Driver are not thread-safe > which could sometimes result in Runtime exceptions like NPE. The errors from > the running query are not handled properly therefore probably causing some > stuffs (files, locks etc) not being cleaned after the query termination. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14926) Keep Schema in consistent state where schemaTool fails or succeeds.
[ https://issues.apache.org/jira/browse/HIVE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15575670#comment-15575670 ] Chaoyu Tang commented on HIVE-14926: [~aihuaxu] Left comments in RB. AFAIK, DDLs would not work with the autocommit false for most RDBMS and their drivers. > Keep Schema in consistent state where schemaTool fails or succeeds. > - > > Key: HIVE-14926 > URL: https://issues.apache.org/jira/browse/HIVE-14926 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14926.1.patch, HIVE-14926.2.patch > > > SchemaTool uses autocommit right now when executing the upgrade or init > scripts. Seems we should use database transaction to commit or roll back to > keep schema consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14799) Query operation are not thread safe during its cancellation
[ https://issues.apache.org/jira/browse/HIVE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15579146#comment-15579146 ] Chaoyu Tang commented on HIVE-14799: [~leftylev] Thanks for that. I am working on the backport of the patch to 2.1.1, and will update the status/fix version when the JIRA is resolved. > Query operation are not thread safe during its cancellation > --- > > Key: HIVE-14799 > URL: https://issues.apache.org/jira/browse/HIVE-14799 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14799.1.patch, HIVE-14799.2.patch, > HIVE-14799.3.patch, HIVE-14799.4.patch, HIVE-14799.5.patch, > HIVE-14799.5.patch, HIVE-14799.6.patch, HIVE-14799.6.patch, > HIVE-14799.7.patch, HIVE-14799.patch > > > When a query is cancelled either via Beeline (Ctrl-C) or API call > TCLIService.Client.CancelOperation, SQLOperation.cancel is invoked in a > different thread from that running the query to close/destroy its > encapsulated Driver object. Both SQLOperation and Driver are not thread-safe > which could sometimes result in Runtime exceptions like NPE. The errors from > the running query are not handled properly therefore probably causing some > stuffs (files, locks etc) not being cleaned after the query termination. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14927) Remove code duplication from tests in TestLdapAtnProviderWithMiniDS
[ https://issues.apache.org/jira/browse/HIVE-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15579245#comment-15579245 ] Chaoyu Tang commented on HIVE-14927: The test did not run successfully. Could you reattach the patch to kick off another run? > Remove code duplication from tests in TestLdapAtnProviderWithMiniDS > --- > > Key: HIVE-14927 > URL: https://issues.apache.org/jira/browse/HIVE-14927 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: HIVE-14927.1.patch, HIVE-14927.2.patch > > > * Extract inner class User and implement a proper builder for it. > * Extract all common code to LdapAuthenticationTestCase class > ** setting up the test case >** executing test case >** result validation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14799) Query operation are not thread safe during its cancellation
[ https://issues.apache.org/jira/browse/HIVE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14799: --- Attachment: HIVE-14799-branch-2.1.patch Attach the patch for 2.1. > Query operation are not thread safe during its cancellation > --- > > Key: HIVE-14799 > URL: https://issues.apache.org/jira/browse/HIVE-14799 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14799-branch-2.1.patch, HIVE-14799.1.patch, > HIVE-14799.2.patch, HIVE-14799.3.patch, HIVE-14799.4.patch, > HIVE-14799.5.patch, HIVE-14799.5.patch, HIVE-14799.6.patch, > HIVE-14799.6.patch, HIVE-14799.7.patch, HIVE-14799.patch > > > When a query is cancelled either via Beeline (Ctrl-C) or API call > TCLIService.Client.CancelOperation, SQLOperation.cancel is invoked in a > different thread from that running the query to close/destroy its > encapsulated Driver object. Both SQLOperation and Driver are not thread-safe > which could sometimes result in Runtime exceptions like NPE. The errors from > the running query are not handled properly therefore probably causing some > stuffs (files, locks etc) not being cleaned after the query termination. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14799) Query operation are not thread safe during its cancellation
[ https://issues.apache.org/jira/browse/HIVE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14799: --- Resolution: Fixed Fix Version/s: 2.1.1 2.2.0 Status: Resolved (was: Patch Available) Committed to 2.2.0 and 2.1.1. Thank [~sershe] and [~ychena] for review. > Query operation are not thread safe during its cancellation > --- > > Key: HIVE-14799 > URL: https://issues.apache.org/jira/browse/HIVE-14799 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14799-branch-2.1.patch, HIVE-14799.1.patch, > HIVE-14799.2.patch, HIVE-14799.3.patch, HIVE-14799.4.patch, > HIVE-14799.5.patch, HIVE-14799.5.patch, HIVE-14799.6.patch, > HIVE-14799.6.patch, HIVE-14799.7.patch, HIVE-14799.patch > > > When a query is cancelled either via Beeline (Ctrl-C) or API call > TCLIService.Client.CancelOperation, SQLOperation.cancel is invoked in a > different thread from that running the query to close/destroy its > encapsulated Driver object. Both SQLOperation and Driver are not thread-safe > which could sometimes result in Runtime exceptions like NPE. The errors from > the running query are not handled properly therefore probably causing some > stuffs (files, locks etc) not being cleaned after the query termination. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13423) Handle the overflow case for decimal datatype for sum()
[
https://issues.apache.org/jira/browse/HIVE-13423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582911#comment-15582911
]
Chaoyu Tang commented on HIVE-13423:
[~aihuaxu] The patch looks good. The issue might also exist in all other
arithmetic functions or operations like plus(+), multiplication(*) etc I
believe. I wonder if we need truncate the scale like SQLServer does to fit the
intermediate data to the precision as discussed in HIVE-14281.
> Handle the overflow case for decimal datatype for sum()
> ---
>
> Key: HIVE-13423
> URL: https://issues.apache.org/jira/browse/HIVE-13423
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13423.1.patch
>
>
> When a column col1 defined as decimal and if the sum of the column overflows,
> we will try to increase the decimal precision by 10. But if it's reaching 38
> (the max precision), the overflow still could happen. Right now, if such case
> happens, the following exception will throw since hive is writing incorrect
> data.
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.readVInt(LazyBinaryUtils.java:314)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:219)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13423) Handle the overflow case for decimal datatype for sum()
[
https://issues.apache.org/jira/browse/HIVE-13423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582911#comment-15582911
]
Chaoyu Tang edited comment on HIVE-13423 at 10/17/16 5:59 PM:
--
[~aihuaxu] The patch looks good. The issue might also exist in all other
arithmetic functions or operations like plus(+), multiplication(*) etc I
believe. I wonder if we need truncate the scale like SQLServer does to fit the
intermediate data to the precision as discussed in HIVE-14281. [~xuefuz], you
have more insights into the decimal and what is your thought?
was (Author: ctang.ma):
[~aihuaxu] The patch looks good. The issue might also exist in all other
arithmetic functions or operations like plus(+), multiplication(*) etc I
believe. I wonder if we need truncate the scale like SQLServer does to fit the
intermediate data to the precision as discussed in HIVE-14281.
> Handle the overflow case for decimal datatype for sum()
> ---
>
> Key: HIVE-13423
> URL: https://issues.apache.org/jira/browse/HIVE-13423
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13423.1.patch
>
>
> When a column col1 defined as decimal and if the sum of the column overflows,
> we will try to increase the decimal precision by 10. But if it's reaching 38
> (the max precision), the overflow still could happen. Right now, if such case
> happens, the following exception will throw since hive is writing incorrect
> data.
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.readVInt(LazyBinaryUtils.java:314)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:219)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14927) Remove code duplication from tests in TestLdapAtnProviderWithMiniDS
[ https://issues.apache.org/jira/browse/HIVE-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583744#comment-15583744 ] Chaoyu Tang commented on HIVE-14927: Yeah, it seems that precommit build have some issues. > Remove code duplication from tests in TestLdapAtnProviderWithMiniDS > --- > > Key: HIVE-14927 > URL: https://issues.apache.org/jira/browse/HIVE-14927 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: HIVE-14927.1.patch, HIVE-14927.2.patch, > HIVE-14927.3.patch > > > * Extract inner class User and implement a proper builder for it. > * Extract all common code to LdapAuthenticationTestCase class > ** setting up the test case >** executing test case >** result validation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14927) Remove code duplication from tests in TestLdapAtnProviderWithMiniDS
[ https://issues.apache.org/jira/browse/HIVE-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14927: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to 2.2.0. Thanks [~yalovyyi] for the patch > Remove code duplication from tests in TestLdapAtnProviderWithMiniDS > --- > > Key: HIVE-14927 > URL: https://issues.apache.org/jira/browse/HIVE-14927 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Fix For: 2.2.0 > > Attachments: HIVE-14927.1.patch, HIVE-14927.2.patch, > HIVE-14927.3.patch, HIVE-14927.4.patch > > > * Extract inner class User and implement a proper builder for it. > * Extract all common code to LdapAuthenticationTestCase class > ** setting up the test case >** executing test case >** result validation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13423) Handle the overflow case for decimal datatype for sum()
[
https://issues.apache.org/jira/browse/HIVE-13423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589253#comment-15589253
]
Chaoyu Tang commented on HIVE-13423:
Sound good to me as well.
> Handle the overflow case for decimal datatype for sum()
> ---
>
> Key: HIVE-13423
> URL: https://issues.apache.org/jira/browse/HIVE-13423
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13423.1.patch
>
>
> When a column col1 defined as decimal and if the sum of the column overflows,
> we will try to increase the decimal precision by 10. But if it's reaching 38
> (the max precision), the overflow still could happen. Right now, if such case
> happens, the following exception will throw since hive is writing incorrect
> data.
> Follow the following steps to repro.
> {noformat}
> CREATE TABLE DECIMAL_PRECISION(dec decimal(38,18));
> INSERT INTO DECIMAL_PRECISION VALUES(98765432109876543210.12345),
> (98765432109876543210.12345);
> SELECT SUM(dec) FROM DECIMAL_PRECISION;
> {noformat}
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.readVInt(LazyBinaryUtils.java:314)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:219)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14909) Preserve the "parent location" of the table when an "alter table rename to " is submitted (the case when the db location is not specified and the Hive de
[ https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14909: --- Attachment: HIVE-14909.patch Since Hive supports the DDL like: create table foo (key int) location 'path_to_location' to create a managed table by specifying its location rather than that under its database. So table rename should respect this specified location, and not change its location or move its data. Its location should be change using a different command 'alter table .. set location ...' instead. > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). > -- > > Key: HIVE-14909 > URL: https://issues.apache.org/jira/browse/HIVE-14909 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.1.0 >Reporter: Adriano >Assignee: Chaoyu Tang > Attachments: HIVE-14909.patch > > > Alter Table operation for db_enc.rename_test failed to move data due to: > '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption > zone.' > When Hive renames a managed table, it always creates the new renamed table > directory under its database directory in order to keep a db/table hierarchy. > In this case, the renamed table directory is created under "default db" > directory "hive/warehouse/". When Hive renames a managed table, it always > creates the new renamed table directory under its database directory in order > to keep a db/table hierarchy. In this case, the renamed table directory is > created under "default' db directory typically set as /hive/warehouse/ . > This error doesn't appear if first create a database which points to a > directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't > have this problem. For example, > create database db_enc location '/hdfs/encrypted_path/db_enc; > use db_enc; > create table rename_test (...) location > '/hdfs/encrypted_path/db_enc/rename_test'; > alter table rename_test rename to test_rename; > The renamed test_rename directory is created under > /hdfs/encrypted_path/db_enc. > Considering that the encryption of a filesystem is part of the evolution > hardening of a system (where the system and the data contained can already > exists) and a db can be already created without location set (because it is > not strictly required)and the default db is outside the same encryption zone > (or in a no-encryption zone) the alter table rename operation will fail. > Improvement: > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14909) Preserve the "parent location" of the table when an "alter table rename to " is submitted (the case when the db location is not specified and the Hive de
[ https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14909: --- Status: Patch Available (was: Open) > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). > -- > > Key: HIVE-14909 > URL: https://issues.apache.org/jira/browse/HIVE-14909 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.1.0 >Reporter: Adriano >Assignee: Chaoyu Tang > Attachments: HIVE-14909.patch > > > Alter Table operation for db_enc.rename_test failed to move data due to: > '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption > zone.' > When Hive renames a managed table, it always creates the new renamed table > directory under its database directory in order to keep a db/table hierarchy. > In this case, the renamed table directory is created under "default db" > directory "hive/warehouse/". When Hive renames a managed table, it always > creates the new renamed table directory under its database directory in order > to keep a db/table hierarchy. In this case, the renamed table directory is > created under "default' db directory typically set as /hive/warehouse/ . > This error doesn't appear if first create a database which points to a > directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't > have this problem. For example, > create database db_enc location '/hdfs/encrypted_path/db_enc; > use db_enc; > create table rename_test (...) location > '/hdfs/encrypted_path/db_enc/rename_test'; > alter table rename_test rename to test_rename; > The renamed test_rename directory is created under > /hdfs/encrypted_path/db_enc. > Considering that the encryption of a filesystem is part of the evolution > hardening of a system (where the system and the data contained can already > exists) and a db can be already created without location set (because it is > not strictly required)and the default db is outside the same encryption zone > (or in a no-encryption zone) the alter table rename operation will fail. > Improvement: > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14609) HS2 cannot drop a function whose associated jar file has been removed
[
https://issues.apache.org/jira/browse/HIVE-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-14609:
---
Attachment: (was: HIVE-12812.patch)
> HS2 cannot drop a function whose associated jar file has been removed
> -
>
> Key: HIVE-14609
> URL: https://issues.apache.org/jira/browse/HIVE-14609
> Project: Hive
> Issue Type: Bug
>Reporter: Yibing Shi
>Assignee: Chaoyu Tang
>
> Create a permanent function with below command:
> {code:sql}
> create function yshi.dummy as 'com.yshi.hive.udf.DummyUDF' using jar
> 'hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar';
> {code}
> After that, delete the HDFS file
> {{hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar}}, and
> *restart HS2 to remove the loaded class*.
> Now the function cannot be dropped:
> {noformat}
> 0: jdbc:hive2://10.17.81.144:1/default> show functions yshi.dummy;
> INFO : Compiling
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded):
> show functions yshi.dummy
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from
> deserializer)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded);
> Time taken: 1.259 seconds
> INFO : Executing
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded):
> show functions yshi.dummy
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.
> INFO : Completed executing
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded);
> Time taken: 0.024 seconds
> INFO : OK
> +-+--+
> | tab_name |
> +-+--+
> | yshi.dummy |
> +-+--+
> 1 row selected (3.877 seconds)
> 0: jdbc:hive2://10.17.81.144:1/default> drop function yshi.dummy;
> INFO : Compiling
> command(queryId=hive_20160821213434_47d14df5-59b3-4ebc-9a48-5e1d9c60c1fc):
> drop function yshi.dummy
> INFO : converting to local
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> ERROR : Failed to read external resource
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> java.lang.RuntimeException: Failed to read external resource
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1200)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1136)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1126)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:304)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:470)
> at
> org.apache.hadoop.hive.ql.exec.Registry.getQualifiedFunctionInfo(Registry.java:456)
> at
> org.apache.hadoop.hive.ql.exec.Registry.getFunctionInfo(Registry.java:245)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:455)
> at
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:99)
> at
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:61)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:451)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
> at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1194)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1181)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:134)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:206)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:316)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:425)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:401)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:258)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:506)
>
[jira] [Updated] (HIVE-14609) HS2 cannot drop a function whose associated jar file has been removed
[
https://issues.apache.org/jira/browse/HIVE-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-14609:
---
Attachment: HIVE-12812.patch
> HS2 cannot drop a function whose associated jar file has been removed
> -
>
> Key: HIVE-14609
> URL: https://issues.apache.org/jira/browse/HIVE-14609
> Project: Hive
> Issue Type: Bug
>Reporter: Yibing Shi
>Assignee: Chaoyu Tang
> Attachments: HIVE-12812.patch
>
>
> Create a permanent function with below command:
> {code:sql}
> create function yshi.dummy as 'com.yshi.hive.udf.DummyUDF' using jar
> 'hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar';
> {code}
> After that, delete the HDFS file
> {{hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar}}, and
> *restart HS2 to remove the loaded class*.
> Now the function cannot be dropped:
> {noformat}
> 0: jdbc:hive2://10.17.81.144:1/default> show functions yshi.dummy;
> INFO : Compiling
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded):
> show functions yshi.dummy
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from
> deserializer)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded);
> Time taken: 1.259 seconds
> INFO : Executing
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded):
> show functions yshi.dummy
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.
> INFO : Completed executing
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded);
> Time taken: 0.024 seconds
> INFO : OK
> +-+--+
> | tab_name |
> +-+--+
> | yshi.dummy |
> +-+--+
> 1 row selected (3.877 seconds)
> 0: jdbc:hive2://10.17.81.144:1/default> drop function yshi.dummy;
> INFO : Compiling
> command(queryId=hive_20160821213434_47d14df5-59b3-4ebc-9a48-5e1d9c60c1fc):
> drop function yshi.dummy
> INFO : converting to local
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> ERROR : Failed to read external resource
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> java.lang.RuntimeException: Failed to read external resource
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1200)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1136)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1126)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:304)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:470)
> at
> org.apache.hadoop.hive.ql.exec.Registry.getQualifiedFunctionInfo(Registry.java:456)
> at
> org.apache.hadoop.hive.ql.exec.Registry.getFunctionInfo(Registry.java:245)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:455)
> at
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:99)
> at
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:61)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:451)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
> at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1194)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1181)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:134)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:206)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:316)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:425)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:401)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:258)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIS
[jira] [Updated] (HIVE-12812) Enable mapred.input.dir.recursive by default to support union with aggregate function
[ https://issues.apache.org/jira/browse/HIVE-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-12812: --- Attachment: HIVE-12812.patch > Enable mapred.input.dir.recursive by default to support union with aggregate > function > - > > Key: HIVE-12812 > URL: https://issues.apache.org/jira/browse/HIVE-12812 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.1, 2.1.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12812.patch, HIVE-12812.patch, HIVE-12812.patch > > > When union remove optimization is enabled, union query with aggregate > function writes its subquery intermediate results to subdirs which needs > mapred.input.dir.recursive to be enabled in order to be fetched. This > property is not defined by default in Hive and often ignored by user, which > causes the query failure and is hard to be debugged. > So we need set mapred.input.dir.recursive to true whenever union remove > optimization is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14909) Preserve the "parent location" of the table when an "alter table rename to " is submitted (the case when the db location is not specified and the Hive de
[
https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-14909:
---
Attachment: HIVE-14909.patch
All four new failed tests were caused by the errors like:
{code}
2016-10-21T19:08:51,810 ERROR [ccc0909e-6764-441b-9eee-10b73d4ca198 main]
exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter
table. java.lang.IllegalArgumentException: Can not create a Path from a null
string
at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:620)
at org.apache.hadoop.hive.ql.exec.DDLTask.alterTable(DDLTask.java:3373)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:376)
{code}
But interestingly, I was not able to reproduce the issue in my local env.
Reattach the patch to kick off another run to see if they can be reproduced.
> Preserve the "parent location" of the table when an "alter table
> rename to " is submitted (the case when the db location is not
> specified and the Hive defult db is outside the same encrypted zone).
> --
>
> Key: HIVE-14909
> URL: https://issues.apache.org/jira/browse/HIVE-14909
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 1.1.0
>Reporter: Adriano
>Assignee: Chaoyu Tang
> Attachments: HIVE-14909.patch, HIVE-14909.patch
>
>
> Alter Table operation for db_enc.rename_test failed to move data due to:
> '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption
> zone.'
> When Hive renames a managed table, it always creates the new renamed table
> directory under its database directory in order to keep a db/table hierarchy.
> In this case, the renamed table directory is created under "default db"
> directory "hive/warehouse/". When Hive renames a managed table, it always
> creates the new renamed table directory under its database directory in order
> to keep a db/table hierarchy. In this case, the renamed table directory is
> created under "default' db directory typically set as /hive/warehouse/ .
> This error doesn't appear if first create a database which points to a
> directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't
> have this problem. For example,
> create database db_enc location '/hdfs/encrypted_path/db_enc;
> use db_enc;
> create table rename_test (...) location
> '/hdfs/encrypted_path/db_enc/rename_test';
> alter table rename_test rename to test_rename;
> The renamed test_rename directory is created under
> /hdfs/encrypted_path/db_enc.
> Considering that the encryption of a filesystem is part of the evolution
> hardening of a system (where the system and the data contained can already
> exists) and a db can be already created without location set (because it is
> not strictly required)and the default db is outside the same encryption zone
> (or in a no-encryption zone) the alter table rename operation will fail.
> Improvement:
> Preserve the "parent location" of the table when an "alter table
> rename to " is submitted (the case when the db location is not
> specified and the Hive defult db is outside the same encrypted zone).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14909) Preserve the "parent location" of the table when an "alter table rename to " is submitted (the case when the db location is not specified and the Hive de
[ https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14909: --- Attachment: HIVE-14909.1.patch A new patch to fix the failed tests. > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). > -- > > Key: HIVE-14909 > URL: https://issues.apache.org/jira/browse/HIVE-14909 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.1.0 >Reporter: Adriano >Assignee: Chaoyu Tang > Attachments: HIVE-14909.1.patch, HIVE-14909.patch, HIVE-14909.patch > > > Alter Table operation for db_enc.rename_test failed to move data due to: > '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption > zone.' > When Hive renames a managed table, it always creates the new renamed table > directory under its database directory in order to keep a db/table hierarchy. > In this case, the renamed table directory is created under "default db" > directory "hive/warehouse/". When Hive renames a managed table, it always > creates the new renamed table directory under its database directory in order > to keep a db/table hierarchy. In this case, the renamed table directory is > created under "default' db directory typically set as /hive/warehouse/ . > This error doesn't appear if first create a database which points to a > directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't > have this problem. For example, > create database db_enc location '/hdfs/encrypted_path/db_enc; > use db_enc; > create table rename_test (...) location > '/hdfs/encrypted_path/db_enc/rename_test'; > alter table rename_test rename to test_rename; > The renamed test_rename directory is created under > /hdfs/encrypted_path/db_enc. > Considering that the encryption of a filesystem is part of the evolution > hardening of a system (where the system and the data contained can already > exists) and a db can be already created without location set (because it is > not strictly required)and the default db is outside the same encryption zone > (or in a no-encryption zone) the alter table rename operation will fail. > Improvement: > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14909) Preserve the "parent location" of the table when an "alter table rename to " is submitted (the case when the db location is not specified and the Hive de
[ https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14909: --- Attachment: (was: HIVE-14909.1.patch) > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). > -- > > Key: HIVE-14909 > URL: https://issues.apache.org/jira/browse/HIVE-14909 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.1.0 >Reporter: Adriano >Assignee: Chaoyu Tang > Attachments: HIVE-14909.patch, HIVE-14909.patch > > > Alter Table operation for db_enc.rename_test failed to move data due to: > '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption > zone.' > When Hive renames a managed table, it always creates the new renamed table > directory under its database directory in order to keep a db/table hierarchy. > In this case, the renamed table directory is created under "default db" > directory "hive/warehouse/". When Hive renames a managed table, it always > creates the new renamed table directory under its database directory in order > to keep a db/table hierarchy. In this case, the renamed table directory is > created under "default' db directory typically set as /hive/warehouse/ . > This error doesn't appear if first create a database which points to a > directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't > have this problem. For example, > create database db_enc location '/hdfs/encrypted_path/db_enc; > use db_enc; > create table rename_test (...) location > '/hdfs/encrypted_path/db_enc/rename_test'; > alter table rename_test rename to test_rename; > The renamed test_rename directory is created under > /hdfs/encrypted_path/db_enc. > Considering that the encryption of a filesystem is part of the evolution > hardening of a system (where the system and the data contained can already > exists) and a db can be already created without location set (because it is > not strictly required)and the default db is outside the same encryption zone > (or in a no-encryption zone) the alter table rename operation will fail. > Improvement: > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14909) Preserve the "parent location" of the table when an "alter table rename to " is submitted (the case when the db location is not specified and the Hive de
[ https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14909: --- Attachment: HIVE-14909.1.patch > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). > -- > > Key: HIVE-14909 > URL: https://issues.apache.org/jira/browse/HIVE-14909 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.1.0 >Reporter: Adriano >Assignee: Chaoyu Tang > Attachments: HIVE-14909.1.patch, HIVE-14909.patch, HIVE-14909.patch > > > Alter Table operation for db_enc.rename_test failed to move data due to: > '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption > zone.' > When Hive renames a managed table, it always creates the new renamed table > directory under its database directory in order to keep a db/table hierarchy. > In this case, the renamed table directory is created under "default db" > directory "hive/warehouse/". When Hive renames a managed table, it always > creates the new renamed table directory under its database directory in order > to keep a db/table hierarchy. In this case, the renamed table directory is > created under "default' db directory typically set as /hive/warehouse/ . > This error doesn't appear if first create a database which points to a > directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't > have this problem. For example, > create database db_enc location '/hdfs/encrypted_path/db_enc; > use db_enc; > create table rename_test (...) location > '/hdfs/encrypted_path/db_enc/rename_test'; > alter table rename_test rename to test_rename; > The renamed test_rename directory is created under > /hdfs/encrypted_path/db_enc. > Considering that the encryption of a filesystem is part of the evolution > hardening of a system (where the system and the data contained can already > exists) and a db can be already created without location set (because it is > not strictly required)and the default db is outside the same encryption zone > (or in a no-encryption zone) the alter table rename operation will fail. > Improvement: > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14909) Preserve the "parent location" of the table when an "alter table rename to " is submitted (the case when the db location is not specified and the Hive de
[ https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14909: --- Attachment: HIVE-14909.1.patch Reattach patch to kick off the precommit tests > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). > -- > > Key: HIVE-14909 > URL: https://issues.apache.org/jira/browse/HIVE-14909 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.1.0 >Reporter: Adriano >Assignee: Chaoyu Tang > Attachments: HIVE-14909.1.patch, HIVE-14909.1.patch, > HIVE-14909.patch, HIVE-14909.patch > > > Alter Table operation for db_enc.rename_test failed to move data due to: > '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption > zone.' > When Hive renames a managed table, it always creates the new renamed table > directory under its database directory in order to keep a db/table hierarchy. > In this case, the renamed table directory is created under "default db" > directory "hive/warehouse/". When Hive renames a managed table, it always > creates the new renamed table directory under its database directory in order > to keep a db/table hierarchy. In this case, the renamed table directory is > created under "default' db directory typically set as /hive/warehouse/ . > This error doesn't appear if first create a database which points to a > directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't > have this problem. For example, > create database db_enc location '/hdfs/encrypted_path/db_enc; > use db_enc; > create table rename_test (...) location > '/hdfs/encrypted_path/db_enc/rename_test'; > alter table rename_test rename to test_rename; > The renamed test_rename directory is created under > /hdfs/encrypted_path/db_enc. > Considering that the encryption of a filesystem is part of the evolution > hardening of a system (where the system and the data contained can already > exists) and a db can be already created without location set (because it is > not strictly required)and the default db is outside the same encryption zone > (or in a no-encryption zone) the alter table rename operation will fail. > Improvement: > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14909) Preserve the "parent location" of the table when an "alter table rename to " is submitted (the case when the db location is not specified and the Hive de
[ https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14909: --- Attachment: HIVE-14909.2.patch I am not able to reproduce the test failure of TestSemanticAnalysis.testAlterTableRename in my local machine. Create a patch with more debug to see what happened. > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). > -- > > Key: HIVE-14909 > URL: https://issues.apache.org/jira/browse/HIVE-14909 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.1.0 >Reporter: Adriano >Assignee: Chaoyu Tang > Attachments: HIVE-14909.1.patch, HIVE-14909.1.patch, > HIVE-14909.2.patch, HIVE-14909.patch, HIVE-14909.patch > > > Alter Table operation for db_enc.rename_test failed to move data due to: > '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption > zone.' > When Hive renames a managed table, it always creates the new renamed table > directory under its database directory in order to keep a db/table hierarchy. > In this case, the renamed table directory is created under "default db" > directory "hive/warehouse/". When Hive renames a managed table, it always > creates the new renamed table directory under its database directory in order > to keep a db/table hierarchy. In this case, the renamed table directory is > created under "default' db directory typically set as /hive/warehouse/ . > This error doesn't appear if first create a database which points to a > directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't > have this problem. For example, > create database db_enc location '/hdfs/encrypted_path/db_enc; > use db_enc; > create table rename_test (...) location > '/hdfs/encrypted_path/db_enc/rename_test'; > alter table rename_test rename to test_rename; > The renamed test_rename directory is created under > /hdfs/encrypted_path/db_enc. > Considering that the encryption of a filesystem is part of the evolution > hardening of a system (where the system and the data contained can already > exists) and a db can be already created without location set (because it is > not strictly required)and the default db is outside the same encryption zone > (or in a no-encryption zone) the alter table rename operation will fail. > Improvement: > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14909) Preserve the "parent location" of the table when an "alter table rename to " is submitted (the case when the db location is not specified and the Hive
[
https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15605293#comment-15605293
]
Chaoyu Tang commented on HIVE-14909:
That the TestSemanticAnalysis.testAlterTableRename failed in the precommit
build is the old table was not created under its database directory in the
test, will look into:
{code}
2016-10-25T02:53:40,484 DEBUG [main] log.PerfLogger:
2016-10-25T02:53:40,484 INFO [main] metastore.HiveMetaStore: 0: alter_table:
db=default tbl=oldname newtbl=newNAME
2016-10-25T02:53:40,484 INFO [main] HiveMetaStore.audit: ugi=hiveptest
ip=unknown-ip-addr cmd=alter_table: db=default tbl=oldname newtbl=newNAME
2016-10-25T02:53:40,491 DEBUG [main] metastore.HiveAlterHandler: HiveAlterTable
is renaming default.oldname to default.newNAME
2016-10-25T02:53:40,496 DEBUG [main] metastore.HiveAlterHandler: Old table
location:
file:/home/hiveptest/104.197.110.94-hiveptest-0/apache-github-source-source/hcatalog/core/build/test/data/org.apache.hive.hcatalog.mapreduce.HCatBaseTest-1477389203834/warehouse/oldname
2016-10-25T02:53:40,496 DEBUG [main] metastore.HiveAlterHandler: New table
location:
file:/home/hiveptest/104.197.110.94-hiveptest-0/apache-github-source-source/hcatalog/core/build/test/data/org.apache.hive.hcatalog.mapreduce.HCatBaseTest-1477389203834/warehouse/oldname
2016-10-25T02:53:40,496 DEBUG [main] metastore.MetaStoreDirectSql: getDatabase:
directsql returning db default
locn[pfile:/home/hiveptest/104.197.110.94-hiveptest-0/apache-github-source-source/hcatalog/core/target/warehouse]
desc [Default Hive database] owner [public] ownertype [ROLE]
2016-10-25T02:53:40,496 DEBUG [main] metastore.HiveAlterHandler: Old database
location:
pfile:/home/hiveptest/104.197.110.94-hiveptest-0/apache-github-source-source/hcatalog/core/target/warehouse
{code}
> Preserve the "parent location" of the table when an "alter table
> rename to " is submitted (the case when the db location is not
> specified and the Hive defult db is outside the same encrypted zone).
> --
>
> Key: HIVE-14909
> URL: https://issues.apache.org/jira/browse/HIVE-14909
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 1.1.0
>Reporter: Adriano
>Assignee: Chaoyu Tang
> Attachments: HIVE-14909.1.patch, HIVE-14909.1.patch,
> HIVE-14909.2.patch, HIVE-14909.patch, HIVE-14909.patch
>
>
> Alter Table operation for db_enc.rename_test failed to move data due to:
> '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption
> zone.'
> When Hive renames a managed table, it always creates the new renamed table
> directory under its database directory in order to keep a db/table hierarchy.
> In this case, the renamed table directory is created under "default db"
> directory "hive/warehouse/". When Hive renames a managed table, it always
> creates the new renamed table directory under its database directory in order
> to keep a db/table hierarchy. In this case, the renamed table directory is
> created under "default' db directory typically set as /hive/warehouse/ .
> This error doesn't appear if first create a database which points to a
> directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't
> have this problem. For example,
> create database db_enc location '/hdfs/encrypted_path/db_enc;
> use db_enc;
> create table rename_test (...) location
> '/hdfs/encrypted_path/db_enc/rename_test';
> alter table rename_test rename to test_rename;
> The renamed test_rename directory is created under
> /hdfs/encrypted_path/db_enc.
> Considering that the encryption of a filesystem is part of the evolution
> hardening of a system (where the system and the data contained can already
> exists) and a db can be already created without location set (because it is
> not strictly required)and the default db is outside the same encryption zone
> (or in a no-encryption zone) the alter table rename operation will fail.
> Improvement:
> Preserve the "parent location" of the table when an "alter table
> rename to " is submitted (the case when the db location is not
> specified and the Hive defult db is outside the same encrypted zone).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15025) Secure-Socket-Layer (SSL) support for HMS
[ https://issues.apache.org/jira/browse/HIVE-15025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15605595#comment-15605595 ] Chaoyu Tang commented on HIVE-15025: Patch looks good, but the failed tests are related? > Secure-Socket-Layer (SSL) support for HMS > - > > Key: HIVE-15025 > URL: https://issues.apache.org/jira/browse/HIVE-15025 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-15025.1.patch, HIVE-15025.2.patch, > HIVE-15025.3.patch > > > HMS server should support SSL encryption. When the server is keberos enabled, > the encryption can be enabled. But if keberos is not enabled, then there is > no encryption between HS2 and HMS. > Similar to HS2, we should support encryption in both cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14909) Preserve the "parent location" of the table when an "alter table rename to " is submitted (the case when the db location is not specified and the Hive de
[ https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14909: --- Attachment: HIVE-14909.3.patch I believe somehow that TestSemanticAnalysis#testAlterTableRename is not right (See HIVE-15059). Modify this test to check the existence of the renamed table and its name, instead of the renamed table location. > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). > -- > > Key: HIVE-14909 > URL: https://issues.apache.org/jira/browse/HIVE-14909 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.1.0 >Reporter: Adriano >Assignee: Chaoyu Tang > Attachments: HIVE-14909.1.patch, HIVE-14909.1.patch, > HIVE-14909.2.patch, HIVE-14909.3.patch, HIVE-14909.patch, HIVE-14909.patch > > > Alter Table operation for db_enc.rename_test failed to move data due to: > '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption > zone.' > When Hive renames a managed table, it always creates the new renamed table > directory under its database directory in order to keep a db/table hierarchy. > In this case, the renamed table directory is created under "default db" > directory "hive/warehouse/". When Hive renames a managed table, it always > creates the new renamed table directory under its database directory in order > to keep a db/table hierarchy. In this case, the renamed table directory is > created under "default' db directory typically set as /hive/warehouse/ . > This error doesn't appear if first create a database which points to a > directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't > have this problem. For example, > create database db_enc location '/hdfs/encrypted_path/db_enc; > use db_enc; > create table rename_test (...) location > '/hdfs/encrypted_path/db_enc/rename_test'; > alter table rename_test rename to test_rename; > The renamed test_rename directory is created under > /hdfs/encrypted_path/db_enc. > Considering that the encryption of a filesystem is part of the evolution > hardening of a system (where the system and the data contained can already > exists) and a db can be already created without location set (because it is > not strictly required)and the default db is outside the same encryption zone > (or in a no-encryption zone) the alter table rename operation will fail. > Improvement: > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14909) Preserve the "parent location" of the table when an "alter table rename to " is submitted (the case when the db location is not specified and the Hive
[ https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15607009#comment-15607009 ] Chaoyu Tang commented on HIVE-14909: Failed tests are not related. [~spena], [~aihuaxu], could you review the patch? Thanks > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). > -- > > Key: HIVE-14909 > URL: https://issues.apache.org/jira/browse/HIVE-14909 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.1.0 >Reporter: Adriano >Assignee: Chaoyu Tang > Attachments: HIVE-14909.1.patch, HIVE-14909.1.patch, > HIVE-14909.2.patch, HIVE-14909.3.patch, HIVE-14909.patch, HIVE-14909.patch > > > Alter Table operation for db_enc.rename_test failed to move data due to: > '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption > zone.' > When Hive renames a managed table, it always creates the new renamed table > directory under its database directory in order to keep a db/table hierarchy. > In this case, the renamed table directory is created under "default db" > directory "hive/warehouse/". When Hive renames a managed table, it always > creates the new renamed table directory under its database directory in order > to keep a db/table hierarchy. In this case, the renamed table directory is > created under "default' db directory typically set as /hive/warehouse/ . > This error doesn't appear if first create a database which points to a > directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't > have this problem. For example, > create database db_enc location '/hdfs/encrypted_path/db_enc; > use db_enc; > create table rename_test (...) location > '/hdfs/encrypted_path/db_enc/rename_test'; > alter table rename_test rename to test_rename; > The renamed test_rename directory is created under > /hdfs/encrypted_path/db_enc. > Considering that the encryption of a filesystem is part of the evolution > hardening of a system (where the system and the data contained can already > exists) and a db can be already created without location set (because it is > not strictly required)and the default db is outside the same encryption zone > (or in a no-encryption zone) the alter table rename operation will fail. > Improvement: > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-15061) Metastore types are sometimes case sensitive
[ https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang reassigned HIVE-15061: -- Assignee: Chaoyu Tang > Metastore types are sometimes case sensitive > > > Key: HIVE-15061 > URL: https://issues.apache.org/jira/browse/HIVE-15061 > Project: Hive > Issue Type: Bug > Components: API >Affects Versions: 1.1.0 >Reporter: Thomas Tauber-Marshall >Assignee: Chaoyu Tang > > Impala recently encountered an issue with the metastore > ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column > stats would get dropped when adding a column to a table. > The reason seems to be that Hive does a case sensitive check on the column > stats types during an "alter table" and expects the types to be all lower > case. This case sensitive check doesn't appear to happen when the stats are > set in the first place. > We're solving this on the Impala end by storing types in the metastore as all > lower case, but Hive's behavior here is very confusing. It should either > always be case sensitive, so that you can't create column stats with types > that Hive considers invalid, or it should never be case sensitive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15061) Metastore types are sometimes case sensitive
[ https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15608633#comment-15608633 ] Chaoyu Tang commented on HIVE-15061: I can take a look. > Metastore types are sometimes case sensitive > > > Key: HIVE-15061 > URL: https://issues.apache.org/jira/browse/HIVE-15061 > Project: Hive > Issue Type: Bug > Components: API >Affects Versions: 1.1.0 >Reporter: Thomas Tauber-Marshall >Assignee: Chaoyu Tang > > Impala recently encountered an issue with the metastore > ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column > stats would get dropped when adding a column to a table. > The reason seems to be that Hive does a case sensitive check on the column > stats types during an "alter table" and expects the types to be all lower > case. This case sensitive check doesn't appear to happen when the stats are > set in the first place. > We're solving this on the Impala end by storing types in the metastore as all > lower case, but Hive's behavior here is very confusing. It should either > always be case sensitive, so that you can't create column stats with types > that Hive considers invalid, or it should never be case sensitive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15061) Metastore types are sometimes case sensitive
[ https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15609006#comment-15609006 ] Chaoyu Tang commented on HIVE-15061: In Hive alter table, both column name and type in the newTable object have been converted to lower case before it is passed to HMS APIs, therefore there won't be the case sensitive issues as seen in Impala. Impala might do the same way as Hive to work around this issue (which seems Impala is doing), or we might seek a way to validate column names/types and do the case conversion in HMS alter table. > Metastore types are sometimes case sensitive > > > Key: HIVE-15061 > URL: https://issues.apache.org/jira/browse/HIVE-15061 > Project: Hive > Issue Type: Bug > Components: API >Affects Versions: 1.1.0 >Reporter: Thomas Tauber-Marshall >Assignee: Chaoyu Tang > > Impala recently encountered an issue with the metastore > ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column > stats would get dropped when adding a column to a table. > The reason seems to be that Hive does a case sensitive check on the column > stats types during an "alter table" and expects the types to be all lower > case. This case sensitive check doesn't appear to happen when the stats are > set in the first place. > We're solving this on the Impala end by storing types in the metastore as all > lower case, but Hive's behavior here is very confusing. It should either > always be case sensitive, so that you can't create column stats with types > that Hive considers invalid, or it should never be case sensitive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15025) Secure-Socket-Layer (SSL) support for HMS
[ https://issues.apache.org/jira/browse/HIVE-15025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15609196#comment-15609196 ] Chaoyu Tang commented on HIVE-15025: +1 > Secure-Socket-Layer (SSL) support for HMS > - > > Key: HIVE-15025 > URL: https://issues.apache.org/jira/browse/HIVE-15025 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-15025.1.patch, HIVE-15025.2.patch, > HIVE-15025.3.patch > > > HMS server should support SSL encryption. When the server is keberos enabled, > the encryption can be enabled. But if keberos is not enabled, then there is > no encryption between HS2 and HMS. > Similar to HS2, we should support encryption in both cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15061) Metastore types are sometimes case sensitive
[ https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-15061: --- Attachment: HIVE-15061.patch This patch is to fix the following issues: 1. alterTableUpdateTableColumnStats and updatePartColumnStatsForAlterColumns did not ignore the case when they compare the columns to determine if their stats should be updated or deleted 2. HMS alterTable allows the uppercase column name and type to be populated to HMS backend DB 3. update_table_column_statistics/update_partition_column_statistics allow the uppercase column type to be populated to HMS backend DB > Metastore types are sometimes case sensitive > > > Key: HIVE-15061 > URL: https://issues.apache.org/jira/browse/HIVE-15061 > Project: Hive > Issue Type: Bug > Components: API >Affects Versions: 1.1.0 >Reporter: Thomas Tauber-Marshall >Assignee: Chaoyu Tang > Attachments: HIVE-15061.patch > > > Impala recently encountered an issue with the metastore > ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column > stats would get dropped when adding a column to a table. > The reason seems to be that Hive does a case sensitive check on the column > stats types during an "alter table" and expects the types to be all lower > case. This case sensitive check doesn't appear to happen when the stats are > set in the first place. > We're solving this on the Impala end by storing types in the metastore as all > lower case, but Hive's behavior here is very confusing. It should either > always be case sensitive, so that you can't create column stats with types > that Hive considers invalid, or it should never be case sensitive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15061) Metastore types are sometimes case sensitive
[ https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-15061: --- Status: Patch Available (was: Open) [~aihuaxu], [~mohitsabharwal], could you review the patch since you have done something related. Thanks > Metastore types are sometimes case sensitive > > > Key: HIVE-15061 > URL: https://issues.apache.org/jira/browse/HIVE-15061 > Project: Hive > Issue Type: Bug > Components: API >Affects Versions: 1.1.0 >Reporter: Thomas Tauber-Marshall >Assignee: Chaoyu Tang > Attachments: HIVE-15061.patch > > > Impala recently encountered an issue with the metastore > ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column > stats would get dropped when adding a column to a table. > The reason seems to be that Hive does a case sensitive check on the column > stats types during an "alter table" and expects the types to be all lower > case. This case sensitive check doesn't appear to happen when the stats are > set in the first place. > We're solving this on the Impala end by storing types in the metastore as all > lower case, but Hive's behavior here is very confusing. It should either > always be case sensitive, so that you can't create column stats with types > that Hive considers invalid, or it should never be case sensitive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14909) Preserve the location of table created with the location clause in table rename
[ https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14909: --- Description: Alter Table operation for db_enc.rename_test failed to move data due to: '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption zone.' When Hive renames a managed table, it always creates the new renamed table directory under its database directory in order to keep a db/table hierarchy. In this case, the renamed table directory is created under "default db" directory "hive/warehouse/". When Hive renames a managed table, it always creates the new renamed table directory under its database directory in order to keep a db/table hierarchy. In this case, the renamed table directory is created under "default' db directory typically set as /hive/warehouse/ . This error doesn't appear if first create a database which points to a directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't have this problem. For example, create database db_enc location '/hdfs/encrypted_path/db_enc; use db_enc; create table rename_test (...) location '/hdfs/encrypted_path/db_enc/rename_test'; alter table rename_test rename to test_rename; The renamed test_rename directory is created under /hdfs/encrypted_path/db_enc. Considering that the encryption of a filesystem is part of the evolution hardening of a system (where the system and the data contained can already exists) and a db can be already created without location set (because it is not strictly required)and the default db is outside the same encryption zone (or in a no-encryption zone) the alter table rename operation will fail. Improvement: Preserve the "parent location" of the table when an "alter table rename to " is submitted (the case when the db location is not specified and the Hive defult db is outside the same encrypted zone). was: Alter Table operation for db_enc.rename_test failed to move data due to: '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption zone.' When Hive renames a managed table, it always creates the new renamed table directory under its database directory in order to keep a db/table hierarchy. In this case, the renamed table directory is created under "default db" directory "hive/warehouse/". When Hive renames a managed table, it always creates the new renamed table directory under its database directory in order to keep a db/table hierarchy. In this case, the renamed table directory is created under "default' db directory typically set as /hive/warehouse/ . This error doesn't appear if first create a database which points to a directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't have this problem. For example, create database db_enc location '/hdfs/encrypted_path/db_enc; use db_enc; create table rename_test (...) location '/hdfs/encrypted_path/db_enc/rename_test'; alter table rename_test rename to test_rename; The renamed test_rename directory is created under /hdfs/encrypted_path/db_enc. Considering that the encryption of a filesystem is part of the evolution hardening of a system (where the system and the data contained can already exists) and a db can be already created without location set (because it is not strictly required)and the default db is outside the same encryption zone (or in a no-encryption zone) the alter table rename operation will fail. Improvement: Preserve the "parent location" of the table when an "alter table rename to " is submitted (the case when the db location is not specified and the Hive defult db is outside the same encrypted zone). Summary: Preserve the location of table created with the location clause in table rename (was: Preserve the "parent location" of the table when an "alter table rename to " is submitted (the case when the db location is not specified and the Hive defult db is outside the same encrypted zone).) change the JIRA summary > Preserve the location of table created with the location clause in table > rename > --- > > Key: HIVE-14909 > URL: https://issues.apache.org/jira/browse/HIVE-14909 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.1.0 >Reporter: Adriano >Assignee: Chaoyu Tang > Attachments: HIVE-14909.1.patch, HIVE-14909.1.patch, > HIVE-14909.2.patch, HIVE-14909.3.patch, HIVE-14909.patch, HIVE-14909.patch > > > Alter Table operation for db_enc.rename_test failed to move data due to: > '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption > zone.' > When Hive renames a managed table, it always creates the new renamed table > directory under its database directory in order to keep a db/table hierarchy. > I
[jira] [Updated] (HIVE-14909) Preserve the location of table created with the location clause in table rename
[ https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14909: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to 2.2.0 and thanks [~aihuaxu] for reviewing the patch. > Preserve the location of table created with the location clause in table > rename > --- > > Key: HIVE-14909 > URL: https://issues.apache.org/jira/browse/HIVE-14909 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.1.0 >Reporter: Adriano >Assignee: Chaoyu Tang > Fix For: 2.2.0 > > Attachments: HIVE-14909.1.patch, HIVE-14909.1.patch, > HIVE-14909.2.patch, HIVE-14909.3.patch, HIVE-14909.patch, HIVE-14909.patch > > > Alter Table operation for db_enc.rename_test failed to move data due to: > '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption > zone.' > When Hive renames a managed table, it always creates the new renamed table > directory under its database directory in order to keep a db/table hierarchy. > In this case, the renamed table directory is created under "default db" > directory "hive/warehouse/". When Hive renames a managed table, it always > creates the new renamed table directory under its database directory in order > to keep a db/table hierarchy. In this case, the renamed table directory is > created under "default' db directory typically set as /hive/warehouse/ . > This error doesn't appear if first create a database which points to a > directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't > have this problem. For example, > create database db_enc location '/hdfs/encrypted_path/db_enc; > use db_enc; > create table rename_test (...) location > '/hdfs/encrypted_path/db_enc/rename_test'; > alter table rename_test rename to test_rename; > The renamed test_rename directory is created under > /hdfs/encrypted_path/db_enc. > Considering that the encryption of a filesystem is part of the evolution > hardening of a system (where the system and the data contained can already > exists) and a db can be already created without location set (because it is > not strictly required)and the default db is outside the same encryption zone > (or in a no-encryption zone) the alter table rename operation will fail. > Improvement: > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15061) Metastore types are sometimes case sensitive
[ https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-15061: --- Attachment: HIVE-15061.1.patch > Metastore types are sometimes case sensitive > > > Key: HIVE-15061 > URL: https://issues.apache.org/jira/browse/HIVE-15061 > Project: Hive > Issue Type: Bug > Components: API >Affects Versions: 1.1.0 >Reporter: Thomas Tauber-Marshall >Assignee: Chaoyu Tang > Attachments: HIVE-15061.1.patch, HIVE-15061.patch > > > Impala recently encountered an issue with the metastore > ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column > stats would get dropped when adding a column to a table. > The reason seems to be that Hive does a case sensitive check on the column > stats types during an "alter table" and expects the types to be all lower > case. This case sensitive check doesn't appear to happen when the stats are > set in the first place. > We're solving this on the Impala end by storing types in the metastore as all > lower case, but Hive's behavior here is very confusing. It should either > always be case sensitive, so that you can't create column stats with types > that Hive considers invalid, or it should never be case sensitive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-15091) Master: Update errata.txt for the missing JIRA number in HIVE-14909 commit msg
[ https://issues.apache.org/jira/browse/HIVE-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang resolved HIVE-15091. Resolution: Fixed Fix Version/s: 2.2.0 > Master: Update errata.txt for the missing JIRA number in HIVE-14909 commit msg > -- > > Key: HIVE-15091 > URL: https://issues.apache.org/jira/browse/HIVE-15091 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.2.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Trivial > Fix For: 2.2.0 > > > Missing the JIRA number in commit msg for master branch, see > https://issues.apache.org/jira/browse/HIVE-14909?focusedCommentId=15614056&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15614056 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14909) Preserve the location of table created with the location clause in table rename
[ https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15615439#comment-15615439 ] Chaoyu Tang commented on HIVE-14909: Oops, my bad. Fixed it in HIVE-15091. Thanks for pointing out. > Preserve the location of table created with the location clause in table > rename > --- > > Key: HIVE-14909 > URL: https://issues.apache.org/jira/browse/HIVE-14909 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.1.0 >Reporter: Adriano >Assignee: Chaoyu Tang > Fix For: 2.2.0 > > Attachments: HIVE-14909.1.patch, HIVE-14909.1.patch, > HIVE-14909.2.patch, HIVE-14909.3.patch, HIVE-14909.patch, HIVE-14909.patch > > > Alter Table operation for db_enc.rename_test failed to move data due to: > '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption > zone.' > When Hive renames a managed table, it always creates the new renamed table > directory under its database directory in order to keep a db/table hierarchy. > In this case, the renamed table directory is created under "default db" > directory "hive/warehouse/". When Hive renames a managed table, it always > creates the new renamed table directory under its database directory in order > to keep a db/table hierarchy. In this case, the renamed table directory is > created under "default' db directory typically set as /hive/warehouse/ . > This error doesn't appear if first create a database which points to a > directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't > have this problem. For example, > create database db_enc location '/hdfs/encrypted_path/db_enc; > use db_enc; > create table rename_test (...) location > '/hdfs/encrypted_path/db_enc/rename_test'; > alter table rename_test rename to test_rename; > The renamed test_rename directory is created under > /hdfs/encrypted_path/db_enc. > Considering that the encryption of a filesystem is part of the evolution > hardening of a system (where the system and the data contained can already > exists) and a db can be already created without location set (because it is > not strictly required)and the default db is outside the same encryption zone > (or in a no-encryption zone) the alter table rename operation will fail. > Improvement: > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15061) Metastore types are sometimes case sensitive
[ https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-15061: --- Attachment: HIVE-15061.1.patch Reattached patch to trigger the precommit test > Metastore types are sometimes case sensitive > > > Key: HIVE-15061 > URL: https://issues.apache.org/jira/browse/HIVE-15061 > Project: Hive > Issue Type: Bug > Components: API >Affects Versions: 1.1.0 >Reporter: Thomas Tauber-Marshall >Assignee: Chaoyu Tang > Attachments: HIVE-15061.1.patch, HIVE-15061.1.patch, HIVE-15061.patch > > > Impala recently encountered an issue with the metastore > ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column > stats would get dropped when adding a column to a table. > The reason seems to be that Hive does a case sensitive check on the column > stats types during an "alter table" and expects the types to be all lower > case. This case sensitive check doesn't appear to happen when the stats are > set in the first place. > We're solving this on the Impala end by storing types in the metastore as all > lower case, but Hive's behavior here is very confusing. It should either > always be case sensitive, so that you can't create column stats with types > that Hive considers invalid, or it should never be case sensitive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14909) Preserve the location of table created with the location clause in table rename
[ https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15615754#comment-15615754 ] Chaoyu Tang commented on HIVE-14909: [~leftylev] I made the change for this change in https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RenameTable, please take a look. Thanks > Preserve the location of table created with the location clause in table > rename > --- > > Key: HIVE-14909 > URL: https://issues.apache.org/jira/browse/HIVE-14909 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.1.0 >Reporter: Adriano >Assignee: Chaoyu Tang > Fix For: 2.2.0 > > Attachments: HIVE-14909.1.patch, HIVE-14909.1.patch, > HIVE-14909.2.patch, HIVE-14909.3.patch, HIVE-14909.patch, HIVE-14909.patch > > > Alter Table operation for db_enc.rename_test failed to move data due to: > '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption > zone.' > When Hive renames a managed table, it always creates the new renamed table > directory under its database directory in order to keep a db/table hierarchy. > In this case, the renamed table directory is created under "default db" > directory "hive/warehouse/". When Hive renames a managed table, it always > creates the new renamed table directory under its database directory in order > to keep a db/table hierarchy. In this case, the renamed table directory is > created under "default' db directory typically set as /hive/warehouse/ . > This error doesn't appear if first create a database which points to a > directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't > have this problem. For example, > create database db_enc location '/hdfs/encrypted_path/db_enc; > use db_enc; > create table rename_test (...) location > '/hdfs/encrypted_path/db_enc/rename_test'; > alter table rename_test rename to test_rename; > The renamed test_rename directory is created under > /hdfs/encrypted_path/db_enc. > Considering that the encryption of a filesystem is part of the evolution > hardening of a system (where the system and the data contained can already > exists) and a db can be already created without location set (because it is > not strictly required)and the default db is outside the same encryption zone > (or in a no-encryption zone) the alter table rename operation will fail. > Improvement: > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14909) Preserve the location of table created with the location clause in table rename
[ https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15615754#comment-15615754 ] Chaoyu Tang edited comment on HIVE-14909 at 10/28/16 3:45 PM: -- [~leftylev] I made the document change for this JIRA in https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RenameTable. Please take a look. Thanks was (Author: ctang.ma): [~leftylev] I made the change for this change in https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RenameTable, please take a look. Thanks > Preserve the location of table created with the location clause in table > rename > --- > > Key: HIVE-14909 > URL: https://issues.apache.org/jira/browse/HIVE-14909 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.1.0 >Reporter: Adriano >Assignee: Chaoyu Tang > Fix For: 2.2.0 > > Attachments: HIVE-14909.1.patch, HIVE-14909.1.patch, > HIVE-14909.2.patch, HIVE-14909.3.patch, HIVE-14909.patch, HIVE-14909.patch > > > Alter Table operation for db_enc.rename_test failed to move data due to: > '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption > zone.' > When Hive renames a managed table, it always creates the new renamed table > directory under its database directory in order to keep a db/table hierarchy. > In this case, the renamed table directory is created under "default db" > directory "hive/warehouse/". When Hive renames a managed table, it always > creates the new renamed table directory under its database directory in order > to keep a db/table hierarchy. In this case, the renamed table directory is > created under "default' db directory typically set as /hive/warehouse/ . > This error doesn't appear if first create a database which points to a > directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't > have this problem. For example, > create database db_enc location '/hdfs/encrypted_path/db_enc; > use db_enc; > create table rename_test (...) location > '/hdfs/encrypted_path/db_enc/rename_test'; > alter table rename_test rename to test_rename; > The renamed test_rename directory is created under > /hdfs/encrypted_path/db_enc. > Considering that the encryption of a filesystem is part of the evolution > hardening of a system (where the system and the data contained can already > exists) and a db can be already created without location set (because it is > not strictly required)and the default db is outside the same encryption zone > (or in a no-encryption zone) the alter table rename operation will fail. > Improvement: > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15061) Metastore types are sometimes case sensitive
[ https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-15061: --- Attachment: HIVE-15061.1.patch > Metastore types are sometimes case sensitive > > > Key: HIVE-15061 > URL: https://issues.apache.org/jira/browse/HIVE-15061 > Project: Hive > Issue Type: Bug > Components: API >Affects Versions: 1.1.0 >Reporter: Thomas Tauber-Marshall >Assignee: Chaoyu Tang > Attachments: HIVE-15061.1.patch, HIVE-15061.1.patch, HIVE-15061.patch > > > Impala recently encountered an issue with the metastore > ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column > stats would get dropped when adding a column to a table. > The reason seems to be that Hive does a case sensitive check on the column > stats types during an "alter table" and expects the types to be all lower > case. This case sensitive check doesn't appear to happen when the stats are > set in the first place. > We're solving this on the Impala end by storing types in the metastore as all > lower case, but Hive's behavior here is very confusing. It should either > always be case sensitive, so that you can't create column stats with types > that Hive considers invalid, or it should never be case sensitive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
