[jira] [Assigned] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman reassigned HIVE-18117: - Assignee: Andrew Sherman (was: Sahil Takiar) > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > > Should be able to do something similar to what we do for HDFS encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18117: -- Attachment: HIVE-18117.1.patch Status: Patch Available (was: Open) > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch > > > Should be able to do something similar to what we do for HDFS encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18117: -- Attachment: (was: HIVE-18117.1.patch) > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > > Should be able to do something similar to what we do for HDFS encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18117: -- Attachment: HIVE-18117.1.patch > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch > > > Should be able to do something similar to what we do for HDFS encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18117: -- Attachment: HIVE-18117.2.patch > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch, HIVE-18117.2.patch > > > Should be able to do something similar to what we do for HDFS encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18117: -- Attachment: HIVE-18117.3.patch > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch, HIVE-18117.2.patch, > HIVE-18117.3.patch > > > Should be able to do something similar to what we do for HDFS encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472807#comment-16472807 ] Andrew Sherman commented on HIVE-18117: --- Updated RB with changes at [https://reviews.apache.org/r/67103/] > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch, HIVE-18117.2.patch, > HIVE-18117.3.patch > > > Should be able to do something similar to what we do for HDFS encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18117: -- Attachment: HIVE-18117.4.patch > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch, HIVE-18117.2.patch, > HIVE-18117.3.patch, HIVE-18117.4.patch > > > Should be able to do something similar to what we do for HDFS encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18117: -- Attachment: HIVE-18117.5.patch > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch, HIVE-18117.2.patch, > HIVE-18117.3.patch, HIVE-18117.4.patch, HIVE-18117.5.patch > > > Should be able to do something similar to what we do for HDFS encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18117: -- Attachment: HIVE-18117.6.patch > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch, HIVE-18117.2.patch, > HIVE-18117.3.patch, HIVE-18117.4.patch, HIVE-18117.5.patch, HIVE-18117.6.patch > > > Should be able to do something similar to what we do for HDFS encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18117: -- Attachment: HIVE-18117.7.patch > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch, HIVE-18117.2.patch, > HIVE-18117.3.patch, HIVE-18117.4.patch, HIVE-18117.5.patch, > HIVE-18117.6.patch, HIVE-18117.7.patch > > > Should be able to do something similar to what we do for HDFS encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18117: -- Attachment: HIVE-18117.8.patch > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch, HIVE-18117.2.patch, > HIVE-18117.3.patch, HIVE-18117.4.patch, HIVE-18117.5.patch, > HIVE-18117.6.patch, HIVE-18117.7.patch, HIVE-18117.8.patch > > > Should be able to do something similar to what we do for HDFS encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18117: -- Attachment: HIVE-18117.9.patch > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch, HIVE-18117.2.patch, > HIVE-18117.3.patch, HIVE-18117.4.patch, HIVE-18117.5.patch, > HIVE-18117.6.patch, HIVE-18117.7.patch, HIVE-18117.8.patch, HIVE-18117.9.patch > > > Should be able to do something similar to what we do for HDFS encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16485744#comment-16485744 ] Andrew Sherman commented on HIVE-18117: --- The 3 asf license warnings in [http://104.198.109.242/logs//PreCommit-HIVE-Build-3/yetus/patch-asflicense-problems.txt] are not caused by this change. The checkstyle warning TestErasureCodingHDFSCliDriver.java:48: public static TestRule cliClassRule = adapter.buildClassRule();:26: warning: Variable 'cliClassRule' must be private and have accessor methodsI Is a false positive as Junit requires that fields annotated with ClassRule must be public. The single most recent failure [https://builds.apache.org/job/PreCommit-HIVE-Build/3/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_sample6_/] is spurious. So I think this change is good to go, please take a look [~stakiar] > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch, HIVE-18117.2.patch, > HIVE-18117.3.patch, HIVE-18117.4.patch, HIVE-18117.5.patch, > HIVE-18117.6.patch, HIVE-18117.7.patch, HIVE-18117.8.patch, HIVE-18117.9.patch > > > Should be able to do something similar to what we do for HDFS encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487615#comment-16487615 ] Andrew Sherman commented on HIVE-18117: --- Thanks [~stakiar] > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch, HIVE-18117.2.patch, > HIVE-18117.3.patch, HIVE-18117.4.patch, HIVE-18117.5.patch, > HIVE-18117.6.patch, HIVE-18117.7.patch, HIVE-18117.8.patch, HIVE-18117.9.patch > > > Should be able to do something similar to what we do for HDFS encryption. > TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor > "ErasureProcessor" > which allows .q files to contain Erasure Coding commands similar to those > provided > by the hdfs ec command > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html. > The Erasure Coding functionality is exposed through a new shim > "HdfsFileErasureCodingPolicy". > At this stage there are two .q files: > erasure_commnds.q (a simple test to show ERASURE commands can run on local fs > via > TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and > erasure_simple.q (which does some trivial queries to demonstrate basic > functionality). > More tests will come in future commits. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18117: -- Description: Should be able to do something similar to what we do for HDFS encryption. TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor "ErasureProcessor" which allows .q files to contain Erasure Coding commands similar to those provided by the hdfs ec command https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html. The Erasure Coding functionality is exposed through a new shim "HdfsFileErasureCodingPolicy". At this stage there are two .q files: erasure_commnds.q (a simple test to show ERASURE commands can run on local fs via TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and erasure_simple.q (which does some trivial queries to demonstrate basic functionality). More tests will come in future commits. was:Should be able to do something similar to what we do for HDFS encryption. > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch, HIVE-18117.2.patch, > HIVE-18117.3.patch, HIVE-18117.4.patch, HIVE-18117.5.patch, > HIVE-18117.6.patch, HIVE-18117.7.patch, HIVE-18117.8.patch, HIVE-18117.9.patch > > > Should be able to do something similar to what we do for HDFS encryption. > TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor > "ErasureProcessor" > which allows .q files to contain Erasure Coding commands similar to those > provided > by the hdfs ec command > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html. > The Erasure Coding functionality is exposed through a new shim > "HdfsFileErasureCodingPolicy". > At this stage there are two .q files: > erasure_commnds.q (a simple test to show ERASURE commands can run on local fs > via > TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and > erasure_simple.q (which does some trivial queries to demonstrate basic > functionality). > More tests will come in future commits. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487667#comment-16487667 ] Andrew Sherman commented on HIVE-18117: --- Flaky test will be disabled by HIVE-19677 but I will go for a clean run > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch, HIVE-18117.2.patch, > HIVE-18117.3.patch, HIVE-18117.4.patch, HIVE-18117.5.patch, > HIVE-18117.6.patch, HIVE-18117.7.patch, HIVE-18117.8.patch, HIVE-18117.9.patch > > > Should be able to do something similar to what we do for HDFS encryption. > TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor > "ErasureProcessor" > which allows .q files to contain Erasure Coding commands similar to those > provided > by the hdfs ec command > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html. > The Erasure Coding functionality is exposed through a new shim > "HdfsFileErasureCodingPolicy". > At this stage there are two .q files: > erasure_commnds.q (a simple test to show ERASURE commands can run on local fs > via > TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and > erasure_simple.q (which does some trivial queries to demonstrate basic > functionality). > More tests will come in future commits. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18117: -- Attachment: HIVE-18117.10.patch > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch, HIVE-18117.10.patch, > HIVE-18117.2.patch, HIVE-18117.3.patch, HIVE-18117.4.patch, > HIVE-18117.5.patch, HIVE-18117.6.patch, HIVE-18117.7.patch, > HIVE-18117.8.patch, HIVE-18117.9.patch > > > Should be able to do something similar to what we do for HDFS encryption. > TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor > "ErasureProcessor" > which allows .q files to contain Erasure Coding commands similar to those > provided > by the hdfs ec command > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html. > The Erasure Coding functionality is exposed through a new shim > "HdfsFileErasureCodingPolicy". > At this stage there are two .q files: > erasure_commnds.q (a simple test to show ERASURE commands can run on local fs > via > TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and > erasure_simple.q (which does some trivial queries to demonstrate basic > functionality). > More tests will come in future commits. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19693) Create hive API on Java 1.9 based
[ https://issues.apache.org/jira/browse/HIVE-19693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489217#comment-16489217 ] Andrew Sherman commented on HIVE-19693: --- Can you explain more about what this means? Thanks > Create hive API on Java 1.9 based > - > > Key: HIVE-19693 > URL: https://issues.apache.org/jira/browse/HIVE-19693 > Project: Hive > Issue Type: Improvement > Components: API >Affects Versions: 2.3.2 >Reporter: Murtaza Hatim Zaveri >Assignee: Murtaza Hatim Zaveri >Priority: Major > Labels: newbie > Fix For: 0.10.1 > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16491140#comment-16491140 ] Andrew Sherman commented on HIVE-18117: --- OK tests look clean now please take a look [~stakiar] when you have time > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch, HIVE-18117.10.patch, > HIVE-18117.2.patch, HIVE-18117.3.patch, HIVE-18117.4.patch, > HIVE-18117.5.patch, HIVE-18117.6.patch, HIVE-18117.7.patch, > HIVE-18117.8.patch, HIVE-18117.9.patch > > > Should be able to do something similar to what we do for HDFS encryption. > TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor > "ErasureProcessor" > which allows .q files to contain Erasure Coding commands similar to those > provided > by the hdfs ec command > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html. > The Erasure Coding functionality is exposed through a new shim > "HdfsFileErasureCodingPolicy". > At this stage there are two .q files: > erasure_commnds.q (a simple test to show ERASURE commands can run on local fs > via > TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and > erasure_simple.q (which does some trivial queries to demonstrate basic > functionality). > More tests will come in future commits. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18940) Hive notifications serialize all write DDL operations
[
https://issues.apache.org/jira/browse/HIVE-18940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405155#comment-16405155
]
Andrew Sherman commented on HIVE-18940:
---
The restriction that the EVENT_ID has to be in the order of commit is
presumably having a major impact on concurrency in the HMS DBMS. It effectively
serializes avery transaction. Has there been any thought to relaxing this
restriction?
> Hive notifications serialize all write DDL operations
> -
>
> Key: HIVE-18940
> URL: https://issues.apache.org/jira/browse/HIVE-18940
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Priority: Major
>
> The implementation of DbNotificationListener uses a single row to store
> current notification ID and uses {{SELECT FOR UPDATE}} to lock the row. This
> serializes all write DDL operations which isn't good.
> We should consider using database auto-increment for notification ID instead.
> Especially on mMySQL/innoDb it is supported natively with relatively
> light-weight locking.
> This creates potential issue for consumers though because such IDs may have
> holes. There are two types of holes - transient hole for a transaction which
> have not committed yet and will be committed shortly and permanent holes for
> transactions that fail. Consumers need to deal with it. It may be useful to
> add DB-generated timestamp as well to assist in recovery from holes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19005) Maximum Session Per User
[ https://issues.apache.org/jira/browse/HIVE-19005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408440#comment-16408440 ] Andrew Sherman commented on HIVE-19005: --- Hi [~belugabehr] does [HIVE-16917] cover what you want? > Maximum Session Per User > > > Key: HIVE-19005 > URL: https://issues.apache.org/jira/browse/HIVE-19005 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Affects Versions: 3.0.0, 2.4.0, 2.3.2 >Reporter: BELUGA BEHR >Priority: Minor > > # Set a maximum number of connections/sessions per user (keeping in mind > proxy users) > # Set a maximum number of sessions globally > Would need to track active sessions in ZK or DB. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing
[ https://issues.apache.org/jira/browse/HIVE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18877: -- Attachment: HIVE-18877.5.patch > HiveSchemaTool.validateSchemaTables() should wrap a SQLException when > rethrowing > > > Key: HIVE-18877 > URL: https://issues.apache.org/jira/browse/HIVE-18877 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18877.1.patch, HIVE-18877.2.patch, > HIVE-18877.3.patch, HIVE-18877.4.patch, HIVE-18877.5.patch > > > If schematool is run with the -verbose flag then it will print a stack trace > for an exception that occurs. If a SQLException is caught during > HiveSchemaTool.validateSchemaTables() then a HiveMetaException is rethrown > containing the text of the SQLException. If we instead throw a > HiveMetaException that wraps the SQLException then the stacktrace will help > with diagnosis of issues where the SQLException contains a generic error > text. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19005) Maximum Session Per User
[ https://issues.apache.org/jira/browse/HIVE-19005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409757#comment-16409757 ] Andrew Sherman commented on HIVE-19005: --- [~belugabehr] [HIVE-16917] says that it provides * hive.server2.limit.connections.per.user * hive.server2.limit.connections.per.ipaddress * hive.server2.limit.connections.per.user.ipaddress so maybe that is enough? > Maximum Session Per User > > > Key: HIVE-19005 > URL: https://issues.apache.org/jira/browse/HIVE-19005 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Affects Versions: 3.0.0, 2.4.0, 2.3.2 >Reporter: BELUGA BEHR >Priority: Minor > > # Set a maximum number of connections/sessions per user (keeping in mind > proxy users) > # Set a maximum number of sessions globally > Would need to track active sessions in ZK or DB. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing
[ https://issues.apache.org/jira/browse/HIVE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18877: -- Attachment: HIVE-18877.6.patch > HiveSchemaTool.validateSchemaTables() should wrap a SQLException when > rethrowing > > > Key: HIVE-18877 > URL: https://issues.apache.org/jira/browse/HIVE-18877 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18877.1.patch, HIVE-18877.2.patch, > HIVE-18877.3.patch, HIVE-18877.4.patch, HIVE-18877.5.patch, HIVE-18877.6.patch > > > If schematool is run with the -verbose flag then it will print a stack trace > for an exception that occurs. If a SQLException is caught during > HiveSchemaTool.validateSchemaTables() then a HiveMetaException is rethrown > containing the text of the SQLException. If we instead throw a > HiveMetaException that wraps the SQLException then the stacktrace will help > with diagnosis of issues where the SQLException contains a generic error > text. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing
[ https://issues.apache.org/jira/browse/HIVE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16416172#comment-16416172 ] Andrew Sherman commented on HIVE-18877: --- I assert that the multiple test failures have nothing to do with this change. [~vihangk1] please can you review when you have time. > HiveSchemaTool.validateSchemaTables() should wrap a SQLException when > rethrowing > > > Key: HIVE-18877 > URL: https://issues.apache.org/jira/browse/HIVE-18877 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18877.1.patch, HIVE-18877.2.patch, > HIVE-18877.3.patch, HIVE-18877.4.patch, HIVE-18877.5.patch, HIVE-18877.6.patch > > > If schematool is run with the -verbose flag then it will print a stack trace > for an exception that occurs. If a SQLException is caught during > HiveSchemaTool.validateSchemaTables() then a HiveMetaException is rethrown > containing the text of the SQLException. If we instead throw a > HiveMetaException that wraps the SQLException then the stacktrace will help > with diagnosis of issues where the SQLException contains a generic error > text. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19062) Update constraint_partition_columns.q.out
[ https://issues.apache.org/jira/browse/HIVE-19062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman reassigned HIVE-19062: - > Update constraint_partition_columns.q.out > - > > Key: HIVE-19062 > URL: https://issues.apache.org/jira/browse/HIVE-19062 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > > TestNegativeCliDriver is error-prone at present but if you run > constraint_partition_columns.q on its own you get a diff. I think this is a > simple regression caused by [HIVE-18726] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19062) Update constraint_partition_columns.q.out
[ https://issues.apache.org/jira/browse/HIVE-19062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-19062: -- Attachment: HIVE-19062.1.patch > Update constraint_partition_columns.q.out > - > > Key: HIVE-19062 > URL: https://issues.apache.org/jira/browse/HIVE-19062 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-19062.1.patch > > > TestNegativeCliDriver is error-prone at present but if you run > constraint_partition_columns.q on its own you get a diff. I think this is a > simple regression caused by [HIVE-18726] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19062) Update constraint_partition_columns.q.out
[ https://issues.apache.org/jira/browse/HIVE-19062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-19062: -- Status: Patch Available (was: Open) > Update constraint_partition_columns.q.out > - > > Key: HIVE-19062 > URL: https://issues.apache.org/jira/browse/HIVE-19062 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-19062.1.patch > > > TestNegativeCliDriver is error-prone at present but if you run > constraint_partition_columns.q on its own you get a diff. I think this is a > simple regression caused by [HIVE-18726] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19062) Update constraint_partition_columns.q.out
[ https://issues.apache.org/jira/browse/HIVE-19062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419824#comment-16419824 ] Andrew Sherman commented on HIVE-19062: --- [~vgarg] please review and push if you agree this is a simple regression caused by [HIVE-18726] > Update constraint_partition_columns.q.out > - > > Key: HIVE-19062 > URL: https://issues.apache.org/jira/browse/HIVE-19062 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-19062.1.patch > > > TestNegativeCliDriver is error-prone at present but if you run > constraint_partition_columns.q on its own you get a diff. I think this is a > simple regression caused by [HIVE-18726] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing
[ https://issues.apache.org/jira/browse/HIVE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424271#comment-16424271 ] Andrew Sherman commented on HIVE-18877: --- [~vihangk1] can you push this at your convenience please? > HiveSchemaTool.validateSchemaTables() should wrap a SQLException when > rethrowing > > > Key: HIVE-18877 > URL: https://issues.apache.org/jira/browse/HIVE-18877 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18877.1.patch, HIVE-18877.2.patch, > HIVE-18877.3.patch, HIVE-18877.4.patch, HIVE-18877.5.patch, HIVE-18877.6.patch > > > If schematool is run with the -verbose flag then it will print a stack trace > for an exception that occurs. If a SQLException is caught during > HiveSchemaTool.validateSchemaTables() then a HiveMetaException is rethrown > containing the text of the SQLException. If we instead throw a > HiveMetaException that wraps the SQLException then the stacktrace will help > with diagnosis of issues where the SQLException contains a generic error > text. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing
[ https://issues.apache.org/jira/browse/HIVE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424398#comment-16424398 ] Andrew Sherman commented on HIVE-18877: --- Thanks [~vihangk1] > HiveSchemaTool.validateSchemaTables() should wrap a SQLException when > rethrowing > > > Key: HIVE-18877 > URL: https://issues.apache.org/jira/browse/HIVE-18877 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Fix For: 3.0.0, 2.4.0 > > Attachments: HIVE-18877.1.patch, HIVE-18877.2.patch, > HIVE-18877.3.patch, HIVE-18877.4.patch, HIVE-18877.5.patch, HIVE-18877.6.patch > > > If schematool is run with the -verbose flag then it will print a stack trace > for an exception that occurs. If a SQLException is caught during > HiveSchemaTool.validateSchemaTables() then a HiveMetaException is rethrown > containing the text of the SQLException. If we instead throw a > HiveMetaException that wraps the SQLException then the stacktrace will help > with diagnosis of issues where the SQLException contains a generic error > text. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18712) Design HMS Api v2
[ https://issues.apache.org/jira/browse/HIVE-18712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16444260#comment-16444260 ] Andrew Sherman commented on HIVE-18712: --- +1 for server side filtering everywhere > Design HMS Api v2 > - > > Key: HIVE-18712 > URL: https://issues.apache.org/jira/browse/HIVE-18712 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: metastore.proto, metastore_index.html, > metastore_swagger.json, metastore_swagger.yaml > > > This is an umbrella Jira covering the design of Hive Metastore API v2. > It is supposed to be a placeholder for discussion and design documents. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-18625:
-
Assignee: Andrew Sherman
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18625:
--
Attachment: HIVE-18625.1.patch
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18625.1.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18625:
--
Status: Patch Available (was: Open)
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 2.3.2, 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18625.1.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18456) Add some tests for HIVE-18367 to check that the table information contains the query correctly
[ https://issues.apache.org/jira/browse/HIVE-18456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18456: -- Status: Patch Available (was: Open) > Add some tests for HIVE-18367 to check that the table information contains > the query correctly > -- > > Key: HIVE-18456 > URL: https://issues.apache.org/jira/browse/HIVE-18456 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18456.1.patch > > > This cannot be tested with a CliDriver test so add a java test to check the > output of 'describe extended', which is changed by HIVE-18367 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18456) Add some tests for HIVE-18367 to check that the table information contains the query correctly
[ https://issues.apache.org/jira/browse/HIVE-18456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18456: -- Attachment: HIVE-18456.1.patch > Add some tests for HIVE-18367 to check that the table information contains > the query correctly > -- > > Key: HIVE-18456 > URL: https://issues.apache.org/jira/browse/HIVE-18456 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18456.1.patch > > > This cannot be tested with a CliDriver test so add a java test to check the > output of 'describe extended', which is changed by HIVE-18367 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356105#comment-16356105
]
Andrew Sherman commented on HIVE-18625:
---
{color:#33}Test failures appear unrelated.{color}
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18625.1.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18456) Add some tests for HIVE-18367 to check that the table information contains the query correctly
[ https://issues.apache.org/jira/browse/HIVE-18456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18456: -- Attachment: HIVE-18456.2.patch > Add some tests for HIVE-18367 to check that the table information contains > the query correctly > -- > > Key: HIVE-18456 > URL: https://issues.apache.org/jira/browse/HIVE-18456 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18456.1.patch, HIVE-18456.2.patch > > > This cannot be tested with a CliDriver test so add a java test to check the > output of 'describe extended', which is changed by HIVE-18367 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18625:
--
Attachment: HIVE-18625.2.patch
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18625.1.patch, HIVE-18625.2.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18625:
--
Attachment: (was: HIVE-18625.2.patch)
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18625.1.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18625:
--
Attachment: HIVE-18625.2.patch
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18625.1.patch, HIVE-18625.2.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358886#comment-16358886
]
Andrew Sherman commented on HIVE-18625:
---
Made changes, ran tests, and failures look unrelated. Please can you take
another look [~stakiar]
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18625.1.patch, HIVE-18625.2.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18456) Add some tests for HIVE-18367 to check that the table information contains the query correctly
[ https://issues.apache.org/jira/browse/HIVE-18456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358893#comment-16358893 ] Andrew Sherman commented on HIVE-18456: --- Test failures look unrelated (not surprisingly as this is a test-only change). Please can you take a look [~pvary] > Add some tests for HIVE-18367 to check that the table information contains > the query correctly > -- > > Key: HIVE-18456 > URL: https://issues.apache.org/jira/browse/HIVE-18456 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18456.1.patch, HIVE-18456.2.patch > > > This cannot be tested with a CliDriver test so add a java test to check the > output of 'describe extended', which is changed by HIVE-18367 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18456) Add some tests for HIVE-18367 to check that the table information contains the query correctly
[ https://issues.apache.org/jira/browse/HIVE-18456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18456: -- Attachment: HIVE-18456.3.patch > Add some tests for HIVE-18367 to check that the table information contains > the query correctly > -- > > Key: HIVE-18456 > URL: https://issues.apache.org/jira/browse/HIVE-18456 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18456.1.patch, HIVE-18456.2.patch, > HIVE-18456.3.patch > > > This cannot be tested with a CliDriver test so add a java test to check the > output of 'describe extended', which is changed by HIVE-18367 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18456) Add some tests for HIVE-18367 to check that the table information contains the query correctly
[ https://issues.apache.org/jira/browse/HIVE-18456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361662#comment-16361662 ] Andrew Sherman commented on HIVE-18456: --- Tidied up per [~pvary] suggestions. Test failures and ASF License warnings are not caused by this change. > Add some tests for HIVE-18367 to check that the table information contains > the query correctly > -- > > Key: HIVE-18456 > URL: https://issues.apache.org/jira/browse/HIVE-18456 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18456.1.patch, HIVE-18456.2.patch, > HIVE-18456.3.patch > > > This cannot be tested with a CliDriver test so add a java test to check the > output of 'describe extended', which is changed by HIVE-18367 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18456) Add some tests for HIVE-18367 to check that the table information contains the query correctly
[ https://issues.apache.org/jira/browse/HIVE-18456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370381#comment-16370381 ] Andrew Sherman commented on HIVE-18456: --- [~pvary] is there any more review needed? Otherwise please can you push this to master? > Add some tests for HIVE-18367 to check that the table information contains > the query correctly > -- > > Key: HIVE-18456 > URL: https://issues.apache.org/jira/browse/HIVE-18456 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18456.1.patch, HIVE-18456.2.patch, > HIVE-18456.3.patch > > > This cannot be tested with a CliDriver test so add a java test to check the > output of 'describe extended', which is changed by HIVE-18367 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18456) Add some tests for HIVE-18367 to check that the table information contains the query correctly
[ https://issues.apache.org/jira/browse/HIVE-18456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371614#comment-16371614 ] Andrew Sherman commented on HIVE-18456: --- Thanls [~pvary] > Add some tests for HIVE-18367 to check that the table information contains > the query correctly > -- > > Key: HIVE-18456 > URL: https://issues.apache.org/jira/browse/HIVE-18456 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18456.1.patch, HIVE-18456.2.patch, > HIVE-18456.3.patch > > > This cannot be tested with a CliDriver test so add a java test to check the > output of 'describe extended', which is changed by HIVE-18367 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371616#comment-16371616
]
Andrew Sherman commented on HIVE-18625:
---
Thanks [~stakiar]
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-18625.1.patch, HIVE-18625.2.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16374724#comment-16374724
]
Andrew Sherman commented on HIVE-18625:
---
Thanks, I can reproduce and I'll take a look later this (California) morning.
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-18625.1.patch, HIVE-18625.2.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18791) Fix TestJdbcWithMiniHS2#testHttpHeaderSize
[ https://issues.apache.org/jira/browse/HIVE-18791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman reassigned HIVE-18791: - > Fix TestJdbcWithMiniHS2#testHttpHeaderSize > -- > > Key: HIVE-18791 > URL: https://issues.apache.org/jira/browse/HIVE-18791 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > > TestJdbcWithMiniHS2#testHttpHeaderSize tests whether config of http header > sizes works by using a long username. The local scratch directory for the > session uses the username as part of its path. When this name is more than > 255 chars (on most modern file systems) then the directory creation will > fail. HIVE-18625 made this failure throw an exception, which has caused a > regression in testHttpHeaderSize. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16375103#comment-16375103
]
Andrew Sherman commented on HIVE-18625:
---
I created HIVE-18791 to track the regression I have created.
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-18625.1.patch, HIVE-18625.2.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18791) Fix TestJdbcWithMiniHS2#testHttpHeaderSize
[ https://issues.apache.org/jira/browse/HIVE-18791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384218#comment-16384218 ] Andrew Sherman commented on HIVE-18791: --- The test TestJdbcWithMiniHS2.testHttpHeaderSize() is testing that HS2 can be reconfigured to use different maximum sizes of http headers when Thrift uses http as its transport. In this mode the http header contains a user name and password encoded using http basic encoding. The original test varies the length of the user name to test (1) that that a too large header fails with 413 Payload Too Large, and (2) that increasing the maximum sizes of http headers allows the large header to work. In HS2 when the connection is made a Session is created (note that in this configuration the user name and password are not verified). The local scratch directory for the session uses the username as part of its path. When this name is more than 255 chars (on most modern file systems) then the directory creation will fail. HIVE-18625 made this failure newly throw an exception, which caused a regression in testHttpHeaderSize() when the session was created with a long user name. Fix this by having the test vary the password rather then the user name. The test is functionally identical but does not hit the directory creation exception. > Fix TestJdbcWithMiniHS2#testHttpHeaderSize > -- > > Key: HIVE-18791 > URL: https://issues.apache.org/jira/browse/HIVE-18791 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > > TestJdbcWithMiniHS2#testHttpHeaderSize tests whether config of http header > sizes works by using a long username. The local scratch directory for the > session uses the username as part of its path. When this name is more than > 255 chars (on most modern file systems) then the directory creation will > fail. HIVE-18625 made this failure throw an exception, which has caused a > regression in testHttpHeaderSize. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18791) Fix TestJdbcWithMiniHS2#testHttpHeaderSize
[ https://issues.apache.org/jira/browse/HIVE-18791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18791: -- Attachment: HIVE-18791.1.patch > Fix TestJdbcWithMiniHS2#testHttpHeaderSize > -- > > Key: HIVE-18791 > URL: https://issues.apache.org/jira/browse/HIVE-18791 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18791.1.patch > > > TestJdbcWithMiniHS2#testHttpHeaderSize tests whether config of http header > sizes works by using a long username. The local scratch directory for the > session uses the username as part of its path. When this name is more than > 255 chars (on most modern file systems) then the directory creation will > fail. HIVE-18625 made this failure throw an exception, which has caused a > regression in testHttpHeaderSize. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18791) Fix TestJdbcWithMiniHS2#testHttpHeaderSize
[ https://issues.apache.org/jira/browse/HIVE-18791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18791: -- Status: Patch Available (was: Open) > Fix TestJdbcWithMiniHS2#testHttpHeaderSize > -- > > Key: HIVE-18791 > URL: https://issues.apache.org/jira/browse/HIVE-18791 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18791.1.patch > > > TestJdbcWithMiniHS2#testHttpHeaderSize tests whether config of http header > sizes works by using a long username. The local scratch directory for the > session uses the username as part of its path. When this name is more than > 255 chars (on most modern file systems) then the directory creation will > fail. HIVE-18625 made this failure throw an exception, which has caused a > regression in testHttpHeaderSize. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18791) Fix TestJdbcWithMiniHS2#testHttpHeaderSize
[ https://issues.apache.org/jira/browse/HIVE-18791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386369#comment-16386369 ] Andrew Sherman commented on HIVE-18791: --- Thanks for looking [~pvary]. Test failures look unrelated (and this is a test-only change) > Fix TestJdbcWithMiniHS2#testHttpHeaderSize > -- > > Key: HIVE-18791 > URL: https://issues.apache.org/jira/browse/HIVE-18791 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18791.1.patch > > > TestJdbcWithMiniHS2#testHttpHeaderSize tests whether config of http header > sizes works by using a long username. The local scratch directory for the > session uses the username as part of its path. When this name is more than > 255 chars (on most modern file systems) then the directory creation will > fail. HIVE-18625 made this failure throw an exception, which has caused a > regression in testHttpHeaderSize. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing
[ https://issues.apache.org/jira/browse/HIVE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman reassigned HIVE-18877: - > HiveSchemaTool.validateSchemaTables() should wrap a SQLException when > rethrowing > > > Key: HIVE-18877 > URL: https://issues.apache.org/jira/browse/HIVE-18877 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > > If schematool is run with the -verbose flag then it will print a stack trace > for an exception that occurs. If a SQLException is caught during > HiveSchemaTool.validateSchemaTables() then a HiveMetaException is rethrown > containing the text of the SQLException. If we instead throw a > HiveMetaException that wraps the SQLException then the stacktrace will help > with diagnosis of issues where the SQLException contains a generic error > text. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing
[ https://issues.apache.org/jira/browse/HIVE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18877: -- Attachment: HIVE-18877.1.patch > HiveSchemaTool.validateSchemaTables() should wrap a SQLException when > rethrowing > > > Key: HIVE-18877 > URL: https://issues.apache.org/jira/browse/HIVE-18877 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18877.1.patch > > > If schematool is run with the -verbose flag then it will print a stack trace > for an exception that occurs. If a SQLException is caught during > HiveSchemaTool.validateSchemaTables() then a HiveMetaException is rethrown > containing the text of the SQLException. If we instead throw a > HiveMetaException that wraps the SQLException then the stacktrace will help > with diagnosis of issues where the SQLException contains a generic error > text. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing
[ https://issues.apache.org/jira/browse/HIVE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18877: -- Status: Patch Available (was: Open) > HiveSchemaTool.validateSchemaTables() should wrap a SQLException when > rethrowing > > > Key: HIVE-18877 > URL: https://issues.apache.org/jira/browse/HIVE-18877 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18877.1.patch > > > If schematool is run with the -verbose flag then it will print a stack trace > for an exception that occurs. If a SQLException is caught during > HiveSchemaTool.validateSchemaTables() then a HiveMetaException is rethrown > containing the text of the SQLException. If we instead throw a > HiveMetaException that wraps the SQLException then the stacktrace will help > with diagnosis of issues where the SQLException contains a generic error > text. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18884) Simplify Logging in Hive Metastore Client
[
https://issues.apache.org/jira/browse/HIVE-18884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-18884:
-
Assignee: Andrew Sherman
> Simplify Logging in Hive Metastore Client
> -
>
> Key: HIVE-18884
> URL: https://issues.apache.org/jira/browse/HIVE-18884
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Labels: noob
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
> The current logging is:
> {code}
> 2018-02-26 07:02:44,883 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Trying to connect to metastore with URI
> thrift://host.company.com:9083
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Connected to metastore.
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Opened a connection to metastore, current connections: 2
> {code}
> Please simplify to something like:
> {code}
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Opened a connection to metastore (URI
> thrift://host.company.com:9083), current connections: 2
> ... or ...
> 2018-02-26 07:02:44,892 ERROR hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Failed to connect to the Metastore Sserver (URI
> thrift://host.company.com:9083)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing
[ https://issues.apache.org/jira/browse/HIVE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18877: -- Attachment: HIVE-18877.2.patch > HiveSchemaTool.validateSchemaTables() should wrap a SQLException when > rethrowing > > > Key: HIVE-18877 > URL: https://issues.apache.org/jira/browse/HIVE-18877 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18877.1.patch, HIVE-18877.2.patch > > > If schematool is run with the -verbose flag then it will print a stack trace > for an exception that occurs. If a SQLException is caught during > HiveSchemaTool.validateSchemaTables() then a HiveMetaException is rethrown > containing the text of the SQLException. If we instead throw a > HiveMetaException that wraps the SQLException then the stacktrace will help > with diagnosis of issues where the SQLException contains a generic error > text. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18791) Fix TestJdbcWithMiniHS2#testHttpHeaderSize
[ https://issues.apache.org/jira/browse/HIVE-18791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390043#comment-16390043 ] Andrew Sherman commented on HIVE-18791: --- [~pvary] please can you push this change when you have time? > Fix TestJdbcWithMiniHS2#testHttpHeaderSize > -- > > Key: HIVE-18791 > URL: https://issues.apache.org/jira/browse/HIVE-18791 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18791.1.patch > > > TestJdbcWithMiniHS2#testHttpHeaderSize tests whether config of http header > sizes works by using a long username. The local scratch directory for the > session uses the username as part of its path. When this name is more than > 255 chars (on most modern file systems) then the directory creation will > fail. HIVE-18625 made this failure throw an exception, which has caused a > regression in testHttpHeaderSize. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18791) Fix TestJdbcWithMiniHS2#testHttpHeaderSize
[ https://issues.apache.org/jira/browse/HIVE-18791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390540#comment-16390540 ] Andrew Sherman commented on HIVE-18791: --- Thanks [~pvary] > Fix TestJdbcWithMiniHS2#testHttpHeaderSize > -- > > Key: HIVE-18791 > URL: https://issues.apache.org/jira/browse/HIVE-18791 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18791.1.patch > > > TestJdbcWithMiniHS2#testHttpHeaderSize tests whether config of http header > sizes works by using a long username. The local scratch directory for the > session uses the username as part of its path. When this name is more than > 255 chars (on most modern file systems) then the directory creation will > fail. HIVE-18625 made this failure throw an exception, which has caused a > regression in testHttpHeaderSize. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing
[ https://issues.apache.org/jira/browse/HIVE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18877: -- Attachment: HIVE-18877.3.patch > HiveSchemaTool.validateSchemaTables() should wrap a SQLException when > rethrowing > > > Key: HIVE-18877 > URL: https://issues.apache.org/jira/browse/HIVE-18877 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18877.1.patch, HIVE-18877.2.patch, > HIVE-18877.3.patch > > > If schematool is run with the -verbose flag then it will print a stack trace > for an exception that occurs. If a SQLException is caught during > HiveSchemaTool.validateSchemaTables() then a HiveMetaException is rethrown > containing the text of the SQLException. If we instead throw a > HiveMetaException that wraps the SQLException then the stacktrace will help > with diagnosis of issues where the SQLException contains a generic error > text. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing
[ https://issues.apache.org/jira/browse/HIVE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18877: -- Attachment: HIVE-18877.4.patch > HiveSchemaTool.validateSchemaTables() should wrap a SQLException when > rethrowing > > > Key: HIVE-18877 > URL: https://issues.apache.org/jira/browse/HIVE-18877 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18877.1.patch, HIVE-18877.2.patch, > HIVE-18877.3.patch, HIVE-18877.4.patch > > > If schematool is run with the -verbose flag then it will print a stack trace > for an exception that occurs. If a SQLException is caught during > HiveSchemaTool.validateSchemaTables() then a HiveMetaException is rethrown > containing the text of the SQLException. If we instead throw a > HiveMetaException that wraps the SQLException then the stacktrace will help > with diagnosis of issues where the SQLException contains a generic error > text. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20384) Fix flakiness of erasure_commands.q
[ https://issues.apache.org/jira/browse/HIVE-20384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580369#comment-16580369 ] Andrew Sherman commented on HIVE-20384: --- LGTM +1 pending test results > Fix flakiness of erasure_commands.q > --- > > Key: HIVE-20384 > URL: https://issues.apache.org/jira/browse/HIVE-20384 > Project: Hive > Issue Type: Bug >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-20384.0.patch > > > Qtest erasure_commands.q might fail if erasure_simple.q precedes it in the > same batch -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20479) Update content/people.mdtext in cms
[
https://issues.apache.org/jira/browse/HIVE-20479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-20479:
-
> Update content/people.mdtext in cms
>
>
> Key: HIVE-20479
> URL: https://issues.apache.org/jira/browse/HIVE-20479
> Project: Hive
> Issue Type: Task
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>Priority: Major
>
> I added myself to the committers list.
>
> {code:java}
> asherman
> Andrew Sherman
> href="http://cloudera.com/";>Cloudera
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-20479) Update content/people.mdtext in cms
[
https://issues.apache.org/jira/browse/HIVE-20479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman resolved HIVE-20479.
---
Resolution: Fixed
Done already
> Update content/people.mdtext in cms
>
>
> Key: HIVE-20479
> URL: https://issues.apache.org/jira/browse/HIVE-20479
> Project: Hive
> Issue Type: Task
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>Priority: Major
>
> I added myself to the committers list.
>
> {code:java}
> asherman
> Andrew Sherman
> href="http://cloudera.com/";>Cloudera
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20437) Handle schema evolution from float, double and decimal
[ https://issues.apache.org/jira/browse/HIVE-20437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606254#comment-16606254 ] Andrew Sherman commented on HIVE-20437: --- +1 LGTM > Handle schema evolution from float, double and decimal > -- > > Key: HIVE-20437 > URL: https://issues.apache.org/jira/browse/HIVE-20437 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-20437.1.patch, HIVE-20437.2.patch, > HIVE-20437.3.patch > > > When data created as float, double or decimal in parquet format is read back > using some other type, errors are seen. Parquet should behave just like any > other format. If the value is valid for the new type, data is retuned > otherwise null has to be returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20437) Handle schema evolution from float, double and decimal
[ https://issues.apache.org/jira/browse/HIVE-20437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-20437: -- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks [~janulatha] for the patch > Handle schema evolution from float, double and decimal > -- > > Key: HIVE-20437 > URL: https://issues.apache.org/jira/browse/HIVE-20437 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-20437.1.patch, HIVE-20437.2.patch, > HIVE-20437.3.patch > > > When data created as float, double or decimal in parquet format is read back > using some other type, errors are seen. Parquet should behave just like any > other format. If the value is valid for the new type, data is retuned > otherwise null has to be returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20526) Add test case for HIVE-20489
[ https://issues.apache.org/jira/browse/HIVE-20526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16612571#comment-16612571 ] Andrew Sherman commented on HIVE-20526: --- +1 LGTM > Add test case for HIVE-20489 > > > Key: HIVE-20526 > URL: https://issues.apache.org/jira/browse/HIVE-20526 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-20526.1.patch > > > Add a test case for the issue discussed in HIVE-20489. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20527) Intern table descriptors from spark task
[ https://issues.apache.org/jira/browse/HIVE-20527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16612572#comment-16612572 ] Andrew Sherman commented on HIVE-20527: --- +1 LGTM pending clean test run > Intern table descriptors from spark task > > > Key: HIVE-20527 > URL: https://issues.apache.org/jira/browse/HIVE-20527 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-20527.1.patch > > > Table descriptors from MR tasks and Tez tasks are interned. This fix is to > intern table desc from spark tasks as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20526) Add test case for HIVE-20489
[ https://issues.apache.org/jira/browse/HIVE-20526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16615167#comment-16615167 ] Andrew Sherman commented on HIVE-20526: --- Pushed to master, thanks [~janulatha] > Add test case for HIVE-20489 > > > Key: HIVE-20526 > URL: https://issues.apache.org/jira/browse/HIVE-20526 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-20526.1.patch, HIVE-20526.1.patch > > > Add a test case for the issue discussed in HIVE-20489. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20527) Intern table descriptors from spark task
[ https://issues.apache.org/jira/browse/HIVE-20527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16615170#comment-16615170 ] Andrew Sherman commented on HIVE-20527: --- Pushed to master, thanks [~janulatha] > Intern table descriptors from spark task > > > Key: HIVE-20527 > URL: https://issues.apache.org/jira/browse/HIVE-20527 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-20527.1.patch, HIVE-20527.1.patch, > HIVE-20527.1.patch > > > Table descriptors from MR tasks and Tez tasks are interned. This fix is to > intern table desc from spark tasks as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20570) Union ALL with hive.optimize.union.remove=true has incorrect plan
[
https://issues.apache.org/jira/browse/HIVE-20570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16619920#comment-16619920
]
Andrew Sherman commented on HIVE-20570:
---
+1 LGTM pending test results
> Union ALL with hive.optimize.union.remove=true has incorrect plan
> -
>
> Key: HIVE-20570
> URL: https://issues.apache.org/jira/browse/HIVE-20570
> Project: Hive
> Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-20570.1.patch, HIVE-20570.2.patch,
> HIVE-20570.3.patch
>
>
> When hive.optimize.union.remove=true and a select query is run with group by,
> the final fetch is waiting only for one of the branches and not both.
> Test Case:
> {code}
> create table if not exists test_table(column1 string, column2 int);
> insert into test_table values('a',1),('b',2);
> set hive.optimize.union.remove=true;
> set mapred.input.dir.recursive=true;
> explain
> select column1 from test_table group by column1
> union all
> select column1 from test_table group by column1;
> {code}
> In the below the two stages correspond to the two parts of union all. But
> the final fetch operator (Stage 0) only depends on one of the stages, but it
> should depend on both.
> Plan:
> {code}
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-2 is a root stage
> *Stage-0 depends on stages: Stage-1*
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: test_table
> Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column
> stats: NONE
> Select Operator
> expressions: column1 (type: string)
> outputColumnNames: column1
> Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE
> Column stats: NONE
> Group By Operator
> keys: column1 (type: string)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE
> Column stats: NONE
> Reduce Output Operator
> key expressions: _col0 (type: string)
> sort order: +
> Map-reduce partition columns: _col0 (type: string)
> Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE
> Column stats: NONE
> Execution mode: vectorized
> Reduce Operator Tree:
> Group By Operator
> keys: KEY._col0 (type: string)
> mode: mergepartial
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column
> stats: NONE
> File Output Operator
> compressed: false
> Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column
> stats: NONE
> table:
> input format: org.apache.hadoop.mapred.SequenceFileInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> Stage: Stage-2
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: test_table
> Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column
> stats: NONE
> Select Operator
> expressions: column1 (type: string)
> outputColumnNames: column1
> Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE
> Column stats: NONE
> Group By Operator
> keys: column1 (type: string)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE
> Column stats: NONE
> Reduce Output Operator
> key expressions: _col0 (type: string)
> sort order: +
> Map-reduce partition columns: _col0 (type: string)
> Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE
> Column stats: NONE
> Execution mode: vectorized
> Reduce Operator Tree:
> Group By Operator
> keys: KEY._col0 (type: string)
> mode: mergepartial
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column
> stats: NONE
> File Output Operator
> compressed: false
> Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column
> stats: NONE
> table:
> input format: org.apache.hadoop.mapred.SequenceFileInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
>
[jira] [Commented] (HIVE-20601) EnvironmentContext null in ALTER_PARTITION event in DbNotificationListener
[ https://issues.apache.org/jira/browse/HIVE-20601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626622#comment-16626622 ] Andrew Sherman commented on HIVE-20601: --- +1 LGTM pending some clean test runs > EnvironmentContext null in ALTER_PARTITION event in DbNotificationListener > -- > > Key: HIVE-20601 > URL: https://issues.apache.org/jira/browse/HIVE-20601 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0, 4.0.0 >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-20601.1.branch-3.patch, HIVE-20601.1.patch > > > Cause : EnvironmentContext not passed here: > [https://github.com/apache/hive/blob/36c33ca066c99dfdb21223a711c0c3f33c85b943/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java#L726] > > It will be useful to have the environmentContext passed to > DbNotificationListener in this case, to know if the alter happened due to a > stat change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20601) EnvironmentContext null in ALTER_PARTITION event in DbNotificationListener
[ https://issues.apache.org/jira/browse/HIVE-20601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627653#comment-16627653 ] Andrew Sherman commented on HIVE-20601: --- Pushed to master, thanks [~bharos92] > EnvironmentContext null in ALTER_PARTITION event in DbNotificationListener > -- > > Key: HIVE-20601 > URL: https://issues.apache.org/jira/browse/HIVE-20601 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0, 4.0.0 >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-20601.1.branch-3.patch, HIVE-20601.1.patch, > HIVE-20601.2.patch > > > Cause : EnvironmentContext not passed here: > [https://github.com/apache/hive/blob/36c33ca066c99dfdb21223a711c0c3f33c85b943/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java#L726] > > It will be useful to have the environmentContext passed to > DbNotificationListener in this case, to know if the alter happened due to a > stat change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20365) Fix warnings when regenerating thrift code
[ https://issues.apache.org/jira/browse/HIVE-20365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632655#comment-16632655 ] Andrew Sherman commented on HIVE-20365: --- One day we will have to really deal with wire format compatibility. As St Augustine said "Lord, make me pure – but not yet!”. +1 LGTM > Fix warnings when regenerating thrift code > -- > > Key: HIVE-20365 > URL: https://issues.apache.org/jira/browse/HIVE-20365 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20365.01.patch > > > When you build thrift code you can see thrift warning like below. > [exec] > [WARNING:hive/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:2167] > No field key specified for rqst, resulting protocol may have conflicts or > not be backwards compatible! > [exec] > [exec] > [WARNING:hive/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:2235] > No field key specified for o2, resulting protocol may have conflicts or not > be backwards compatible! > [exec] > [exec] > [WARNING:hive/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:2167] > No field key specified for rqst, resulting protocol may have conflicts or > not be backwards compatible! > [exec] > [exec] > [WARNING:hive/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:2235] > No field key specified for o2, resulting protocol may have conflicts or not > be backwards compatible! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20610) TestDbNotificationListener should not use /tmp directory
[
https://issues.apache.org/jira/browse/HIVE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636183#comment-16636183
]
Andrew Sherman commented on HIVE-20610:
---
+1 LGTM pending test results
> TestDbNotificationListener should not use /tmp directory
>
>
> Key: HIVE-20610
> URL: https://issues.apache.org/jira/browse/HIVE-20610
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20610.1.patch, HIVE-20610.2.patch
>
>
> Using /tmp directory creates exceptions for tests like dropTable :
> {code:java}
> 2018-09-19T06:42:04,818 INFO [main] metastore.HiveMetaStore: 0: drop_table :
> tbl=hive.default.droptbl
> 2018-09-19T06:42:04,819 INFO [main] HiveMetaStore.audit: ugi=hiveptest
> ip=unknown-ip-addr cmd=drop_table : tbl=hive.default.droptbl
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.ICE-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.XIM-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.X11-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/hsperfdata_root]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.font-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.Test-unix]: it still exists.
> 2018-09-19T06:42:05,072 ERROR [main] utils.FileUtils: Failed to delete
> file:/tmp
> 2018-09-19T06:42:05,072 ERROR [main] utils.MetaStoreUtils: Got exception:
> org.apache.hadoop.hive.metastore.api.MetaException Unable to delete
> directory: file:/tmp
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to delete
> directory: file:/tmp
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreFsImpl.deleteDir(HiveMetaStoreFsImpl.java:45)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:365)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:353)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.deleteTableData(HiveMetaStore.java:2562)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:2523)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:2685)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_102]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy33.drop_table_with_environment_context(Unknown
> Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.drop_table_with_environment_context(HiveMetaStoreClient.java:3204)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1492)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1432)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable(TestDbNotificationListener.java:522)
> [test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]{code}
>
>
--
This message was sent by Atlassian
[jira] [Commented] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636304#comment-16636304 ] Andrew Sherman commented on HIVE-20306: --- +1 LGTM with small spelling fixes from RB > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch, > HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch, > HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20545) Ability to exclude potentially large parameters in HMS Notifications
[ https://issues.apache.org/jira/browse/HIVE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636399#comment-16636399 ] Andrew Sherman commented on HIVE-20545: --- +1 LGTM > Ability to exclude potentially large parameters in HMS Notifications > > > Key: HIVE-20545 > URL: https://issues.apache.org/jira/browse/HIVE-20545 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-20545.1.patch, HIVE-20545.2.patch, > HIVE-20545.3.branch-3.patch, HIVE-20545.3.patch, HIVE-20545.4.patch, > HIVE-20545.6.patch, HIVE-20545.7.patch > > > Clients can add large-sized parameters in Table/Partition objects. So we need > to enable adding regex patterns through HiveConf to match parameters to be > filtered from table and partition objects before serialization in HMS > notifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18054) Make Lineage work with concurrent queries on a Session
[ https://issues.apache.org/jira/browse/HIVE-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18054: -- Attachment: HIVE-18054.12.patch > Make Lineage work with concurrent queries on a Session > --- > > Key: HIVE-18054 > URL: https://issues.apache.org/jira/browse/HIVE-18054 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman > Attachments: HIVE-18054.1.patch, HIVE-18054.10.patch, > HIVE-18054.11.patch, HIVE-18054.12.patch, HIVE-18054.2.patch, > HIVE-18054.3.patch, HIVE-18054.4.patch, HIVE-18054.5.patch, > HIVE-18054.6.patch, HIVE-18054.7.patch, HIVE-18054.8.patch, HIVE-18054.9.patch > > > A Hive Session can contain multiple concurrent sql Operations. > Lineage is currently tracked in SessionState and is cleared when a query > completes. This results in Lineage for other running queries being lost. > To fix this, move LineageState from SessionState to QueryState. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18054) Make Lineage work with concurrent queries on a Session
[ https://issues.apache.org/jira/browse/HIVE-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18054: -- Attachment: HIVE-18054.13.patch try again to get tests to run > Make Lineage work with concurrent queries on a Session > --- > > Key: HIVE-18054 > URL: https://issues.apache.org/jira/browse/HIVE-18054 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman > Attachments: HIVE-18054.1.patch, HIVE-18054.10.patch, > HIVE-18054.11.patch, HIVE-18054.12.patch, HIVE-18054.13.patch, > HIVE-18054.2.patch, HIVE-18054.3.patch, HIVE-18054.4.patch, > HIVE-18054.5.patch, HIVE-18054.6.patch, HIVE-18054.7.patch, > HIVE-18054.8.patch, HIVE-18054.9.patch > > > A Hive Session can contain multiple concurrent sql Operations. > Lineage is currently tracked in SessionState and is cleared when a query > completes. This results in Lineage for other running queries being lost. > To fix this, move LineageState from SessionState to QueryState. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18228) Azure credential properties should be added to the HiveConf hidden list
[
https://issues.apache.org/jira/browse/HIVE-18228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18228:
--
Attachment: HIVE-18228.3.patch
try to get tests to run
> Azure credential properties should be added to the HiveConf hidden list
> ---
>
> Key: HIVE-18228
> URL: https://issues.apache.org/jira/browse/HIVE-18228
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-18228.1.patch, HIVE-18228.2.patch,
> HIVE-18228.3.patch
>
>
> The HIVE_CONF_HIDDEN_LIST("hive.conf.hidden.list") already contains keys
> contaiing aws credentials. The Azure properties to be added are:
> * dfs.adls.oauth2.credential
> * fs.adl.oauth2.credential
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18263) Ptest execution are multiple times slower sometimes due to dying executor slaves
[
https://issues.apache.org/jira/browse/HIVE-18263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16287840#comment-16287840
]
Andrew Sherman commented on HIVE-18263:
---
Thanks [~szita] for working on this important but unglamourous issue
> Ptest execution are multiple times slower sometimes due to dying executor
> slaves
>
>
> Key: HIVE-18263
> URL: https://issues.apache.org/jira/browse/HIVE-18263
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: HIVE-18263.0.patch
>
>
> PreCommit-HIVE-Build job has been seen running very long from time to time.
> Usually it should take about 1.5 hours, but in some cases it took over 4-5
> hours.
> Looking in the logs of one such execution I've seen that some commands that
> were sent to test executing slaves returned 255. Here this typically means
> that there is unknown return code for the remote call since hiveptest-server
> can't reach these slaves anymore.
> In the hiveptest-server logs it is seen that some slaves were killed while
> running the job normally, and here is why:
> * Hive's ptest-server checks periodically in every 60 minutes the status of
> slaves. It also keeps track of slaves that were terminated.
> ** If upon such check it is found that a slave that was already killed
> ([mTerminatedHosts
> map|https://github.com/apache/hive/blob/master/testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/CloudExecutionContextProvider.java#L93]
> contains its IP) is still running, it will try and terminate it again.
> * The server also maintains a file on its local FS that contains the IP of
> hosts that were used before. (This probably for resilience reasons)
> ** This file is read when tomcat server starts and if any of the IPs in the
> file are seen as running slaves, ptest will terminate these first so it can
> begin with a fresh start
> ** The IPs of these terminated instances already make their way into
> {{mTerminatedHosts}} upon initialization...
> * The cloud provider may reuse some older IPs, so it is not too rare that the
> same IP that belonged to a terminated host is assigned to a new one.
> This is problematic: Hive ptest's slave caretaker thread kicks in every 60
> minutes and might see a running host that has the same IP as an old slave had
> which was terminated at startup. It will think that this host should be
> terminated since it already tried 60 minutes ago as its IP is in
> {{mTerminatedHosts}}
> We have to fix this by making sure that if a new slave is created, we check
> the contents of {{mTerminatedHosts}} and remove this IP from it if it is
> there.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18054) Make Lineage work with concurrent queries on a Session
[ https://issues.apache.org/jira/browse/HIVE-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16288080#comment-16288080 ] Andrew Sherman commented on HIVE-18054: --- Test failures are unconnected with this change. So I think this is ready to push [~stakiar] if you agree. > Make Lineage work with concurrent queries on a Session > --- > > Key: HIVE-18054 > URL: https://issues.apache.org/jira/browse/HIVE-18054 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman > Attachments: HIVE-18054.1.patch, HIVE-18054.10.patch, > HIVE-18054.11.patch, HIVE-18054.12.patch, HIVE-18054.13.patch, > HIVE-18054.2.patch, HIVE-18054.3.patch, HIVE-18054.4.patch, > HIVE-18054.5.patch, HIVE-18054.6.patch, HIVE-18054.7.patch, > HIVE-18054.8.patch, HIVE-18054.9.patch > > > A Hive Session can contain multiple concurrent sql Operations. > Lineage is currently tracked in SessionState and is cleared when a query > completes. This results in Lineage for other running queries being lost. > To fix this, move LineageState from SessionState to QueryState. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18228) Azure credential properties should be added to the HiveConf hidden list
[
https://issues.apache.org/jira/browse/HIVE-18228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16288112#comment-16288112
]
Andrew Sherman commented on HIVE-18228:
---
Test failures look unconnected to this change, so I think this is ready to push
if you agree [~pvary]
> Azure credential properties should be added to the HiveConf hidden list
> ---
>
> Key: HIVE-18228
> URL: https://issues.apache.org/jira/browse/HIVE-18228
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-18228.1.patch, HIVE-18228.2.patch,
> HIVE-18228.3.patch
>
>
> The HIVE_CONF_HIDDEN_LIST("hive.conf.hidden.list") already contains keys
> contaiing aws credentials. The Azure properties to be added are:
> * dfs.adls.oauth2.credential
> * fs.adl.oauth2.credential
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18268) Hive Prepared Statement when split with double quoted in query fails
[
https://issues.apache.org/jira/browse/HIVE-18268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289598#comment-16289598
]
Andrew Sherman commented on HIVE-18268:
---
I think your change is to allow a ? as a string literal (if it is quoted).
But in your test
{noformat}
String sql = "select 1 from x where qa=\"?\"";
HivePreparedStatement ps = new HivePreparedStatement(connection, client,
sessHandle, sql);
ps.setString(1, "v");
{noformat}
you are setting a parameter. So is the quoted ? supposed to be a literal
Thanks
> Hive Prepared Statement when split with double quoted in query fails
>
>
> Key: HIVE-18268
> URL: https://issues.apache.org/jira/browse/HIVE-18268
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.3.2
>Reporter: Choi JaeHwan
>Assignee: Choi JaeHwan
> Fix For: 2.3.3
>
> Attachments: HIVE-18268.patch
>
>
> HIVE-13625, Change sql statement split when odd number of escape characters,
> and add parameter counter validation, above
> {code:java}
> // prev code
> StringBuilder newSql = new StringBuilder(parts.get(0));
> for(int i=1;i if(!parameters.containsKey(i)){
> throw new SQLException("Parameter #"+i+" is unset");
> }
> newSql.append(parameters.get(i));
> newSql.append(parts.get(i));
> }
> // change from HIVE-13625
> int paramLoc = 1;
> while (getCharIndexFromSqlByParamLocation(sql, '?', paramLoc) > 0) {
> // check the user has set the needs parameters
> if (parameters.containsKey(paramLoc)) {
> int tt = getCharIndexFromSqlByParamLocation(newSql.toString(), '?',
> 1);
> newSql.deleteCharAt(tt);
> newSql.insert(tt, parameters.get(paramLoc));
> }
> paramLoc++;
> }
> {code}
> If the number of split SQL and the number of parameters are not matched, an
> SQLException is thrown
> Currently, when splitting SQL, there is no processing for double quoted, and
> when the token ('?' ) is between double quote, SQL is split.
> i think when the token between double quoted is literal, it is correct to not
> split.
> for example, above the query;
> {code:java}
> // Some comments here
> 1: String query = " select 1 from x where qa="?" "
> 2: String query = " SELECT 1 FROM `x` WHERE (trecord LIKE "ALA[d_?]%")
> {code}
> ? is literal, then query do not split.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18265) desc formatted/extended or show create table can not fully display the result when field or table comment contains tab character
[
https://issues.apache.org/jira/browse/HIVE-18265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289665#comment-16289665
]
Andrew Sherman commented on HIVE-18265:
---
Is it possible to add a test for this? Thanks
> desc formatted/extended or show create table can not fully display the result
> when field or table comment contains tab character
>
>
> Key: HIVE-18265
> URL: https://issues.apache.org/jira/browse/HIVE-18265
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 3.0.0
>Reporter: Hui Huang
>Assignee: Hui Huang
> Fix For: 3.0.0
>
> Attachments: HIVE-18265.patch
>
>
> Here are some examples:
> create table test_comment (id1 string comment 'full_\tname1', id2 string
> comment 'full_\tname2', id3 string comment 'full_\tname3') stored as textfile;
> When execute `show create table test_comment`, we can see the following
> content in the console,
> {quote}
> createtab_stmt
> CREATE TABLE `test_comment`(
> `id1` string COMMENT 'full_
> `id2` string COMMENT 'full_
> `id3` string COMMENT 'full_
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'hdfs://xxx/user/huanghui/warehouse/huanghuitest.db/test_comment'
> TBLPROPERTIES (
> 'transient_lastDdlTime'='1513095570')
> {quote}
> And the output of `desc formatted table ` is a little similar,
> {quote}
> col_name data_type comment
> \# col_name data_type comment
> id1 string full_
> id2 string full_
> id3 string full_
> \# Detailed Table Information
> (ignore)...
> {quote}
> When execute `desc extended test_comment`, the problem is more obvious,
> {quote}
> col_name data_type comment
> id1 string full_
> id2 string full_
> id3 string full_
> Detailed Table InformationTable(tableName:test_comment,
> dbName:huanghuitest, owner:huanghui, createTime:1513095570, lastAccessTime:0,
> retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:id1, type:string,
> comment:full_name1), FieldSchema(name:id2, type:string, comment:full_
> {quote}
> *the rest of the content is lost*.
> The content is not really lost, it's just can not display normal. Because
> hive store the result in LazyStruct, and LazyStruct use '\t' as field
> separator:
> {code:java}
> // LazyStruct.java#parse()
> // Go through all bytes in the byte[]
> while (fieldByteEnd <= structByteEnd) {
> if (fieldByteEnd == structByteEnd || bytes[fieldByteEnd] == separator) {
> // Reached the end of a field?
> if (lastColumnTakesRest && fieldId == fields.length - 1) {
> fieldByteEnd = structByteEnd;
> }
> startPosition[fieldId] = fieldByteBegin;
> fieldId++;
> if (fieldId == fields.length || fieldByteEnd == structByteEnd) {
> // All fields have been parsed, or bytes have been parsed.
> // We need to set the startPosition of fields.length to ensure we
> // can use the same formula to calculate the length of each field.
> // For missing fields, their starting positions will all be the
> same,
> // which will make their lengths to be -1 and uncheckedGetField will
> // return these fields as NULLs.
> for (int i = fieldId; i <= fields.length; i++) {
> startPosition[i] = fieldByteEnd + 1;
> }
> break;
> }
> fieldByteBegin = fieldByteEnd + 1;
> fieldByteEnd++;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18228) Azure credential properties should be added to the HiveConf hidden list
[
https://issues.apache.org/jira/browse/HIVE-18228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289774#comment-16289774
]
Andrew Sherman commented on HIVE-18228:
---
Thanks [~pvary] for the commit and the suggestion about documentation.
> Azure credential properties should be added to the HiveConf hidden list
> ---
>
> Key: HIVE-18228
> URL: https://issues.apache.org/jira/browse/HIVE-18228
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-18228.1.patch, HIVE-18228.2.patch,
> HIVE-18228.3.patch
>
>
> The HIVE_CONF_HIDDEN_LIST("hive.conf.hidden.list") already contains keys
> contaiing aws credentials. The Azure properties to be added are:
> * dfs.adls.oauth2.credential
> * fs.adl.oauth2.credential
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18281) HiveServer2 HA for LLAP and Workload Manager
[ https://issues.apache.org/jira/browse/HIVE-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292841#comment-16292841 ] Andrew Sherman commented on HIVE-18281: --- Can this be used in configurations that are not using Tez and LLAP? > HiveServer2 HA for LLAP and Workload Manager > > > Key: HIVE-18281 > URL: https://issues.apache.org/jira/browse/HIVE-18281 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-18281.WIP.patch, HSI-HA.pdf > > > When running HS2 with LLAP and Workload Manager, HS2 becomes single point of > failure as some of the states for workload management and scheduling are > maintained in-memory. > The proposal is to support Active/Passive mode of high availability in which, > all HS2 and tez AMs registers with ZooKeeper and a leader have to be chosen > which will maintain stateful information. Clients using service discovery > will always connect to the leader for submitting queries. There will also be > some responsibilities for the leader, failover handling, tez session > reconnect etc. Will upload some more detailed information in a separate doc. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18283) Better error message and error code for HoS exceptions
[
https://issues.apache.org/jira/browse/HIVE-18283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292899#comment-16292899
]
Andrew Sherman commented on HIVE-18283:
---
As the new getHiveException() method could be fragile, is it worth having a
unit test that exercises it with some actual errror messages?
> Better error message and error code for HoS exceptions
> --
>
> Key: HIVE-18283
> URL: https://issues.apache.org/jira/browse/HIVE-18283
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-18283.0.patch
>
>
> Right now HoS only use a few error codes. For the majority of the errors,
> user will see an error code 1 followed by a lengthy stacktrace. This is not
> ideal since:
> 1. It is often hard to find the root cause - sometimes it is hidden deeply
> inside the stacktrace.
> 2. After identifying the root cause, it is not easy to find a fix. Often user
> have to copy & paste the error message and google them.
> 3. It is not clear whether the error is transient or not, depending on which
> user may want to retry the query.
> To improve the above, this JIRA propose to assign error code & canonical
> error messages for different HoS errors. We can take advantage of the
> existing {{ErrorMsg}} class.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18054) Make Lineage work with concurrent queries on a Session
[ https://issues.apache.org/jira/browse/HIVE-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295323#comment-16295323 ] Andrew Sherman commented on HIVE-18054: --- Thanks [~stakiar] > Make Lineage work with concurrent queries on a Session > --- > > Key: HIVE-18054 > URL: https://issues.apache.org/jira/browse/HIVE-18054 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman > Fix For: 3.0.0 > > Attachments: HIVE-18054.1.patch, HIVE-18054.10.patch, > HIVE-18054.11.patch, HIVE-18054.12.patch, HIVE-18054.13.patch, > HIVE-18054.2.patch, HIVE-18054.3.patch, HIVE-18054.4.patch, > HIVE-18054.5.patch, HIVE-18054.6.patch, HIVE-18054.7.patch, > HIVE-18054.8.patch, HIVE-18054.9.patch > > > A Hive Session can contain multiple concurrent sql Operations. > Lineage is currently tracked in SessionState and is cleared when a query > completes. This results in Lineage for other running queries being lost. > To fix this, move LineageState from SessionState to QueryState. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18283) Better error message and error code for HoS exceptions
[
https://issues.apache.org/jira/browse/HIVE-18283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295967#comment-16295967
]
Andrew Sherman commented on HIVE-18283:
---
Patch looks good to me.
If you do rework it again you might
1) change the order of arguments to assertEquals as the expected value is
supposed to be first in junit
2) refactor out some of testGetHiveException() so that you can call
{noformat}
checkExpectedException("\t diagnostics: Application
application_1508358311878_3322732 "
+ "failed 1 times due to ApplicationMaster for attempt "
+ "appattempt_1508358311878_3322732_01 timed out. Failing
the application.",
ErrorMsg.SPARK_CREATE_CLIENT_TIMEOUT);
checkExpectedException("\t diagnostics: Application
application_1508358311878_333 "
+ "submitted by user hive to unknown queue: foo",
ErrorMsg.SPARK_CREATE_CLIENT_INVALID_QUEUE);
{noformat}
etc. as this would make it more readable and extendable.
But these are nits, the basic code is good and will be useful
> Better error message and error code for HoS exceptions
> --
>
> Key: HIVE-18283
> URL: https://issues.apache.org/jira/browse/HIVE-18283
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-18283.0.patch, HIVE-18283.1.patch,
> HIVE-18283.2.patch
>
>
> Right now HoS only use a few error codes. For the majority of the errors,
> user will see an error code 1 followed by a lengthy stacktrace. This is not
> ideal since:
> 1. It is often hard to find the root cause - sometimes it is hidden deeply
> inside the stacktrace.
> 2. After identifying the root cause, it is not easy to find a fix. Often user
> have to copy & paste the error message and google them.
> 3. It is not clear whether the error is transient or not, depending on which
> user may want to retry the query.
> To improve the above, this JIRA propose to assign error code & canonical
> error messages for different HoS errors. We can take advantage of the
> existing {{ErrorMsg}} class.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18283) Better error message and error code for HoS exceptions
[
https://issues.apache.org/jira/browse/HIVE-18283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297042#comment-16297042
]
Andrew Sherman commented on HIVE-18283:
---
Thanks [~csun]. The patch looks good to me. +1
> Better error message and error code for HoS exceptions
> --
>
> Key: HIVE-18283
> URL: https://issues.apache.org/jira/browse/HIVE-18283
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-18283.0.patch, HIVE-18283.1.patch,
> HIVE-18283.2.patch, HIVE-18283.3.patch
>
>
> Right now HoS only use a few error codes. For the majority of the errors,
> user will see an error code 1 followed by a lengthy stacktrace. This is not
> ideal since:
> 1. It is often hard to find the root cause - sometimes it is hidden deeply
> inside the stacktrace.
> 2. After identifying the root cause, it is not easy to find a fix. Often user
> have to copy & paste the error message and google them.
> 3. It is not clear whether the error is transient or not, depending on which
> user may want to retry the query.
> To improve the above, this JIRA propose to assign error code & canonical
> error messages for different HoS errors. We can take advantage of the
> existing {{ErrorMsg}} class.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18310) Test 'vector_reduce_groupby_duplicate_cols.q' is misspelled in testconfiguration.properties
[
https://issues.apache.org/jira/browse/HIVE-18310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-18310:
-
> Test 'vector_reduce_groupby_duplicate_cols.q' is misspelled in
> testconfiguration.properties
> ---
>
> Key: HIVE-18310
> URL: https://issues.apache.org/jira/browse/HIVE-18310
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>Priority: Minor
>
> The new testvector_reduce_groupby_duplicate_cols.q was introduced in
> [HIVE-18258] but is misspelled in testconfiguration.properties:
> {noformat}
> - vector_reduce_grpupby_duplicate_cols.q,\
> + vector_reduce_groupby_duplicate_cols.q,\
> {noformat}
> I noticed this because TestDanglingQOuts.checkDanglingQOut failed
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18310) Test 'vector_reduce_groupby_duplicate_cols.q' is misspelled in testconfiguration.properties
[
https://issues.apache.org/jira/browse/HIVE-18310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18310:
--
Status: Patch Available (was: Open)
> Test 'vector_reduce_groupby_duplicate_cols.q' is misspelled in
> testconfiguration.properties
> ---
>
> Key: HIVE-18310
> URL: https://issues.apache.org/jira/browse/HIVE-18310
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18310.1.patch
>
>
> The new testvector_reduce_groupby_duplicate_cols.q was introduced in
> [HIVE-18258] but is misspelled in testconfiguration.properties:
> {noformat}
> - vector_reduce_grpupby_duplicate_cols.q,\
> + vector_reduce_groupby_duplicate_cols.q,\
> {noformat}
> I noticed this because TestDanglingQOuts.checkDanglingQOut failed
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18310) Test 'vector_reduce_groupby_duplicate_cols.q' is misspelled in testconfiguration.properties
[
https://issues.apache.org/jira/browse/HIVE-18310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18310:
--
Attachment: HIVE-18310.1.patch
> Test 'vector_reduce_groupby_duplicate_cols.q' is misspelled in
> testconfiguration.properties
> ---
>
> Key: HIVE-18310
> URL: https://issues.apache.org/jira/browse/HIVE-18310
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18310.1.patch
>
>
> The new testvector_reduce_groupby_duplicate_cols.q was introduced in
> [HIVE-18258] but is misspelled in testconfiguration.properties:
> {noformat}
> - vector_reduce_grpupby_duplicate_cols.q,\
> + vector_reduce_groupby_duplicate_cols.q,\
> {noformat}
> I noticed this because TestDanglingQOuts.checkDanglingQOut failed
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18291) An exception should be raised if the result is outside the range of decimal
[
https://issues.apache.org/jira/browse/HIVE-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297105#comment-16297105
]
Andrew Sherman commented on HIVE-18291:
---
Thanks [~mgaido] [~kgyrtkirk] for the comparisons. +1 for following sql
standard by default with a switch for backwards compatibility
> An exception should be raised if the result is outside the range of decimal
> ---
>
> Key: HIVE-18291
> URL: https://issues.apache.org/jira/browse/HIVE-18291
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 2.3.0
>Reporter: Marco Gaido
>Assignee: Daniel Voros
>
> Citing SQL:2011 on page 27 available at
> http://standards.iso.org/ittf/PubliclyAvailableStandards/c053681_ISO_IEC_9075-1_2011.zip:
> {noformat}
> If the result cannot be represented exactly in the result type, then whether
> it is rounded
> or truncated is implementation-defined. An exception condition is raised if
> the result is
> outside the range of numeric values of the result type, or if the arithmetic
> operation
> is not defined for the operands.
> {noformat}
> Currently Hive is returning NULL instead of throwing an exception if the
> result is out of range, eg.:
> {code}
> > select 100.01*100.01;
> +---+
> | _c0 |
> +---+
> | NULL |
> +---+
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
