[jira] [Commented] (HIVE-15061) Metastore types are sometimes case sensitive
[ https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622361#comment-15622361 ] Mohit Sabharwal commented on HIVE-15061: LGTM +1 (I double-checked other places where coltypes are set like StatObjectConverter - found that we are converting to lower case there already) > Metastore types are sometimes case sensitive > > > Key: HIVE-15061 > URL: https://issues.apache.org/jira/browse/HIVE-15061 > Project: Hive > Issue Type: Bug > Components: API >Affects Versions: 1.1.0 >Reporter: Thomas Tauber-Marshall >Assignee: Chaoyu Tang > Attachments: HIVE-15061.1.patch, HIVE-15061.1.patch, HIVE-15061.patch > > > Impala recently encountered an issue with the metastore > ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column > stats would get dropped when adding a column to a table. > The reason seems to be that Hive does a case sensitive check on the column > stats types during an "alter table" and expects the types to be all lower > case. This case sensitive check doesn't appear to happen when the stats are > set in the first place. > We're solving this on the Impala end by storing types in the metastore as all > lower case, but Hive's behavior here is very confusing. It should either > always be case sensitive, so that you can't create column stats with types > that Hive considers invalid, or it should never be case sensitive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15061) Metastore types are sometimes case sensitive
[ https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622176#comment-15622176 ] Chaoyu Tang commented on HIVE-15061: The failed test seems not related to the patch. [~aihuaxu], [~mohitsabharwal] could you review it, thanks. > Metastore types are sometimes case sensitive > > > Key: HIVE-15061 > URL: https://issues.apache.org/jira/browse/HIVE-15061 > Project: Hive > Issue Type: Bug > Components: API >Affects Versions: 1.1.0 >Reporter: Thomas Tauber-Marshall >Assignee: Chaoyu Tang > Attachments: HIVE-15061.1.patch, HIVE-15061.1.patch, HIVE-15061.patch > > > Impala recently encountered an issue with the metastore > ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column > stats would get dropped when adding a column to a table. > The reason seems to be that Hive does a case sensitive check on the column > stats types during an "alter table" and expects the types to be all lower > case. This case sensitive check doesn't appear to happen when the stats are > set in the first place. > We're solving this on the Impala end by storing types in the metastore as all > lower case, but Hive's behavior here is very confusing. It should either > always be case sensitive, so that you can't create column stats with types > that Hive considers invalid, or it should never be case sensitive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15061) Metastore types are sometimes case sensitive
[
https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15617092#comment-15617092
]
Hive QA commented on HIVE-15061:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12835876/HIVE-15061.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10626 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.druid.TestHiveDruidQueryBasedInputFormat.testCreateSplitsIntervals
(batchId=229)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1871/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1871/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1871/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12835876 - PreCommit-HIVE-Build
> Metastore types are sometimes case sensitive
>
>
> Key: HIVE-15061
> URL: https://issues.apache.org/jira/browse/HIVE-15061
> Project: Hive
> Issue Type: Bug
> Components: API
>Affects Versions: 1.1.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Chaoyu Tang
> Attachments: HIVE-15061.1.patch, HIVE-15061.1.patch, HIVE-15061.patch
>
>
> Impala recently encountered an issue with the metastore
> ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column
> stats would get dropped when adding a column to a table.
> The reason seems to be that Hive does a case sensitive check on the column
> stats types during an "alter table" and expects the types to be all lower
> case. This case sensitive check doesn't appear to happen when the stats are
> set in the first place.
> We're solving this on the Impala end by storing types in the metastore as all
> lower case, but Hive's behavior here is very confusing. It should either
> always be case sensitive, so that you can't create column stats with types
> that Hive considers invalid, or it should never be case sensitive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15061) Metastore types are sometimes case sensitive
[
https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610425#comment-15610425
]
Hive QA commented on HIVE-15061:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12835429/HIVE-15061.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10621 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_columnar]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_create]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_table_with_subschema]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_complex_all]
(batchId=54)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_create]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_complex_all]
(batchId=146)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0]
(batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] (batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] (batchId=164)
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema[1]
(batchId=170)
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteVarchar
(batchId=170)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1834/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1834/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1834/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12835429 - PreCommit-HIVE-Build
> Metastore types are sometimes case sensitive
>
>
> Key: HIVE-15061
> URL: https://issues.apache.org/jira/browse/HIVE-15061
> Project: Hive
> Issue Type: Bug
> Components: API
>Affects Versions: 1.1.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Chaoyu Tang
> Attachments: HIVE-15061.patch
>
>
> Impala recently encountered an issue with the metastore
> ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column
> stats would get dropped when adding a column to a table.
> The reason seems to be that Hive does a case sensitive check on the column
> stats types during an "alter table" and expects the types to be all lower
> case. This case sensitive check doesn't appear to happen when the stats are
> set in the first place.
> We're solving this on the Impala end by storing types in the metastore as all
> lower case, but Hive's behavior here is very confusing. It should either
> always be case sensitive, so that you can't create column stats with types
> that Hive considers invalid, or it should never be case sensitive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15061) Metastore types are sometimes case sensitive
[ https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609006#comment-15609006 ] Chaoyu Tang commented on HIVE-15061: In Hive alter table, both column name and type in the newTable object have been converted to lower case before it is passed to HMS APIs, therefore there won't be the case sensitive issues as seen in Impala. Impala might do the same way as Hive to work around this issue (which seems Impala is doing), or we might seek a way to validate column names/types and do the case conversion in HMS alter table. > Metastore types are sometimes case sensitive > > > Key: HIVE-15061 > URL: https://issues.apache.org/jira/browse/HIVE-15061 > Project: Hive > Issue Type: Bug > Components: API >Affects Versions: 1.1.0 >Reporter: Thomas Tauber-Marshall >Assignee: Chaoyu Tang > > Impala recently encountered an issue with the metastore > ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column > stats would get dropped when adding a column to a table. > The reason seems to be that Hive does a case sensitive check on the column > stats types during an "alter table" and expects the types to be all lower > case. This case sensitive check doesn't appear to happen when the stats are > set in the first place. > We're solving this on the Impala end by storing types in the metastore as all > lower case, but Hive's behavior here is very confusing. It should either > always be case sensitive, so that you can't create column stats with types > that Hive considers invalid, or it should never be case sensitive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15061) Metastore types are sometimes case sensitive
[ https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15608633#comment-15608633 ] Chaoyu Tang commented on HIVE-15061: I can take a look. > Metastore types are sometimes case sensitive > > > Key: HIVE-15061 > URL: https://issues.apache.org/jira/browse/HIVE-15061 > Project: Hive > Issue Type: Bug > Components: API >Affects Versions: 1.1.0 >Reporter: Thomas Tauber-Marshall >Assignee: Chaoyu Tang > > Impala recently encountered an issue with the metastore > ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column > stats would get dropped when adding a column to a table. > The reason seems to be that Hive does a case sensitive check on the column > stats types during an "alter table" and expects the types to be all lower > case. This case sensitive check doesn't appear to happen when the stats are > set in the first place. > We're solving this on the Impala end by storing types in the metastore as all > lower case, but Hive's behavior here is very confusing. It should either > always be case sensitive, so that you can't create column stats with types > that Hive considers invalid, or it should never be case sensitive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
