[jira] [Commented] (HIVE-12728) Apply DDL restrictions for ORC schema evolution
[
https://issues.apache.org/jira/browse/HIVE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15071117#comment-15071117
]
Hive QA commented on HIVE-12728:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12779373/HIVE-12728.3.patch
{color:green}SUCCESS:{color} +1 due to 16 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 9989 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6470/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6470/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6470/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12779373 - PreCommit-HIVE-TRUNK-Build
> Apply DDL restrictions for ORC schema evolution
> ---
>
> Key: HIVE-12728
> URL: https://issues.apache.org/jira/browse/HIVE-12728
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-12728.1.patch, HIVE-12728.2.patch,
> HIVE-12728.3.patch
>
>
> HIVE-11981 added schema evolution for ORC. However, it does not enforce any
> restrictions in DDL that can break schema evolution. Following changes have
> to be enforced in DDL to support the assumptions in schema evolution (that
> columns will only be added).
> 1) Restrict changing the file format of the table
> 2) Restrict changing the serde of the table
> 3) Restrict replacing columns to not drop columns or do unsupported type
> widening
> 4) Restrict reordering columns
> 5) Restrict unsupported type promotions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12728) Apply DDL restrictions for ORC schema evolution
[ https://issues.apache.org/jira/browse/HIVE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070547#comment-15070547 ] Prasanth Jayachandran commented on HIVE-12728: -- Fixes test failures. > Apply DDL restrictions for ORC schema evolution > --- > > Key: HIVE-12728 > URL: https://issues.apache.org/jira/browse/HIVE-12728 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0, 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-12728.1.patch, HIVE-12728.2.patch, > HIVE-12728.3.patch > > > HIVE-11981 added schema evolution for ORC. However, it does not enforce any > restrictions in DDL that can break schema evolution. Following changes have > to be enforced in DDL to support the assumptions in schema evolution (that > columns will only be added). > 1) Restrict changing the file format of the table > 2) Restrict changing the serde of the table > 3) Restrict replacing columns to not drop columns or do unsupported type > widening > 4) Restrict reordering columns > 5) Restrict unsupported type promotions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12728) Apply DDL restrictions for ORC schema evolution
[
https://issues.apache.org/jira/browse/HIVE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069372#comment-15069372
]
Hive QA commented on HIVE-12728:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12779122/HIVE-12728.2.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 29 failed/errored test(s), 9987 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_date_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dbtxnmgr_ddl1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_orc_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_analyze
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_int_type_promotion
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_temp_table_options1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_varchar_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_analyze
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6453/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6453/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6453/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 29 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12779122 - PreCommit-HIVE-TRUNK-Build
> Apply DDL restrictions for ORC schema evolution
> ---
>
> Key: HIVE-12728
> URL: https://issues.apache.org/jira/browse/HIVE-12728
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-12728.1.patch, HIVE-12728.2.patch
>
>
> HIVE-11981 added schema evolution for ORC. However, it does not enforce any
> restrictions in DDL that can break schema evolution. Following changes have
> to be enforced in DDL to support the assumptions in schema evolution (that
> columns will only be added).
> 1) Restrict changing the file format of the table
> 2) Restrict changing the serde of the table
> 3) Restrict replacing columns to not drop columns or do unsupported type
> widening
> 4) Restrict reordering columns
> 5) Restrict unsupported type promotions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12728) Apply DDL restrictions for ORC schema evolution
[ https://issues.apache.org/jira/browse/HIVE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068705#comment-15068705 ] Prasanth Jayachandran commented on HIVE-12728: -- [~mmccline] and [~hagleitn] Thanks for the review! Addressed both of your review comments. > Apply DDL restrictions for ORC schema evolution > --- > > Key: HIVE-12728 > URL: https://issues.apache.org/jira/browse/HIVE-12728 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0, 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-12728.1.patch, HIVE-12728.2.patch > > > HIVE-11981 added schema evolution for ORC. However, it does not enforce any > restrictions in DDL that can break schema evolution. Following changes have > to be enforced in DDL to support the assumptions in schema evolution (that > columns will only be added). > 1) Restrict changing the file format of the table > 2) Restrict changing the serde of the table > 3) Restrict replacing columns to not drop columns or do unsupported type > widening > 4) Restrict reordering columns > 5) Restrict unsupported type promotions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12728) Apply DDL restrictions for ORC schema evolution
[ https://issues.apache.org/jira/browse/HIVE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068484#comment-15068484 ] Gunther Hagleitner commented on HIVE-12728: --- Comments on rb. > Apply DDL restrictions for ORC schema evolution > --- > > Key: HIVE-12728 > URL: https://issues.apache.org/jira/browse/HIVE-12728 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0, 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-12728.1.patch > > > HIVE-11981 added schema evolution for ORC. However, it does not enforce any > restrictions in DDL that can break schema evolution. Following changes have > to be enforced in DDL to support the assumptions in schema evolution (that > columns will only be added). > 1) Restrict changing the file format of the table > 2) Restrict changing the serde of the table > 3) Restrict replacing columns > 4) Restrict reordering columns > 5) Restrict unsupported type promotions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12728) Apply DDL restrictions for ORC schema evolution
[ https://issues.apache.org/jira/browse/HIVE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067738#comment-15067738 ] Prasanth Jayachandran commented on HIVE-12728: -- [~mmccline] Could you please review the patch? > Apply DDL restrictions for ORC schema evolution > --- > > Key: HIVE-12728 > URL: https://issues.apache.org/jira/browse/HIVE-12728 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0, 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-12728.1.patch > > > HIVE-11981 added schema evolution for ORC. However, it does not enforce any > restrictions in DDL that can break schema evolution. Following changes have > to be enforced in DDL to support the assumptions in schema evolution (that > columns will only be added). > 1) Restrict changing the file format of the table > 2) Restrict changing the serde of the table > 3) Restrict replacing columns > 4) Restrict reordering columns > 5) Restrict unsupported type promotions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12728) Apply DDL restrictions for ORC schema evolution
[ https://issues.apache.org/jira/browse/HIVE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068065#comment-15068065 ] Matt McCline commented on HIVE-12728: - Added 2 review board comments -- otherwise LGTM +1 pending tests. Thanks for making the change. > Apply DDL restrictions for ORC schema evolution > --- > > Key: HIVE-12728 > URL: https://issues.apache.org/jira/browse/HIVE-12728 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0, 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-12728.1.patch > > > HIVE-11981 added schema evolution for ORC. However, it does not enforce any > restrictions in DDL that can break schema evolution. Following changes have > to be enforced in DDL to support the assumptions in schema evolution (that > columns will only be added). > 1) Restrict changing the file format of the table > 2) Restrict changing the serde of the table > 3) Restrict replacing columns > 4) Restrict reordering columns > 5) Restrict unsupported type promotions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
