[jira] [Commented] (HIVE-10591) Support limited integer type promotion in ORC
[
https://issues.apache.org/jira/browse/HIVE-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14537049#comment-14537049
]
Lefty Leverenz commented on HIVE-10591:
---
Should this be documented?
{quote}
Following type promotions can be supported without any casting
smallint -> int
smallint -> bigint
int -> bigint
{quote}
> Support limited integer type promotion in ORC
> -
>
> Key: HIVE-10591
> URL: https://issues.apache.org/jira/browse/HIVE-10591
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 1.3.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Fix For: 1.2.0, 1.3.0
>
> Attachments: HIVE-10591.1.patch, HIVE-10591.2.patch,
> HIVE-10591.2.patch, HIVE-10591.3.patch, HIVE-10591.3.patch, HIVE-10591.3.patch
>
>
> ORC currently does not support schema-on-read. If we alter an ORC table with
> 'int' type to 'bigint' and if we query the altered table ClassCastException
> will be thrown as the schema on read from table descriptor will expect
> LongWritable whereas ORC will return IntWritable based on file schema stored
> within ORC file. OrcSerde currently doesn't do any type conversions or type
> promotions for performance reasons in inner loop. Since smallints, ints and
> bigints are stored in the same way in ORC, it will be possible be allow such
> type promotions without hurting performance. Following type promotions can be
> supported without any casting
> smallint -> int
> smallint -> bigint
> int -> bigint
> Tinyint promotion is not possible without casting as tinyints are stored
> using RLE byte writer whereas smallints, ints and bigints are stored using
> RLE integer writer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10591) Support limited integer type promotion in ORC
[ https://issues.apache.org/jira/browse/HIVE-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14536979#comment-14536979 ] Prasanth Jayachandran commented on HIVE-10591: -- Committed to branch-1.2 as well. > Support limited integer type promotion in ORC > - > > Key: HIVE-10591 > URL: https://issues.apache.org/jira/browse/HIVE-10591 > Project: Hive > Issue Type: New Feature >Affects Versions: 1.3.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 1.2.0, 1.3.0 > > Attachments: HIVE-10591.1.patch, HIVE-10591.2.patch, > HIVE-10591.2.patch, HIVE-10591.3.patch, HIVE-10591.3.patch, HIVE-10591.3.patch > > > ORC currently does not support schema-on-read. If we alter an ORC table with > 'int' type to 'bigint' and if we query the altered table ClassCastException > will be thrown as the schema on read from table descriptor will expect > LongWritable whereas ORC will return IntWritable based on file schema stored > within ORC file. OrcSerde currently doesn't do any type conversions or type > promotions for performance reasons in inner loop. Since smallints, ints and > bigints are stored in the same way in ORC, it will be possible be allow such > type promotions without hurting performance. Following type promotions can be > supported without any casting > smallint -> int > smallint -> bigint > int -> bigint > Tinyint promotion is not possible without casting as tinyints are stored > using RLE byte writer whereas smallints, ints and bigints are stored using > RLE integer writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10591) Support limited integer type promotion in ORC
[ https://issues.apache.org/jira/browse/HIVE-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14536963#comment-14536963 ] Prasanth Jayachandran commented on HIVE-10591: -- The encryption_insert_partition_static.q test ran successfully when I ran it locally. Also the test diff shows file size differences which has nothing to do with this patch as the changes of this patch is read side only. > Support limited integer type promotion in ORC > - > > Key: HIVE-10591 > URL: https://issues.apache.org/jira/browse/HIVE-10591 > Project: Hive > Issue Type: New Feature >Affects Versions: 1.3.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 1.3.0 > > Attachments: HIVE-10591.1.patch, HIVE-10591.2.patch, > HIVE-10591.2.patch, HIVE-10591.3.patch, HIVE-10591.3.patch, HIVE-10591.3.patch > > > ORC currently does not support schema-on-read. If we alter an ORC table with > 'int' type to 'bigint' and if we query the altered table ClassCastException > will be thrown as the schema on read from table descriptor will expect > LongWritable whereas ORC will return IntWritable based on file schema stored > within ORC file. OrcSerde currently doesn't do any type conversions or type > promotions for performance reasons in inner loop. Since smallints, ints and > bigints are stored in the same way in ORC, it will be possible be allow such > type promotions without hurting performance. Following type promotions can be > supported without any casting > smallint -> int > smallint -> bigint > int -> bigint > Tinyint promotion is not possible without casting as tinyints are stored > using RLE byte writer whereas smallints, ints and bigints are stored using > RLE integer writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10591) Support limited integer type promotion in ORC
[
https://issues.apache.org/jira/browse/HIVE-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14536906#comment-14536906
]
Hive QA commented on HIVE-10591:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12731561/HIVE-10591.3.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 8921 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3831/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3831/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3831/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12731561 - PreCommit-HIVE-TRUNK-Build
> Support limited integer type promotion in ORC
> -
>
> Key: HIVE-10591
> URL: https://issues.apache.org/jira/browse/HIVE-10591
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 1.3.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-10591.1.patch, HIVE-10591.2.patch,
> HIVE-10591.2.patch, HIVE-10591.3.patch, HIVE-10591.3.patch, HIVE-10591.3.patch
>
>
> ORC currently does not support schema-on-read. If we alter an ORC table with
> 'int' type to 'bigint' and if we query the altered table ClassCastException
> will be thrown as the schema on read from table descriptor will expect
> LongWritable whereas ORC will return IntWritable based on file schema stored
> within ORC file. OrcSerde currently doesn't do any type conversions or type
> promotions for performance reasons in inner loop. Since smallints, ints and
> bigints are stored in the same way in ORC, it will be possible be allow such
> type promotions without hurting performance. Following type promotions can be
> supported without any casting
> smallint -> int
> smallint -> bigint
> int -> bigint
> Tinyint promotion is not possible without casting as tinyints are stored
> using RLE byte writer whereas smallints, ints and bigints are stored using
> RLE integer writer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10591) Support limited integer type promotion in ORC
[ https://issues.apache.org/jira/browse/HIVE-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1457#comment-1457 ] Gunther Hagleitner commented on HIVE-10591: --- +1 > Support limited integer type promotion in ORC > - > > Key: HIVE-10591 > URL: https://issues.apache.org/jira/browse/HIVE-10591 > Project: Hive > Issue Type: New Feature >Affects Versions: 1.3.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-10591.1.patch, HIVE-10591.2.patch, > HIVE-10591.2.patch, HIVE-10591.3.patch > > > ORC currently does not support schema-on-read. If we alter an ORC table with > 'int' type to 'bigint' and if we query the altered table ClassCastException > will be thrown as the schema on read from table descriptor will expect > LongWritable whereas ORC will return IntWritable based on file schema stored > within ORC file. OrcSerde currently doesn't do any type conversions or type > promotions for performance reasons in inner loop. Since smallints, ints and > bigints are stored in the same way in ORC, it will be possible be allow such > type promotions without hurting performance. Following type promotions can be > supported without any casting > smallint -> int > smallint -> bigint > int -> bigint > Tinyint promotion is not possible without casting as tinyints are stored > using RLE byte writer whereas smallints, ints and bigints are stored using > RLE integer writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10591) Support limited integer type promotion in ORC
[
https://issues.apache.org/jira/browse/HIVE-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529491#comment-14529491
]
Hive QA commented on HIVE-10591:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12730595/HIVE-10591.2.patch
{color:red}ERROR:{color} -1 due to 53 failed/errored test(s), 8901 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_parts
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_all_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_tmp_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_no_match
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_update_delete
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_acid
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_all_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_tmp_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_where_no_match
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_where_non_partitioned
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_unencrypted_tbl
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_load_data_to_encrypted_tables
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_all_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_tmp_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_no_match
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_update_delete
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_tmp_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_no_match
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_non_partitioned
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_disallow_transform
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_droppartition
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_sba_drop_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_alterpart_loc
org.apache.hadoop.hive.ql.TestTxnCommands2.testBucketizedInputFormat
org.apache.hadoop.hive.ql.TestTxnCommands2.testDeleteIn
org.apache.hadoop.hive.ql.TestTxnCommands2.testUpdateMixedCase
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropView
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessing
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessingCustomSetWhitelistAppend
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.majorCompactAfterAbort
org.apache.hadoop.hive.ql.txn.compa
[jira] [Commented] (HIVE-10591) Support limited integer type promotion in ORC
[
https://issues.apache.org/jira/browse/HIVE-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529200#comment-14529200
]
Hive QA commented on HIVE-10591:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12730389/HIVE-10591.2.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3738/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3738/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3738/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Tests exited with: ExecutionException: java.util.concurrent.ExecutionException:
java.lang.IllegalArgumentException: resource batch-exec.vm not found.
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12730389 - PreCommit-HIVE-TRUNK-Build
> Support limited integer type promotion in ORC
> -
>
> Key: HIVE-10591
> URL: https://issues.apache.org/jira/browse/HIVE-10591
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 1.3.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-10591.1.patch, HIVE-10591.2.patch
>
>
> ORC currently does not support schema-on-read. If we alter an ORC table with
> 'int' type to 'bigint' and if we query the altered table ClassCastException
> will be thrown as the schema on read from table descriptor will expect
> LongWritable whereas ORC will return IntWritable based on file schema stored
> within ORC file. OrcSerde currently doesn't do any type conversions or type
> promotions for performance reasons in inner loop. Since smallints, ints and
> bigints are stored in the same way in ORC, it will be possible be allow such
> type promotions without hurting performance. Following type promotions can be
> supported without any casting
> smallint -> int
> smallint -> bigint
> int -> bigint
> Tinyint promotion is not possible without casting as tinyints are stored
> using RLE byte writer whereas smallints, ints and bigints are stored using
> RLE integer writer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
