[jira] [Commented] (HIVE-10591) Support limited integer type promotion in ORC
[
https://issues.apache.org/jira/browse/HIVE-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537049#comment-14537049
]
Lefty Leverenz commented on HIVE-10591:
---
Should this be documented?
{quote}
Following type promotions can be supported without any casting
smallint - int
smallint - bigint
int - bigint
{quote}
Support limited integer type promotion in ORC
-
Key: HIVE-10591
URL: https://issues.apache.org/jira/browse/HIVE-10591
Project: Hive
Issue Type: New Feature
Affects Versions: 1.3.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran
Fix For: 1.2.0, 1.3.0
Attachments: HIVE-10591.1.patch, HIVE-10591.2.patch,
HIVE-10591.2.patch, HIVE-10591.3.patch, HIVE-10591.3.patch, HIVE-10591.3.patch
ORC currently does not support schema-on-read. If we alter an ORC table with
'int' type to 'bigint' and if we query the altered table ClassCastException
will be thrown as the schema on read from table descriptor will expect
LongWritable whereas ORC will return IntWritable based on file schema stored
within ORC file. OrcSerde currently doesn't do any type conversions or type
promotions for performance reasons in inner loop. Since smallints, ints and
bigints are stored in the same way in ORC, it will be possible be allow such
type promotions without hurting performance. Following type promotions can be
supported without any casting
smallint - int
smallint - bigint
int - bigint
Tinyint promotion is not possible without casting as tinyints are stored
using RLE byte writer whereas smallints, ints and bigints are stored using
RLE integer writer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10591) Support limited integer type promotion in ORC
[ https://issues.apache.org/jira/browse/HIVE-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14536963#comment-14536963 ] Prasanth Jayachandran commented on HIVE-10591: -- The encryption_insert_partition_static.q test ran successfully when I ran it locally. Also the test diff shows file size differences which has nothing to do with this patch as the changes of this patch is read side only. Support limited integer type promotion in ORC - Key: HIVE-10591 URL: https://issues.apache.org/jira/browse/HIVE-10591 Project: Hive Issue Type: New Feature Affects Versions: 1.3.0 Reporter: Prasanth Jayachandran Assignee: Prasanth Jayachandran Fix For: 1.3.0 Attachments: HIVE-10591.1.patch, HIVE-10591.2.patch, HIVE-10591.2.patch, HIVE-10591.3.patch, HIVE-10591.3.patch, HIVE-10591.3.patch ORC currently does not support schema-on-read. If we alter an ORC table with 'int' type to 'bigint' and if we query the altered table ClassCastException will be thrown as the schema on read from table descriptor will expect LongWritable whereas ORC will return IntWritable based on file schema stored within ORC file. OrcSerde currently doesn't do any type conversions or type promotions for performance reasons in inner loop. Since smallints, ints and bigints are stored in the same way in ORC, it will be possible be allow such type promotions without hurting performance. Following type promotions can be supported without any casting smallint - int smallint - bigint int - bigint Tinyint promotion is not possible without casting as tinyints are stored using RLE byte writer whereas smallints, ints and bigints are stored using RLE integer writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10591) Support limited integer type promotion in ORC
[ https://issues.apache.org/jira/browse/HIVE-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14536979#comment-14536979 ] Prasanth Jayachandran commented on HIVE-10591: -- Committed to branch-1.2 as well. Support limited integer type promotion in ORC - Key: HIVE-10591 URL: https://issues.apache.org/jira/browse/HIVE-10591 Project: Hive Issue Type: New Feature Affects Versions: 1.3.0 Reporter: Prasanth Jayachandran Assignee: Prasanth Jayachandran Fix For: 1.2.0, 1.3.0 Attachments: HIVE-10591.1.patch, HIVE-10591.2.patch, HIVE-10591.2.patch, HIVE-10591.3.patch, HIVE-10591.3.patch, HIVE-10591.3.patch ORC currently does not support schema-on-read. If we alter an ORC table with 'int' type to 'bigint' and if we query the altered table ClassCastException will be thrown as the schema on read from table descriptor will expect LongWritable whereas ORC will return IntWritable based on file schema stored within ORC file. OrcSerde currently doesn't do any type conversions or type promotions for performance reasons in inner loop. Since smallints, ints and bigints are stored in the same way in ORC, it will be possible be allow such type promotions without hurting performance. Following type promotions can be supported without any casting smallint - int smallint - bigint int - bigint Tinyint promotion is not possible without casting as tinyints are stored using RLE byte writer whereas smallints, ints and bigints are stored using RLE integer writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10591) Support limited integer type promotion in ORC
[
https://issues.apache.org/jira/browse/HIVE-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14536906#comment-14536906
]
Hive QA commented on HIVE-10591:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12731561/HIVE-10591.3.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 8921 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3831/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3831/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3831/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12731561 - PreCommit-HIVE-TRUNK-Build
Support limited integer type promotion in ORC
-
Key: HIVE-10591
URL: https://issues.apache.org/jira/browse/HIVE-10591
Project: Hive
Issue Type: New Feature
Affects Versions: 1.3.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran
Attachments: HIVE-10591.1.patch, HIVE-10591.2.patch,
HIVE-10591.2.patch, HIVE-10591.3.patch, HIVE-10591.3.patch, HIVE-10591.3.patch
ORC currently does not support schema-on-read. If we alter an ORC table with
'int' type to 'bigint' and if we query the altered table ClassCastException
will be thrown as the schema on read from table descriptor will expect
LongWritable whereas ORC will return IntWritable based on file schema stored
within ORC file. OrcSerde currently doesn't do any type conversions or type
promotions for performance reasons in inner loop. Since smallints, ints and
bigints are stored in the same way in ORC, it will be possible be allow such
type promotions without hurting performance. Following type promotions can be
supported without any casting
smallint - int
smallint - bigint
int - bigint
Tinyint promotion is not possible without casting as tinyints are stored
using RLE byte writer whereas smallints, ints and bigints are stored using
RLE integer writer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10591) Support limited integer type promotion in ORC
[ https://issues.apache.org/jira/browse/HIVE-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1457#comment-1457 ] Gunther Hagleitner commented on HIVE-10591: --- +1 Support limited integer type promotion in ORC - Key: HIVE-10591 URL: https://issues.apache.org/jira/browse/HIVE-10591 Project: Hive Issue Type: New Feature Affects Versions: 1.3.0 Reporter: Prasanth Jayachandran Assignee: Prasanth Jayachandran Attachments: HIVE-10591.1.patch, HIVE-10591.2.patch, HIVE-10591.2.patch, HIVE-10591.3.patch ORC currently does not support schema-on-read. If we alter an ORC table with 'int' type to 'bigint' and if we query the altered table ClassCastException will be thrown as the schema on read from table descriptor will expect LongWritable whereas ORC will return IntWritable based on file schema stored within ORC file. OrcSerde currently doesn't do any type conversions or type promotions for performance reasons in inner loop. Since smallints, ints and bigints are stored in the same way in ORC, it will be possible be allow such type promotions without hurting performance. Following type promotions can be supported without any casting smallint - int smallint - bigint int - bigint Tinyint promotion is not possible without casting as tinyints are stored using RLE byte writer whereas smallints, ints and bigints are stored using RLE integer writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10591) Support limited integer type promotion in ORC
[
https://issues.apache.org/jira/browse/HIVE-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14529491#comment-14529491
]
Hive QA commented on HIVE-10591:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12730595/HIVE-10591.2.patch
{color:red}ERROR:{color} -1 due to 53 failed/errored test(s), 8901 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_parts
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_all_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_tmp_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_no_match
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_update_delete
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_acid
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_all_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_tmp_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_where_no_match
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_where_non_partitioned
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_unencrypted_tbl
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_load_data_to_encrypted_tables
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_all_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_tmp_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_no_match
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_update_delete
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_tmp_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_no_match
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_non_partitioned
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_disallow_transform
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_droppartition
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_sba_drop_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_alterpart_loc
org.apache.hadoop.hive.ql.TestTxnCommands2.testBucketizedInputFormat
org.apache.hadoop.hive.ql.TestTxnCommands2.testDeleteIn
org.apache.hadoop.hive.ql.TestTxnCommands2.testUpdateMixedCase
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropView
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessing
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessingCustomSetWhitelistAppend
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.majorCompactAfterAbort
[jira] [Commented] (HIVE-10591) Support limited integer type promotion in ORC
[
https://issues.apache.org/jira/browse/HIVE-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14529200#comment-14529200
]
Hive QA commented on HIVE-10591:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12730389/HIVE-10591.2.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3738/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3738/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3738/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Tests exited with: ExecutionException: java.util.concurrent.ExecutionException:
java.lang.IllegalArgumentException: resource batch-exec.vm not found.
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12730389 - PreCommit-HIVE-TRUNK-Build
Support limited integer type promotion in ORC
-
Key: HIVE-10591
URL: https://issues.apache.org/jira/browse/HIVE-10591
Project: Hive
Issue Type: New Feature
Affects Versions: 1.3.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran
Attachments: HIVE-10591.1.patch, HIVE-10591.2.patch
ORC currently does not support schema-on-read. If we alter an ORC table with
'int' type to 'bigint' and if we query the altered table ClassCastException
will be thrown as the schema on read from table descriptor will expect
LongWritable whereas ORC will return IntWritable based on file schema stored
within ORC file. OrcSerde currently doesn't do any type conversions or type
promotions for performance reasons in inner loop. Since smallints, ints and
bigints are stored in the same way in ORC, it will be possible be allow such
type promotions without hurting performance. Following type promotions can be
supported without any casting
smallint - int
smallint - bigint
int - bigint
Tinyint promotion is not possible without casting as tinyints are stored
using RLE byte writer whereas smallints, ints and bigints are stored using
RLE integer writer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
