[jira] [Commented] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet
[ https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15933792#comment-15933792 ] Vaibhav Gumashta commented on HIVE-15082: - Removing target 1.2.2 and moving to 1.3.0. Please feel free to revert if you think this should go in 1.2.2 (or if this gets reviewed before RC for 1.2.2 is cut). > Hive-1.2 cannot read data from complex data types with TIMESTAMP column, > stored in Parquet > -- > > Key: HIVE-15082 > URL: https://issues.apache.org/jira/browse/HIVE-15082 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Blocker > Attachments: HIVE-15082.1-branch-1.2.patch, > HIVE-15082.1-branch-1.2.patch, HIVE-15082-branch-1.2.patch, > HIVE-15082-branch-1.patch > > > *STEP 1. Create test data* > {code:sql} > select * from dual; > {code} > *EXPECTED RESULT:* > {noformat} > Pretty_UnIQUe_StrinG > {noformat} > {code:sql} > create table test_parquet1(login timestamp) stored as parquet; > insert overwrite table test_parquet1 select from_unixtime(unix_timestamp()) > from dual; > select * from test_parquet1 limit 1; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp as result. > {noformat} > 2016-10-27 10:58:19 > {noformat} > *STEP 2. Store timestamp in array in parquet file* > {code:sql} > create table test_parquet2(x array) stored as parquet; > insert overwrite table test_parquet2 select array(login) from test_parquet1; > select * from test_parquet2; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp in brackets as result. > {noformat} > ["2016-10-27 10:58:19"] > {noformat} > *ACTUAL RESULT:* > {noformat} > ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with > exception java.io.IOException:parquet.io.ParquetDecodingException: Can not > read value at 0 in block -1 in file > hdfs:///user/hive/warehouse/test_parquet2/00_0 > java.io.IOException: parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > *ROOT-CAUSE:* > Incorrect initialization of {{metadata}} {{HashMap}} causes that it has > {{null}} value in enumeration > {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when > executing following line: > {code:java} > boolean skipConversion = > Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname)); > {code} > in element {{ETIMESTAMP_CONVERTER}}. > JVM throws NPE and parquet library can not read data from file and throws > {noformat} > java.io.IOException:parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > for its turn. > *SOLUTION:* > Perform initialization in separate method to skip overriding it with {{null}} > value in block of code > {code:java} > if (parent != null) { > setMetadata(parent.getMetadata()); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet
[ https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15931347#comment-15931347 ] Vaibhav Gumashta commented on HIVE-15082: - No extra failures with this patch: https://builds.apache.org/job/PreCommit-HIVE-Build/4224/#showFailuresLink, when compared with HIVE-15007: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_from_utc_timestamp org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_merge_multi_expressions org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_to_utc_timestamp org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_exchgpartition2lel org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_auto_smb_mapjoin_14 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_empty org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez1 org.apache.hive.minikdc.TestMiniHiveKdc.testLogin org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore I'll commit this to 1.2 shortly. > Hive-1.2 cannot read data from complex data types with TIMESTAMP column, > stored in Parquet > -- > > Key: HIVE-15082 > URL: https://issues.apache.org/jira/browse/HIVE-15082 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Blocker > Attachments: HIVE-15082.1-branch-1.2.patch, > HIVE-15082.1-branch-1.2.patch, HIVE-15082-branch-1.2.patch, > HIVE-15082-branch-1.patch > > > *STEP 1. Create test data* > {code:sql} > select * from dual; > {code} > *EXPECTED RESULT:* > {noformat} > Pretty_UnIQUe_StrinG > {noformat} > {code:sql} > create table test_parquet1(login timestamp) stored as parquet; > insert overwrite table test_parquet1 select from_unixtime(unix_timestamp()) > from dual; > select * from test_parquet1 limit 1; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp as result. > {noformat} > 2016-10-27 10:58:19 > {noformat} > *STEP 2. Store timestamp in array in parquet file* > {code:sql} > create table test_parquet2(x array) stored as parquet; > insert overwrite table test_parquet2 select array(login) from test_parquet1; > select * from test_parquet2; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp in brackets as result. > {noformat} > ["2016-10-27 10:58:19"] > {noformat} > *ACTUAL RESULT:* > {noformat} > ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with > exception java.io.IOException:parquet.io.ParquetDecodingException: Can not > read value at 0 in block -1 in file > hdfs:///user/hive/warehouse/test_parquet2/00_0 > java.io.IOException: parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > *ROOT-CAUSE:* > Incorrect initialization of {{metadata}} {{HashMap}} causes that it has > {{null}} value in enumeration > {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when > executing following line: > {code:java} > boolean skipConversion = > Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname)); > {code} > in element {{ETIMESTAMP_CONVERTER}}. > JVM throws NPE and parquet library can not read data from file and throws > {noformat} > java.io.IOException:parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > for its turn. > *SOLUTION:* > Perform initialization in separate method to skip overriding it with {{null}} > value in block of code > {code:java} > if (parent != null) { > setMetadata(parent.getMetadata()); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet
[ https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15930682#comment-15930682 ] Vaibhav Gumashta commented on HIVE-15082: - [~osayankin] Sorry for the delay on this. I've started working again on release planning for 1.2.2 and have submitted the patch again for QA run. > Hive-1.2 cannot read data from complex data types with TIMESTAMP column, > stored in Parquet > -- > > Key: HIVE-15082 > URL: https://issues.apache.org/jira/browse/HIVE-15082 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Blocker > Attachments: HIVE-15082.1-branch-1.2.patch, > HIVE-15082.1-branch-1.2.patch, HIVE-15082-branch-1.2.patch, > HIVE-15082-branch-1.patch > > > *STEP 1. Create test data* > {code:sql} > select * from dual; > {code} > *EXPECTED RESULT:* > {noformat} > Pretty_UnIQUe_StrinG > {noformat} > {code:sql} > create table test_parquet1(login timestamp) stored as parquet; > insert overwrite table test_parquet1 select from_unixtime(unix_timestamp()) > from dual; > select * from test_parquet1 limit 1; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp as result. > {noformat} > 2016-10-27 10:58:19 > {noformat} > *STEP 2. Store timestamp in array in parquet file* > {code:sql} > create table test_parquet2(x array) stored as parquet; > insert overwrite table test_parquet2 select array(login) from test_parquet1; > select * from test_parquet2; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp in brackets as result. > {noformat} > ["2016-10-27 10:58:19"] > {noformat} > *ACTUAL RESULT:* > {noformat} > ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with > exception java.io.IOException:parquet.io.ParquetDecodingException: Can not > read value at 0 in block -1 in file > hdfs:///user/hive/warehouse/test_parquet2/00_0 > java.io.IOException: parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > *ROOT-CAUSE:* > Incorrect initialization of {{metadata}} {{HashMap}} causes that it has > {{null}} value in enumeration > {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when > executing following line: > {code:java} > boolean skipConversion = > Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname)); > {code} > in element {{ETIMESTAMP_CONVERTER}}. > JVM throws NPE and parquet library can not read data from file and throws > {noformat} > java.io.IOException:parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > for its turn. > *SOLUTION:* > Perform initialization in separate method to skip overriding it with {{null}} > value in block of code > {code:java} > if (parent != null) { > setMetadata(parent.getMetadata()); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet
[ https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15658102#comment-15658102 ] Oleksiy Sayankin commented on HIVE-15082: - Yeah, great, [~leftylev]! You were absolutely right saying to add a patch number! Next is to understand why some tests are failed. To be continued... > Hive-1.2 cannot read data from complex data types with TIMESTAMP column, > stored in Parquet > -- > > Key: HIVE-15082 > URL: https://issues.apache.org/jira/browse/HIVE-15082 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Blocker > Attachments: HIVE-15082-branch-1.2.patch, HIVE-15082-branch-1.patch, > HIVE-15082.1-branch-1.2.patch > > > *STEP 1. Create test data* > {code:sql} > select * from dual; > {code} > *EXPECTED RESULT:* > {noformat} > Pretty_UnIQUe_StrinG > {noformat} > {code:sql} > create table test_parquet1(login timestamp) stored as parquet; > insert overwrite table test_parquet1 select from_unixtime(unix_timestamp()) > from dual; > select * from test_parquet1 limit 1; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp as result. > {noformat} > 2016-10-27 10:58:19 > {noformat} > *STEP 2. Store timestamp in array in parquet file* > {code:sql} > create table test_parquet2(x array) stored as parquet; > insert overwrite table test_parquet2 select array(login) from test_parquet1; > select * from test_parquet2; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp in brackets as result. > {noformat} > ["2016-10-27 10:58:19"] > {noformat} > *ACTUAL RESULT:* > {noformat} > ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with > exception java.io.IOException:parquet.io.ParquetDecodingException: Can not > read value at 0 in block -1 in file > hdfs:///user/hive/warehouse/test_parquet2/00_0 > java.io.IOException: parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > *ROOT-CAUSE:* > Incorrect initialization of {{metadata}} {{HashMap}} causes that it has > {{null}} value in enumeration > {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when > executing following line: > {code:java} > boolean skipConversion = > Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname)); > {code} > in element {{ETIMESTAMP_CONVERTER}}. > JVM throws NPE and parquet library can not read data from file and throws > {noformat} > java.io.IOException:parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > for its turn. > *SOLUTION:* > Perform initialization in separate method to skip overriding it with {{null}} > value in block of code > {code:java} > if (parent != null) { > setMetadata(parent.getMetadata()); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet
[ https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15657922#comment-15657922 ] Lefty Leverenz commented on HIVE-15082: --- Hooray, it worked! > Hive-1.2 cannot read data from complex data types with TIMESTAMP column, > stored in Parquet > -- > > Key: HIVE-15082 > URL: https://issues.apache.org/jira/browse/HIVE-15082 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Blocker > Attachments: HIVE-15082-branch-1.2.patch, HIVE-15082-branch-1.patch, > HIVE-15082.1-branch-1.2.patch > > > *STEP 1. Create test data* > {code:sql} > select * from dual; > {code} > *EXPECTED RESULT:* > {noformat} > Pretty_UnIQUe_StrinG > {noformat} > {code:sql} > create table test_parquet1(login timestamp) stored as parquet; > insert overwrite table test_parquet1 select from_unixtime(unix_timestamp()) > from dual; > select * from test_parquet1 limit 1; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp as result. > {noformat} > 2016-10-27 10:58:19 > {noformat} > *STEP 2. Store timestamp in array in parquet file* > {code:sql} > create table test_parquet2(x array) stored as parquet; > insert overwrite table test_parquet2 select array(login) from test_parquet1; > select * from test_parquet2; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp in brackets as result. > {noformat} > ["2016-10-27 10:58:19"] > {noformat} > *ACTUAL RESULT:* > {noformat} > ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with > exception java.io.IOException:parquet.io.ParquetDecodingException: Can not > read value at 0 in block -1 in file > hdfs:///user/hive/warehouse/test_parquet2/00_0 > java.io.IOException: parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > *ROOT-CAUSE:* > Incorrect initialization of {{metadata}} {{HashMap}} causes that it has > {{null}} value in enumeration > {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when > executing following line: > {code:java} > boolean skipConversion = > Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname)); > {code} > in element {{ETIMESTAMP_CONVERTER}}. > JVM throws NPE and parquet library can not read data from file and throws > {noformat} > java.io.IOException:parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > for its turn. > *SOLUTION:* > Perform initialization in separate method to skip overriding it with {{null}} > value in block of code > {code:java} > if (parent != null) { > setMetadata(parent.getMetadata()); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet
[ https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15657314#comment-15657314 ] Hive QA commented on HIVE-15082: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12838537/HIVE-15082.1-branch-1.2.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 125 failed/errored test(s), 7910 tests executed *Failed tests:* {noformat} TestAdminUser - did not produce a TEST-*.xml file (likely timed out) (batchId=340) TestAuthorizationPreEventListener - did not produce a TEST-*.xml file (likely timed out) (batchId=371) TestAuthzApiEmbedAuthorizerInEmbed - did not produce a TEST-*.xml file (likely timed out) (batchId=350) TestAuthzApiEmbedAuthorizerInRemote - did not produce a TEST-*.xml file (likely timed out) (batchId=356) TestBeeLineWithArgs - did not produce a TEST-*.xml file (likely timed out) (batchId=378) TestCLIAuthzSessionContext - did not produce a TEST-*.xml file (likely timed out) (batchId=394) TestClientSideAuthorizationProvider - did not produce a TEST-*.xml file (likely timed out) (batchId=370) TestCompactor - did not produce a TEST-*.xml file (likely timed out) (batchId=360) TestCreateUdfEntities - did not produce a TEST-*.xml file (likely timed out) (batchId=359) TestCustomAuthentication - did not produce a TEST-*.xml file (likely timed out) (batchId=379) TestDBTokenStore - did not produce a TEST-*.xml file (likely timed out) (batchId=325) TestDDLWithRemoteMetastoreSecondNamenode - did not produce a TEST-*.xml file (likely timed out) (batchId=358) TestDynamicSerDe - did not produce a TEST-*.xml file (likely timed out) (batchId=328) TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed out) (batchId=337) TestEmbeddedThriftBinaryCLIService - did not produce a TEST-*.xml file (likely timed out) (batchId=382) TestFilterHooks - did not produce a TEST-*.xml file (likely timed out) (batchId=332) TestFolderPermissions - did not produce a TEST-*.xml file (likely timed out) (batchId=365) TestHS2AuthzContext - did not produce a TEST-*.xml file (likely timed out) (batchId=397) TestHS2AuthzSessionContext - did not produce a TEST-*.xml file (likely timed out) (batchId=398) TestHS2ImpersonationWithRemoteMS - did not produce a TEST-*.xml file (likely timed out) (batchId=386) TestHiveAuthorizerCheckInvocation - did not produce a TEST-*.xml file (likely timed out) (batchId=374) TestHiveAuthorizerShowFilters - did not produce a TEST-*.xml file (likely timed out) (batchId=373) TestHiveHistory - did not produce a TEST-*.xml file (likely timed out) (batchId=376) TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out) (batchId=352) TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file (likely timed out) (batchId=342) TestHiveMetaTool - did not produce a TEST-*.xml file (likely timed out) (batchId=355) TestHiveServer2 - did not produce a TEST-*.xml file (likely timed out) (batchId=400) TestHiveServer2SessionTimeout - did not produce a TEST-*.xml file (likely timed out) (batchId=401) TestHiveSessionImpl - did not produce a TEST-*.xml file (likely timed out) (batchId=383) TestHs2Hooks - did not produce a TEST-*.xml file (likely timed out) (batchId=357) TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out) (batchId=388) TestJdbcMetadataApiAuth - did not produce a TEST-*.xml file (likely timed out) (batchId=399) TestJdbcWithLocalClusterSpark - did not produce a TEST-*.xml file (likely timed out) (batchId=393) TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file (likely timed out) (batchId=390) TestJdbcWithMiniMr - did not produce a TEST-*.xml file (likely timed out) (batchId=389) TestJdbcWithSQLAuthUDFBlacklist - did not produce a TEST-*.xml file (likely timed out) (batchId=395) TestJdbcWithSQLAuthorization - did not produce a TEST-*.xml file (likely timed out) (batchId=396) TestLocationQueries - did not produce a TEST-*.xml file (likely timed out) (batchId=363) TestMTQueries - did not produce a TEST-*.xml file (likely timed out) (batchId=361) TestMarkPartition - did not produce a TEST-*.xml file (likely timed out) (batchId=349) TestMarkPartitionRemote - did not produce a TEST-*.xml file (likely timed out) (batchId=353) TestMetaStoreAuthorization - did not produce a TEST-*.xml file (likely timed out) (batchId=338) TestMetaStoreConnectionUrlHook - did not produce a TEST-*.xml file (likely timed out) (batchId=336) TestMetaStoreEndFunctionListener - did not produce a TEST-*.xml file (likely timed out) (batchId=335) TestMetaStoreEventListener - did not produce a TEST-*.xml file (likely timed out) (batchId=345) TestMetaStoreEventListenerOnlyOnCommit - did not produce a TEST-*.xml file (likely timed out) (batchId=348) TestMetaStoreInitListener - did not produce a TEST
[jira] [Commented] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet
[ https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656755#comment-15656755 ] Oleksiy Sayankin commented on HIVE-15082: - I added to file name and yes, I canceled patch and resubmitted it again each time I attached new patch to the issue. > Hive-1.2 cannot read data from complex data types with TIMESTAMP column, > stored in Parquet > -- > > Key: HIVE-15082 > URL: https://issues.apache.org/jira/browse/HIVE-15082 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Blocker > Attachments: HIVE-15082-branch-1.2.patch, HIVE-15082-branch-1.patch, > HIVE-15082.1-branch-1.2.patch > > > *STEP 1. Create test data* > {code:sql} > select * from dual; > {code} > *EXPECTED RESULT:* > {noformat} > Pretty_UnIQUe_StrinG > {noformat} > {code:sql} > create table test_parquet1(login timestamp) stored as parquet; > insert overwrite table test_parquet1 select from_unixtime(unix_timestamp()) > from dual; > select * from test_parquet1 limit 1; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp as result. > {noformat} > 2016-10-27 10:58:19 > {noformat} > *STEP 2. Store timestamp in array in parquet file* > {code:sql} > create table test_parquet2(x array) stored as parquet; > insert overwrite table test_parquet2 select array(login) from test_parquet1; > select * from test_parquet2; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp in brackets as result. > {noformat} > ["2016-10-27 10:58:19"] > {noformat} > *ACTUAL RESULT:* > {noformat} > ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with > exception java.io.IOException:parquet.io.ParquetDecodingException: Can not > read value at 0 in block -1 in file > hdfs:///user/hive/warehouse/test_parquet2/00_0 > java.io.IOException: parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > *ROOT-CAUSE:* > Incorrect initialization of {{metadata}} {{HashMap}} causes that it has > {{null}} value in enumeration > {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when > executing following line: > {code:java} > boolean skipConversion = > Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname)); > {code} > in element {{ETIMESTAMP_CONVERTER}}. > JVM throws NPE and parquet library can not read data from file and throws > {noformat} > java.io.IOException:parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > for its turn. > *SOLUTION:* > Perform initialization in separate method to skip overriding it with {{null}} > value in block of code > {code:java} > if (parent != null) { > setMetadata(parent.getMetadata()); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet
[ https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655093#comment-15655093 ] Lefty Leverenz commented on HIVE-15082: --- Okay, then your syntax looks correct although you might need a patch-num because it isn't the first patch submitted, even though the first patch is for a different branch. Also, did you cancel the patch and resubmit it with the button at the top of the page? > Hive-1.2 cannot read data from complex data types with TIMESTAMP column, > stored in Parquet > -- > > Key: HIVE-15082 > URL: https://issues.apache.org/jira/browse/HIVE-15082 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Blocker > Attachments: HIVE-15082-branch-1.2.patch, HIVE-15082-branch-1.patch > > > *STEP 1. Create test data* > {code:sql} > select * from dual; > {code} > *EXPECTED RESULT:* > {noformat} > Pretty_UnIQUe_StrinG > {noformat} > {code:sql} > create table test_parquet1(login timestamp) stored as parquet; > insert overwrite table test_parquet1 select from_unixtime(unix_timestamp()) > from dual; > select * from test_parquet1 limit 1; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp as result. > {noformat} > 2016-10-27 10:58:19 > {noformat} > *STEP 2. Store timestamp in array in parquet file* > {code:sql} > create table test_parquet2(x array) stored as parquet; > insert overwrite table test_parquet2 select array(login) from test_parquet1; > select * from test_parquet2; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp in brackets as result. > {noformat} > ["2016-10-27 10:58:19"] > {noformat} > *ACTUAL RESULT:* > {noformat} > ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with > exception java.io.IOException:parquet.io.ParquetDecodingException: Can not > read value at 0 in block -1 in file > hdfs:///user/hive/warehouse/test_parquet2/00_0 > java.io.IOException: parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > *ROOT-CAUSE:* > Incorrect initialization of {{metadata}} {{HashMap}} causes that it has > {{null}} value in enumeration > {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when > executing following line: > {code:java} > boolean skipConversion = > Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname)); > {code} > in element {{ETIMESTAMP_CONVERTER}}. > JVM throws NPE and parquet library can not read data from file and throws > {noformat} > java.io.IOException:parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > for its turn. > *SOLUTION:* > Perform initialization in separate method to skip overriding it with {{null}} > value in block of code > {code:java} > if (parent != null) { > setMetadata(parent.getMetadata()); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet
[ https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654301#comment-15654301 ] Oleksiy Sayankin commented on HIVE-15082: - Thanks for helping [~leftylev]. I have found an issue https://issues.apache.org/jira/browse/HIVE-15007. Please see the list of patches attached there. You can see that they used the name HIVE-15007-branch-1.2.patch and so did I in my case. I do not know why, but their patch was applied. According to this pattern: HIVE-[.][-].patch I have filled the gaps: * is 15082 * is empty in my case * is branch-1.2 The pattern is from here: https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-CreatingaPatch > Hive-1.2 cannot read data from complex data types with TIMESTAMP column, > stored in Parquet > -- > > Key: HIVE-15082 > URL: https://issues.apache.org/jira/browse/HIVE-15082 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Blocker > Attachments: HIVE-15082-branch-1.2.patch, HIVE-15082-branch-1.patch > > > *STEP 1. Create test data* > {code:sql} > select * from dual; > {code} > *EXPECTED RESULT:* > {noformat} > Pretty_UnIQUe_StrinG > {noformat} > {code:sql} > create table test_parquet1(login timestamp) stored as parquet; > insert overwrite table test_parquet1 select from_unixtime(unix_timestamp()) > from dual; > select * from test_parquet1 limit 1; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp as result. > {noformat} > 2016-10-27 10:58:19 > {noformat} > *STEP 2. Store timestamp in array in parquet file* > {code:sql} > create table test_parquet2(x array) stored as parquet; > insert overwrite table test_parquet2 select array(login) from test_parquet1; > select * from test_parquet2; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp in brackets as result. > {noformat} > ["2016-10-27 10:58:19"] > {noformat} > *ACTUAL RESULT:* > {noformat} > ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with > exception java.io.IOException:parquet.io.ParquetDecodingException: Can not > read value at 0 in block -1 in file > hdfs:///user/hive/warehouse/test_parquet2/00_0 > java.io.IOException: parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > *ROOT-CAUSE:* > Incorrect initialization of {{metadata}} {{HashMap}} causes that it has > {{null}} value in enumeration > {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when > executing following line: > {code:java} > boolean skipConversion = > Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname)); > {code} > in element {{ETIMESTAMP_CONVERTER}}. > JVM throws NPE and parquet library can not read data from file and throws > {noformat} > java.io.IOException:parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > for its turn. > *SOLUTION:* > Perform initialization in separate method to skip overriding it with {{null}} > value in block of code > {code:java} > if (parent != null) { > setMetadata(parent.getMetadata()); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet
[ https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653262#comment-15653262 ] Lefty Leverenz commented on HIVE-15082: --- [~osayankin], it looks like you swapped the locations of patch-num and branch-name on patch 2, unless you meant it to be for branch-1.2 (but it still needs a patch-num). > Hive-1.2 cannot read data from complex data types with TIMESTAMP column, > stored in Parquet > -- > > Key: HIVE-15082 > URL: https://issues.apache.org/jira/browse/HIVE-15082 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Blocker > Attachments: HIVE-15082-branch-1.2.patch, HIVE-15082-branch-1.patch > > > *STEP 1. Create test data* > {code:sql} > select * from dual; > {code} > *EXPECTED RESULT:* > {noformat} > Pretty_UnIQUe_StrinG > {noformat} > {code:sql} > create table test_parquet1(login timestamp) stored as parquet; > insert overwrite table test_parquet1 select from_unixtime(unix_timestamp()) > from dual; > select * from test_parquet1 limit 1; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp as result. > {noformat} > 2016-10-27 10:58:19 > {noformat} > *STEP 2. Store timestamp in array in parquet file* > {code:sql} > create table test_parquet2(x array) stored as parquet; > insert overwrite table test_parquet2 select array(login) from test_parquet1; > select * from test_parquet2; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp in brackets as result. > {noformat} > ["2016-10-27 10:58:19"] > {noformat} > *ACTUAL RESULT:* > {noformat} > ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with > exception java.io.IOException:parquet.io.ParquetDecodingException: Can not > read value at 0 in block -1 in file > hdfs:///user/hive/warehouse/test_parquet2/00_0 > java.io.IOException: parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > *ROOT-CAUSE:* > Incorrect initialization of {{metadata}} {{HashMap}} causes that it has > {{null}} value in enumeration > {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when > executing following line: > {code:java} > boolean skipConversion = > Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname)); > {code} > in element {{ETIMESTAMP_CONVERTER}}. > JVM throws NPE and parquet library can not read data from file and throws > {noformat} > java.io.IOException:parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > for its turn. > *SOLUTION:* > Perform initialization in separate method to skip overriding it with {{null}} > value in block of code > {code:java} > if (parent != null) { > setMetadata(parent.getMetadata()); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet
[ https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15643658#comment-15643658 ] Oleksiy Sayankin commented on HIVE-15082: - [~vgumashta], I have already attached the patch with name *HIVE-15082-branch-1.2.patch* according to pattern HIVE-[.][-].patch For some reason it gave no effect and patch is not applied. So if you know what is wrong here with patch name and how to fix it, please advise. > Hive-1.2 cannot read data from complex data types with TIMESTAMP column, > stored in Parquet > -- > > Key: HIVE-15082 > URL: https://issues.apache.org/jira/browse/HIVE-15082 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Blocker > Attachments: HIVE-15082-branch-1.2.patch, HIVE-15082-branch-1.patch > > > *STEP 1. Create test data* > {code:sql} > select * from dual; > {code} > *EXPECTED RESULT:* > {noformat} > Pretty_UnIQUe_StrinG > {noformat} > {code:sql} > create table test_parquet1(login timestamp) stored as parquet; > insert overwrite table test_parquet1 select from_unixtime(unix_timestamp()) > from dual; > select * from test_parquet1 limit 1; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp as result. > {noformat} > 2016-10-27 10:58:19 > {noformat} > *STEP 2. Store timestamp in array in parquet file* > {code:sql} > create table test_parquet2(x array) stored as parquet; > insert overwrite table test_parquet2 select array(login) from test_parquet1; > select * from test_parquet2; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp in brackets as result. > {noformat} > ["2016-10-27 10:58:19"] > {noformat} > *ACTUAL RESULT:* > {noformat} > ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with > exception java.io.IOException:parquet.io.ParquetDecodingException: Can not > read value at 0 in block -1 in file > hdfs:///user/hive/warehouse/test_parquet2/00_0 > java.io.IOException: parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > *ROOT-CAUSE:* > Incorrect initialization of {{metadata}} {{HashMap}} causes that it has > {{null}} value in enumeration > {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when > executing following line: > {code:java} > boolean skipConversion = > Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname)); > {code} > in element {{ETIMESTAMP_CONVERTER}}. > JVM throws NPE and parquet library can not read data from file and throws > {noformat} > java.io.IOException:parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > for its turn. > *SOLUTION:* > Perform initialization in separate method to skip overriding it with {{null}} > value in block of code > {code:java} > if (parent != null) { > setMetadata(parent.getMetadata()); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet
[ https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15631096#comment-15631096 ] Vaibhav Gumashta commented on HIVE-15082: - [~osayankin] Thanks for the patch. Can you submit this again for a QA run against branch-1.2 > Hive-1.2 cannot read data from complex data types with TIMESTAMP column, > stored in Parquet > -- > > Key: HIVE-15082 > URL: https://issues.apache.org/jira/browse/HIVE-15082 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin > Fix For: 1.2.2 > > Attachments: HIVE-15082-branch-1.2.patch, HIVE-15082-branch-1.patch > > > *STEP 1. Create test data* > {code:sql} > select * from dual; > {code} > *EXPECTED RESULT:* > {noformat} > Pretty_UnIQUe_StrinG > {noformat} > {code:sql} > create table test_parquet1(login timestamp) stored as parquet; > insert overwrite table test_parquet1 select from_unixtime(unix_timestamp()) > from dual; > select * from test_parquet1 limit 1; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp as result. > {noformat} > 2016-10-27 10:58:19 > {noformat} > *STEP 2. Store timestamp in array in parquet file* > {code:sql} > create table test_parquet2(x array) stored as parquet; > insert overwrite table test_parquet2 select array(login) from test_parquet1; > select * from test_parquet2; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp in brackets as result. > {noformat} > ["2016-10-27 10:58:19"] > {noformat} > *ACTUAL RESULT:* > {noformat} > ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with > exception java.io.IOException:parquet.io.ParquetDecodingException: Can not > read value at 0 in block -1 in file > hdfs:///user/hive/warehouse/test_parquet2/00_0 > java.io.IOException: parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > *ROOT-CAUSE:* > Incorrect initialization of {{metadata}} {{HashMap}} causes that it has > {{null}} value in enumeration > {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when > executing following line: > {code:java} > boolean skipConversion = > Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname)); > {code} > in element {{ETIMESTAMP_CONVERTER}}. > JVM throws NPE and parquet library can not read data from file and throws > {noformat} > java.io.IOException:parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > for its turn. > *SOLUTION:* > Perform initialization in separate method to skip overriding it with {{null}} > value in block of code > {code:java} > if (parent != null) { > setMetadata(parent.getMetadata()); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15082) Hive-1.2 cannot read data from complex data types with TIMESTAMP column, stored in Parquet
[ https://issues.apache.org/jira/browse/HIVE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15631068#comment-15631068 ] Vaibhav Gumashta commented on HIVE-15082: - Removing fix version 1.2.2 as this is not a blocker. > Hive-1.2 cannot read data from complex data types with TIMESTAMP column, > stored in Parquet > -- > > Key: HIVE-15082 > URL: https://issues.apache.org/jira/browse/HIVE-15082 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin > Fix For: 1.2.3 > > Attachments: HIVE-15082-branch-1.2.patch, HIVE-15082-branch-1.patch > > > *STEP 1. Create test data* > {code:sql} > select * from dual; > {code} > *EXPECTED RESULT:* > {noformat} > Pretty_UnIQUe_StrinG > {noformat} > {code:sql} > create table test_parquet1(login timestamp) stored as parquet; > insert overwrite table test_parquet1 select from_unixtime(unix_timestamp()) > from dual; > select * from test_parquet1 limit 1; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp as result. > {noformat} > 2016-10-27 10:58:19 > {noformat} > *STEP 2. Store timestamp in array in parquet file* > {code:sql} > create table test_parquet2(x array) stored as parquet; > insert overwrite table test_parquet2 select array(login) from test_parquet1; > select * from test_parquet2; > {code} > *EXPECTED RESULT:* > No exceptions. Current timestamp in brackets as result. > {noformat} > ["2016-10-27 10:58:19"] > {noformat} > *ACTUAL RESULT:* > {noformat} > ERROR [main]: CliDriver (SessionState.java:printError(963)) - Failed with > exception java.io.IOException:parquet.io.ParquetDecodingException: Can not > read value at 0 in block -1 in file > hdfs:///user/hive/warehouse/test_parquet2/00_0 > java.io.IOException: parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > *ROOT-CAUSE:* > Incorrect initialization of {{metadata}} {{HashMap}} causes that it has > {{null}} value in enumeration > {{org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter}} when > executing following line: > {code:java} > boolean skipConversion = > Boolean.valueOf(metadata.get(HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION.varname)); > {code} > in element {{ETIMESTAMP_CONVERTER}}. > JVM throws NPE and parquet library can not read data from file and throws > {noformat} > java.io.IOException:parquet.io.ParquetDecodingException: Can not read value > at 0 in block -1 in file hdfs:///user/hive/warehouse/test_parquet2/00_0 > {noformat} > for its turn. > *SOLUTION:* > Perform initialization in separate method to skip overriding it with {{null}} > value in block of code > {code:java} > if (parent != null) { > setMetadata(parent.getMetadata()); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)