[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419908#comment-16419908 ] Vihang Karajgaonkar commented on HIVE-15995: hmm.. this was unexpected. I think we can address this in a separate JIRA. Technically it looks like a different problem. Can you create a different JIRA for that? Rest of the patch looks good to me. +1 > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, > HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.5.patch, HIVE-15995.patch, > cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = > '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table > Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415780#comment-16415780 ] Adam Szita commented on HIVE-15995: --- [~vihangk1] good find! I tested this manually by looking into the HMS DB after every step.. It looks like the describe command is always returning the table's schema, not the partition's. This is due to [Partition.getColsInternal()|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L491] is skipping the retrieval of columns from the tPartition object, because the {{hasMetastoreBasedSchema}} returns false for an Avro table... So although the expected results are there in HMS but describe won't show us. Any ideas to overcome this? > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, > HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.5.patch, HIVE-15995.patch, > cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = > '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table > Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414623#comment-16414623 ] Vihang Karajgaonkar commented on HIVE-15995: HI [~szita] Thanks for making the changes. Is the q.out file correct? Eg: I see the following sequence of statements in the qfile {noformat} --case: partial partition spec 86 ALTER TABLE avro_extschema_url_parted SET 87 TBLPROPERTIES ('avro.schema.url'='${system:test.tmp.dir}/grad2.avsc'); 88 ALTER TABLE avro_extschema_url_parted PARTITION (p1=2018) UPDATE COLUMNS; 89 ALTER TABLE avro_extschema_url_parted UNSET TBLPROPERTIES ('avro.schema.url'); 90 91 DESCRIBE avro_extschema_url_parted; 92 DESCRIBE avro_extschema_url_parted PARTITION (p1=2017, p2=11); 93 DESCRIBE avro_extschema_url_parted PARTITION (p1=2018, p2=2); 94 DESCRIBE avro_extschema_url_parted PARTITION (p1=2018, p2=3); {noformat} Shouldn't the describe command return schema based on grad2.avsc for (p1=2018, p2=2) and (p1=2018, p2=3) case? Am I misunderstanding something? Sorry for the back and forth but just wanted to confirm if the q.out is the expected behavior > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, > HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.5.patch, HIVE-15995.patch, > cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = > '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table > Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413640#comment-16413640 ] Adam Szita commented on HIVE-15995: --- As I see it the failed tests in above report are unrelated. > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, > HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.5.patch, HIVE-15995.patch, > cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = > '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table > Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412824#comment-16412824 ] Hive QA commented on HIVE-15995: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12915705/HIVE-15995.5.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 31 failed/errored test(s), 13425 tests executed *Failed tests:* {noformat} TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=92) [infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q] TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=95) [udf_invalid.q,authorization_uri_export.q,default_constraint_complex_default_value.q,druid_datasource2.q,view_update.q,default_partition_name.q,authorization_public_create.q,load_wrong_fileformat_rc_seq.q,default_constraint_invalid_type.q,altern1.q,describe_xpath1.q,drop_view_failure2.q,temp_table_rename.q,invalid_select_column_with_subquery.q,udf_trunc_error1.q,insert_view_failure.q,dbtxnmgr_nodbunlock.q,authorization_show_columns.q,cte_recursion.q,merge_constraint_notnull.q,load_part_nospec.q,clusterbyorderby.q,orc_type_promotion2.q,ctas_noperm_loc.q,udf_instr_wrong_args_len.q,invalid_create_tbl2.q,part_col_complex_type.q,authorization_drop_db_empty.q,smb_mapjoin_14.q,subquery_scalar_multi_rows.q,alter_partition_coltype_2columns.q,subquery_corr_in_agg.q,insert_overwrite_notnull_constraint.q,authorization_show_grant_otheruser_wtab.q,regex_col_groupby.q,udaf_collect_set_unsupported.q,ptf_negative_DuplicateWindowAlias.q,exim_22_export_authfail.q,udf_likeany_wrong1.q,groupby_key.q,ambiguous_col.q,groupby3_multi_distinct.q,authorization_alter_drop_ptn.q,invalid_cast_from_binary_5.q,show_create_table_does_not_exist.q,invalid_select_column.q,exim_20_managed_location_over_existing.q,interval_3.q,authorization_compile.q,join35.q,merge_negative_3.q,udf_concat_ws_wrong3.q,create_or_replace_view8.q,create_external_with_notnull_constraint.q,split_sample_out_of_range.q,materialized_view_no_transactional_rewrite.q,authorization_show_grant_otherrole.q,create_with_constraints_duplicate_name.q,invalid_stddev_samp_syntax.q,authorization_view_disable_cbo_7.q,autolocal1.q,avro_non_nullable_union.q,load_orc_negative_part.q,drop_view_failure1.q,columnstats_partlvl_invalid_values_autogather.q,exim_13_nonnative_import.q,alter_table_wrong_regex.q,add_partition_with_whitelist.q,udf_next_day_error_2.q,authorization_select.q,udf_trunc_error2.q,authorization_view_7.q,udf_format_number_wrong5.q,touch2.q,exim_03_nonpart_noncompat_colschema.q,orc_type_promotion1.q,lateral_view_alias.q,show_tables_bad_db1.q,unset_table_property.q,alter_non_native.q,nvl_mismatch_type.q,load_orc_negative3.q,authorization_create_role_no_admin.q,invalid_distinct1.q,authorization_grant_server.q,orc_type_promotion3_acid.q,hms_using_serde_alter_table_update_columns.q,show_tables_bad1.q,macro_unused_parameter.q,drop_invalid_constraint3.q,drop_partition_filter_failure.q,char_pad_convert_fail3.q,exim_23_import_exist_authfail.q,drop_invalid_constraint4.q,authorization_create_macro1.q,archive1.q,subquery_multiple_cols_in_select.q,change_hive_hdfs_session_path.q,udf_trunc_error3.q,invalid_variance_syntax.q,authorization_truncate_2.q,invalid_avg_syntax.q,invalid_select_column_with_tablename.q,mm_truncate_cols.q,groupby_grouping_sets1.q,druid_location.q,groupby2_multi_distinct.q,authorization_sba_drop_table.q,dynamic_partitions_with_whitelist.q,compare_string_bigint_2.q,udf_greatest_error_2.q,authorization_view_6.q,show_tablestatus.q,duplicate_alias_in_transform_schema.q,create_with_fk_uk_same_tab.q,udtf_not_supported3.q,alter_table_constraint_invalid_fk_col2.q,udtf_not_supported1.q,dbtxnmgr_notableunlock.q,ptf_negative_InvalidValueBoundary.q,alter_table_constraint_duplicate_pk.q,udf_printf_wrong4.q,create_view_failure9.q,udf_elt_wrong_type.q,selectDistinctStarNeg_1.q,invalid_mapjoin1.q,load_stored_as_dirs.q,input1.q,udf_sort_array_wrong1.q,invalid_distinct2.q,invalid_select_fn.q,authorization_role_grant_otherrole.q,archive4.q,load_nonpart_authfail.q,recursive_view.q,authorization_view_disable_cbo_1.q,desc_failure4.q,create_not_acid.q,udf_sort_array_wrong3.q,char_pad_convert_fail0.q,udf_map_values_arg_type.q,alter_view_failure6_2.q,alter_partition_change_col_nonexist.q,update_non_acid_table.q,authorization_view_disable_cbo_5.q,ct_noperm_loc.q,interval_1.q,authorization_show_grant_otheruser_all.q,authorization_view_2.q,show_tables_bad2.q,groupby_rollup2.q,truncate_column_seqfile.q,create_view_failure5.q,authorization_create_view.q,ptf_window_boundaries.q,
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412817#comment-16412817 ] Hive QA commented on HIVE-15995: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 39s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 55s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 44s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 37s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 34s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 30s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 55s{color} | {color:red} root: The patch generated 5 new + 808 unchanged - 0 fixed = 813 total (was 808) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 47s{color} | {color:red} ql: The patch generated 5 new + 808 unchanged - 0 fixed = 813 total (was 808) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 49s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 22s{color} | {color:red} The patch generated 49 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 45m 27s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-9807/dev-support/hive-personality.sh | | git revision | master / 696affa | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9807/yetus/diff-checkstyle-root.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9807/yetus/diff-checkstyle-ql.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-9807/yetus/patch-asflicense-problems.txt | | modules | C: . ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9807/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, > HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.5.patch, HIVE-15995.patch, > cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new D
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410694#comment-16410694 ] Hive QA commented on HIVE-15995: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12915705/HIVE-15995.5.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 329 failed/errored test(s), 13486 tests executed *Failed tests:* {noformat} TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=92) [infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q] TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=95) [udf_invalid.q,authorization_uri_export.q,default_constraint_complex_default_value.q,druid_datasource2.q,view_update.q,default_partition_name.q,authorization_public_create.q,load_wrong_fileformat_rc_seq.q,default_constraint_invalid_type.q,altern1.q,describe_xpath1.q,drop_view_failure2.q,temp_table_rename.q,invalid_select_column_with_subquery.q,udf_trunc_error1.q,insert_view_failure.q,dbtxnmgr_nodbunlock.q,authorization_show_columns.q,cte_recursion.q,merge_constraint_notnull.q,load_part_nospec.q,clusterbyorderby.q,orc_type_promotion2.q,ctas_noperm_loc.q,udf_instr_wrong_args_len.q,invalid_create_tbl2.q,part_col_complex_type.q,authorization_drop_db_empty.q,smb_mapjoin_14.q,subquery_scalar_multi_rows.q,alter_partition_coltype_2columns.q,subquery_corr_in_agg.q,insert_overwrite_notnull_constraint.q,authorization_show_grant_otheruser_wtab.q,regex_col_groupby.q,udaf_collect_set_unsupported.q,ptf_negative_DuplicateWindowAlias.q,exim_22_export_authfail.q,udf_likeany_wrong1.q,groupby_key.q,ambiguous_col.q,groupby3_multi_distinct.q,authorization_alter_drop_ptn.q,invalid_cast_from_binary_5.q,show_create_table_does_not_exist.q,invalid_select_column.q,exim_20_managed_location_over_existing.q,interval_3.q,authorization_compile.q,join35.q,merge_negative_3.q,udf_concat_ws_wrong3.q,create_or_replace_view8.q,create_external_with_notnull_constraint.q,split_sample_out_of_range.q,materialized_view_no_transactional_rewrite.q,authorization_show_grant_otherrole.q,create_with_constraints_duplicate_name.q,invalid_stddev_samp_syntax.q,authorization_view_disable_cbo_7.q,autolocal1.q,avro_non_nullable_union.q,load_orc_negative_part.q,drop_view_failure1.q,columnstats_partlvl_invalid_values_autogather.q,exim_13_nonnative_import.q,alter_table_wrong_regex.q,add_partition_with_whitelist.q,udf_next_day_error_2.q,authorization_select.q,udf_trunc_error2.q,authorization_view_7.q,udf_format_number_wrong5.q,touch2.q,exim_03_nonpart_noncompat_colschema.q,orc_type_promotion1.q,lateral_view_alias.q,show_tables_bad_db1.q,unset_table_property.q,alter_non_native.q,nvl_mismatch_type.q,load_orc_negative3.q,authorization_create_role_no_admin.q,invalid_distinct1.q,authorization_grant_server.q,orc_type_promotion3_acid.q,hms_using_serde_alter_table_update_columns.q,show_tables_bad1.q,macro_unused_parameter.q,drop_invalid_constraint3.q,drop_partition_filter_failure.q,char_pad_convert_fail3.q,exim_23_import_exist_authfail.q,drop_invalid_constraint4.q,authorization_create_macro1.q,archive1.q,subquery_multiple_cols_in_select.q,change_hive_hdfs_session_path.q,udf_trunc_error3.q,invalid_variance_syntax.q,authorization_truncate_2.q,invalid_avg_syntax.q,invalid_select_column_with_tablename.q,mm_truncate_cols.q,groupby_grouping_sets1.q,druid_location.q,groupby2_multi_distinct.q,authorization_sba_drop_table.q,dynamic_partitions_with_whitelist.q,compare_string_bigint_2.q,udf_greatest_error_2.q,authorization_view_6.q,show_tablestatus.q,duplicate_alias_in_transform_schema.q,create_with_fk_uk_same_tab.q,udtf_not_supported3.q,alter_table_constraint_invalid_fk_col2.q,udtf_not_supported1.q,dbtxnmgr_notableunlock.q,ptf_negative_InvalidValueBoundary.q,alter_table_constraint_duplicate_pk.q,udf_printf_wrong4.q,create_view_failure9.q,udf_elt_wrong_type.q,selectDistinctStarNeg_1.q,invalid_mapjoin1.q,load_stored_as_dirs.q,input1.q,udf_sort_array_wrong1.q,invalid_distinct2.q,invalid_select_fn.q,authorization_role_grant_otherrole.q,archive4.q,load_nonpart_authfail.q,recursive_view.q,authorization_view_disable_cbo_1.q,desc_failure4.q,create_not_acid.q,udf_sort_array_wrong3.q,char_pad_convert_fail0.q,udf_map_values_arg_type.q,alter_view_failure6_2.q,alter_partition_change_col_nonexist.q,update_non_acid_table.q,authorization_view_disable_cbo_5.q,ct_noperm_loc.q,interval_1.q,authorization_show_grant_otheruser_all.q,authorization_view_2.q,show_tables_bad2.q,groupby_rollup2.q,truncate_column_seqfile.q,create_view_failure5.q,authorization_create_view.q,ptf_window_boundaries.q
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410668#comment-16410668 ] Hive QA commented on HIVE-15995: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 51s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 33s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 30s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 25s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 32s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 50s{color} | {color:red} root: The patch generated 5 new + 808 unchanged - 0 fixed = 813 total (was 808) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 41s{color} | {color:red} ql: The patch generated 5 new + 808 unchanged - 0 fixed = 813 total (was 808) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 13s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 43m 54s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-9777/dev-support/hive-personality.sh | | git revision | master / 353a9cf | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9777/yetus/diff-checkstyle-root.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9777/yetus/diff-checkstyle-ql.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-9777/yetus/patch-asflicense-problems.txt | | modules | C: . ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9777/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, > HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.5.patch, HIVE-15995.patch, > cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new D
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409733#comment-16409733 ] Adam Szita commented on HIVE-15995: --- [~vihangk1], yup that makes sense, I've added support for partition spec on "update columns" see patch: [^HIVE-15995.5.patch] > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, > HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.5.patch, HIVE-15995.patch, > cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = > '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table > Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16389925#comment-16389925 ] Vihang Karajgaonkar commented on HIVE-15995: Hi [~szita] thanks for the updated patch. I was looking at the other queries which take in {{cascade}} option and I noticed that they have a partition spec in the query syntax. Eg: add/rename column is under alterTblPartitionStatementSuffix in HiveParser.g I think it has to do with the option of adding partition spec to the command. Do you think it makes sense to add a optional partition spec as well similar to add/replace columns query? I think column descriptors are also at partition level. Eg. you can do a {{describe table foo partition (p1=10);}} and hive will show the column information for that partition. So users can potentially update the columns at a partition level as well. The current patch is either just the table no partitions or table with all partitions. There is no way to update columns for a subset of partitions. Also, in the qtest can you add a test for a partitioned table? Include describe table .. partition .. syntax to confirm that partition columns are also getting updated. > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, > HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = > '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table > Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16389410#comment-16389410 ] Adam Szita commented on HIVE-15995: --- Failing tests in the report are flaky/unrelated (even avro_schema_evolution_native...). [~vihangk1] let me know if you think this change is ready for commit. > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, > HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = > '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table > Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388263#comment-16388263 ] Hive QA commented on HIVE-15995: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12913215/HIVE-15995.4.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 13484 tests executed *Failed tests:* {noformat} TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=95) [udf_invalid.q,authorization_uri_export.q,default_constraint_complex_default_value.q,druid_datasource2.q,view_update.q,default_partition_name.q,authorization_public_create.q,load_wrong_fileformat_rc_seq.q,default_constraint_invalid_type.q,altern1.q,describe_xpath1.q,drop_view_failure2.q,temp_table_rename.q,invalid_select_column_with_subquery.q,udf_trunc_error1.q,insert_view_failure.q,dbtxnmgr_nodbunlock.q,authorization_show_columns.q,cte_recursion.q,merge_constraint_notnull.q,load_part_nospec.q,clusterbyorderby.q,orc_type_promotion2.q,ctas_noperm_loc.q,udf_instr_wrong_args_len.q,invalid_create_tbl2.q,part_col_complex_type.q,authorization_drop_db_empty.q,smb_mapjoin_14.q,subquery_scalar_multi_rows.q,alter_partition_coltype_2columns.q,subquery_corr_in_agg.q,insert_overwrite_notnull_constraint.q,authorization_show_grant_otheruser_wtab.q,regex_col_groupby.q,udaf_collect_set_unsupported.q,ptf_negative_DuplicateWindowAlias.q,exim_22_export_authfail.q,udf_likeany_wrong1.q,groupby_key.q,ambiguous_col.q,groupby3_multi_distinct.q,authorization_alter_drop_ptn.q,invalid_cast_from_binary_5.q,show_create_table_does_not_exist.q,invalid_select_column.q,exim_20_managed_location_over_existing.q,interval_3.q,authorization_compile.q,join35.q,merge_negative_3.q,udf_concat_ws_wrong3.q,create_or_replace_view8.q,create_external_with_notnull_constraint.q,split_sample_out_of_range.q,materialized_view_no_transactional_rewrite.q,authorization_show_grant_otherrole.q,create_with_constraints_duplicate_name.q,invalid_stddev_samp_syntax.q,authorization_view_disable_cbo_7.q,autolocal1.q,avro_non_nullable_union.q,load_orc_negative_part.q,drop_view_failure1.q,columnstats_partlvl_invalid_values_autogather.q,exim_13_nonnative_import.q,alter_table_wrong_regex.q,add_partition_with_whitelist.q,udf_next_day_error_2.q,authorization_select.q,udf_trunc_error2.q,authorization_view_7.q,udf_format_number_wrong5.q,touch2.q,exim_03_nonpart_noncompat_colschema.q,orc_type_promotion1.q,lateral_view_alias.q,show_tables_bad_db1.q,unset_table_property.q,alter_non_native.q,nvl_mismatch_type.q,load_orc_negative3.q,authorization_create_role_no_admin.q,invalid_distinct1.q,authorization_grant_server.q,orc_type_promotion3_acid.q,hms_using_serde_alter_table_update_columns.q,show_tables_bad1.q,macro_unused_parameter.q,drop_invalid_constraint3.q,drop_partition_filter_failure.q,char_pad_convert_fail3.q,exim_23_import_exist_authfail.q,drop_invalid_constraint4.q,authorization_create_macro1.q,archive1.q,subquery_multiple_cols_in_select.q,change_hive_hdfs_session_path.q,udf_trunc_error3.q,invalid_variance_syntax.q,authorization_truncate_2.q,invalid_avg_syntax.q,invalid_select_column_with_tablename.q,mm_truncate_cols.q,groupby_grouping_sets1.q,druid_location.q,groupby2_multi_distinct.q,authorization_sba_drop_table.q,dynamic_partitions_with_whitelist.q,compare_string_bigint_2.q,udf_greatest_error_2.q,authorization_view_6.q,show_tablestatus.q,duplicate_alias_in_transform_schema.q,create_with_fk_uk_same_tab.q,udtf_not_supported3.q,alter_table_constraint_invalid_fk_col2.q,udtf_not_supported1.q,dbtxnmgr_notableunlock.q,ptf_negative_InvalidValueBoundary.q,alter_table_constraint_duplicate_pk.q,udf_printf_wrong4.q,create_view_failure9.q,udf_elt_wrong_type.q,selectDistinctStarNeg_1.q,invalid_mapjoin1.q,load_stored_as_dirs.q,input1.q,udf_sort_array_wrong1.q,invalid_distinct2.q,invalid_select_fn.q,authorization_role_grant_otherrole.q,archive4.q,load_nonpart_authfail.q,recursive_view.q,authorization_view_disable_cbo_1.q,desc_failure4.q,create_not_acid.q,udf_sort_array_wrong3.q,char_pad_convert_fail0.q,udf_map_values_arg_type.q,alter_view_failure6_2.q,alter_partition_change_col_nonexist.q,update_non_acid_table.q,authorization_view_disable_cbo_5.q,ct_noperm_loc.q,interval_1.q,authorization_show_grant_otheruser_all.q,authorization_view_2.q,show_tables_bad2.q,groupby_rollup2.q,truncate_column_seqfile.q,create_view_failure5.q,authorization_create_view.q,ptf_window_boundaries.q,ctasnullcol.q,input_part0_neg_2.q,create_or_replace_view1.q,udf_max.q,exim_01_nonpart_over_loaded.q,msck_repair_1.q,orc_change_fileformat_acid.q,udf_nonexistent_resource.q,exim_19_external_over_existing.q,serde_regex2.q,msck_repair_2.q,exim_06_nonpart_noncompat_storage.q,illegal_partition_type4.q,udf_sort_array_by_wrong1.q,create_or_replace_view5.q,windowing_leadlag_in_
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388205#comment-16388205 ] Hive QA commented on HIVE-15995: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 1s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 25s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 33s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 25s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 52s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 6s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 27s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 44s{color} | {color:red} root: The patch generated 4 new + 808 unchanged - 0 fixed = 812 total (was 808) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 42s{color} | {color:red} ql: The patch generated 4 new + 808 unchanged - 0 fixed = 812 total (was 808) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 15s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 42m 23s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-9513/dev-support/hive-personality.sh | | git revision | master / 6001f51 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9513/yetus/diff-checkstyle-root.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9513/yetus/diff-checkstyle-ql.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-9513/yetus/patch-asflicense-problems.txt | | modules | C: . ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9513/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, > HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing >
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387965#comment-16387965 ] Adam Szita commented on HIVE-15995: --- [~vihangk1] thanks for taking a look! Agree with the cascade proposal, I've added this option in my new patch ([^HIVE-15995.4.patch]) In the q test what I do is basically update the Avro schema and then remove the setting avro.schema.url/literal. After removal HMS will either have the old schema when I call describe, or it will have the new schema, if I also called {{update columns}} before removing the setting. That's what the q tests are based on. On the test failures, I'll see what result I'll get after this latest change and will triage it. > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, > HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = > '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table > Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373792#comment-16373792 ] Vihang Karajgaonkar commented on HIVE-15995: Thanks [~szita] for the patch. The patch assumes that user wants to do a cascade the schema change to all the partitions as well. Instead of that I think it would be good if we introduce a optional field to the command specifying clearly whether the user wants to cascade the schema change to partitions or not. This is in line with the {{[CASCADE|RESTRICT]}} usage in the {{ALTER TABLE table_name ADD|REPLACE COLUMNS ... [CASCADE|RESTRICT]}} syntax. The default should be RESTRICT which means that only table metadata should be updated. If the user specifies CASCADE only then the metadata update should be cascaded to partitions. Assuming that the user wants to cascade could be dangerous esp. if there are older partitions which doesn't have all the column information. Secondly, does the q-test fail without the patch? I thought the describe command already uses the Deserializer internally so the updated schema will be seen regardless. I think you will need to write a junit test and confirm that metadata is updated using metastore API. Also, can you take a look at the test failures above? Some of them look new (I am starting to lose track of the regular failures now) Not sure if they are related to the patch or not. > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, > HIVE-15995.3.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = > '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table > Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373761#comment-16373761 ] Hive QA commented on HIVE-15995: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12911582/HIVE-15995.3.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 37 failed/errored test(s), 13013 tests executed *Failed tests:* {noformat} TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=93) [nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,create_table_failure4.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,nopart_load.q,alter_table_non_partitioned_table_cascade.q,load_wrong_fileformat.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,udf_min.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,authorization_insert_noinspriv.q,duplicate_insert3.q,authorization_desc_table_nosel.q,invalid_select_column.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,alter_external_with_constraint.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,authorization_create_macro1.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,char_pad_convert_fail0.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_fail2.q,authorization_set_role_neg1.q,serde_regex3.q,authorization_delete_nodeletepriv.q,materialized_view_delete.q,create_or_replace_view6.q,bucket_mapjoin_wrong_table_metadata_2.q,msck_repair_3.q,udf_sort_array_by_wrong2.q,local_mapred_error_cache.q,alter_external_acid.q,mm_concatenate.q,authorization_fail_3.q,set_hiveconf_internal_variable0.q,udf_last_day_error_2.q,alter_table_constraint_invalid_ref.q,create_table_wrong_regex.q,describe_xpath4.q,joi
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373731#comment-16373731 ] Hive QA commented on HIVE-15995: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 40s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 51s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 9s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 2s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 59s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 12s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 7s{color} | {color:red} root: The patch generated 4 new + 828 unchanged - 0 fixed = 832 total (was 828) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 43s{color} | {color:red} ql: The patch generated 4 new + 828 unchanged - 0 fixed = 832 total (was 828) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 22s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 45m 43s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-9319/dev-support/hive-personality.sh | | git revision | master / 7a8f105 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9319/yetus/diff-checkstyle-root.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9319/yetus/diff-checkstyle-ql.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-9319/yetus/patch-asflicense-problems.txt | | modules | C: . ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9319/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, > HIVE-15995.3.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns w
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373099#comment-16373099 ] Adam Szita commented on HIVE-15995: --- Rebased patch: [^HIVE-15995.3.patch] > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, > HIVE-15995.3.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = > '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table > Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360989#comment-16360989 ] Hive QA commented on HIVE-15995: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12910176/HIVE-15995.2.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 13170 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=241) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=161) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_opt_shuffle_serde] (batchId=180) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=122) org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query1] (batchId=251) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=222) org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsByValuesNoTblName[Embedded] (batchId=206) org.apache.hadoop.hive.metastore.client.TestTablesGetExists.testGetAllTablesCaseInsensitive[Embedded] (batchId=206) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=235) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=235) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=235) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9173/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9173/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9173/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 22 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12910176 - PreCommit-HIVE-Build > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, > HIVE-15995.patch, cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update s
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360935#comment-16360935 ] Hive QA commented on HIVE-15995: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 3m 47s{color} | {color:red} root in master failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 2m 43s{color} | {color:red} root in master failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m 31s{color} | {color:red} ql in master failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 20s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 45s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 44s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 32s{color} | {color:red} root generated 29 new + 5 unchanged - 2 fixed = 34 total (was 7) {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 12s{color} | {color:green} ql generated 0 new + 5 unchanged - 44 fixed = 5 total (was 49) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 18s{color} | {color:red} root: The patch generated 4 new + 825 unchanged - 0 fixed = 829 total (was 825) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 51s{color} | {color:red} ql: The patch generated 4 new + 825 unchanged - 0 fixed = 829 total (was 825) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 6m 14s{color} | {color:red} root generated 1 new + 335 unchanged - 0 fixed = 336 total (was 335) {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 47m 15s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 887233d | | Default Java | 1.8.0_111 | | mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-9173/yetus/branch-mvninstall-root.txt | | compile | http://104.198.109.242/logs//PreCommit-HIVE-Build-9173/yetus/branch-compile-root.txt | | compile | http://104.198.109.242/logs//PreCommit-HIVE-Build-9173/yetus/branch-compile-ql.txt | | javac | http://104.198.109.242/logs//PreCommit-HIVE-Build-9173/yetus/diff-compile-javac-root.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9173/yetus/diff-checkstyle-root.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9173/yetus/diff-checkstyle-ql.txt | | javadoc | http://104.198.109.242/logs//PreCommit-HIVE-Build-9173/yetus/diff-javadoc-javadoc-root.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-9173/yetus/patch-asflicense-problems.txt | | modules | C: . ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9173/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https:/
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360692#comment-16360692 ] Hive QA commented on HIVE-15995: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12910176/HIVE-15995.2.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9172/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9172/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9172/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2018-02-12 12:40:55.303 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-9172/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2018-02-12 12:40:55.307 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 887233d HIVE-18646: Update errata.txt for HIVE-18617 (Daniel Voros via Zoltan Haindrich) + git clean -f -d + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 887233d HIVE-18646: Update errata.txt for HIVE-18617 (Daniel Voros via Zoltan Haindrich) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2018-02-12 12:40:59.613 + rm -rf ../yetus rm: cannot remove ?../yetus?: Directory not empty ' {noformat} This message is automatically generated. ATTACHMENT ID: 12910176 - PreCommit-HIVE-Build > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, > HIVE-15995.patch, cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = > '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table > Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360522#comment-16360522 ] Adam Szita commented on HIVE-15995: --- Attached new patch, fixed whitespace problems, not fixing checkstyle problems to match the code environment the affected lines are in.. > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, > HIVE-15995.patch, cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = > '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table > Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357013#comment-16357013 ] Hive QA commented on HIVE-15995: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12909622/HIVE-15995.1.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 12997 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin] (batchId=166) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=161) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=122) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=221) org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionWithAuthInfoNoDbName[Embedded] (batchId=206) org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded] (batchId=206) org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap (batchId=282) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9094/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9094/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9094/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 23 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12909622 - PreCommit-HIVE-Build > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356988#comment-16356988 ] Hive QA commented on HIVE-15995: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 41s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 12s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 47s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 34s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 29s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 48s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 47s{color} | {color:red} ql: The patch generated 4 new + 825 unchanged - 0 fixed = 829 total (was 825) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 57s{color} | {color:red} root: The patch generated 4 new + 825 unchanged - 0 fixed = 829 total (was 825) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 44s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 46m 2s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 6e9b63e | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9094/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9094/yetus/diff-checkstyle-root.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-9094/yetus/whitespace-eol.txt | | modules | C: ql . U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9094/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metas
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1635#comment-1635 ] Adam Szita commented on HIVE-15995: --- As I haven't received any answer I took the liberty to upload a revised patch here, because one of our customers has also experienced this issue. [^HIVE-15995.1.patch] is based on the original patch with the following changes: * It is rebased (had to because of standalone hms) * It also has a safety catch so that this feature only works for non-hms serde tables (this could cause problems with those serdes on partitioned tables) * It now has (q) test cases for verification Please let me know what you think [~pvary], [~michal.ferlinski], [~aihuaxu], [~vihangk1] > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0, 3.0.0 >Reporter: Michal Ferlinski >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-15995.1.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = > '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table > Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352490#comment-16352490 ] Hive QA commented on HIVE-15995: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12854469/HIVE-15995.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9021/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9021/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9021/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2018-02-05 14:49:05.454 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-9021/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2018-02-05 14:49:05.486 + cd apache-github-source-source + git fetch origin >From https://github.com/apache/hive 0a328f0..a128c0e master -> origin/master + git reset --hard HEAD HEAD is now at 0a328f0 HIVE-18546: Remove unnecessary code introduced in HIVE-14498 (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan) + git clean -f -d + git checkout master Already on 'master' Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded. (use "git pull" to update your local branch) + git reset --hard origin/master HEAD is now at a128c0e HIVE-18578: Some class has missed the ASF header (Saijin Huang via Zoltan Haindrich) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2018-02-05 14:49:15.271 + rm -rf ../yetus + mkdir ../yetus + git gc + cp -R . ../yetus + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9021/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableDesc.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveOperationType.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java: does not exist in index error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g:1145 Falling back to three-way merge... Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g' cleanly. error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveOperationType.java:121 Falling back to three-way merge... Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveOperationType.java' cleanly. error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java:229 Falling back to three-way merge... Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java' cleanly. Going to apply patch with: git apply -p1 error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g:1145 Falling back to three-way merge... Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g' cleanly. error: patch failed: ql/src/java/org/apa
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352372#comment-16352372 ] Adam Szita commented on HIVE-15995: --- [~michal.ferlinski], [~pxiong] any update on this please? Do I understand correctly that this patch is ready to go in master branch? > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0 >Reporter: Michal Ferlinski >Assignee: Michal Ferlinski >Priority: Major > Fix For: 2.1.0 > > Attachments: HIVE-15995.patch, cx1.avsc, cx2.avsc > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = > '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table > Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15942022#comment-15942022 ] Pengcheng Xiong commented on HIVE-15995: Hello, I am deferring this to Hive 3.0 as we are going to cut the first RC and it is not marked as blocker. Please feel free to commit to the branch if this can be resolved before the release. > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0 >Reporter: Michal Ferlinski >Assignee: Michal Ferlinski > Fix For: 2.1.0 > > Attachments: cx1.avsc, cx2.avsc, HIVE-15995.patch > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = > '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table > Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15883545#comment-15883545 ] Hive QA commented on HIVE-15995: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12854469/HIVE-15995.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10226 tests executed *Failed tests:* {noformat} TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver (batchId=161) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=223) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=129) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3762/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3762/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3762/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12854469 - PreCommit-HIVE-Build > Syncing metastore table with serde schema > - > > Key: HIVE-15995 > URL: https://issues.apache.org/jira/browse/HIVE-15995 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.1.0 >Reporter: Michal Ferlinski >Assignee: Michal Ferlinski > Fix For: 2.1.0 > > Attachments: cx1.avsc, cx2.avsc, HIVE-15995.patch > > > Hive enables table schema evolution via properties. For avro e.g. we can > alter the 'avro.schema.url' property to update table schema to the next > version. Updating properties however doesn't affect column list stored in > metastore DB so the table is not in the newest version when returned from > metastore API. This is problem for tools working with metastore (e.g. Presto). > To solve this issue I suggest to introduce new DDL statement syncing > metastore columns with those from serde: > {code} > ALTER TABLE user_test1 UPDATE COLUMNS > {code} > Note that this is format independent solution. > To reproduce, follow the instructions below: > - Create table based on avro schema version 1 (cxv1.avsc) > {code} > CREATE EXTERNAL TABLE user_test1 > PARTITIONED BY (dt string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > LOCATION > '/tmp/schema-evolution/user_test1' > TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc'); > {code} > - Update schema to version 2 (cx2.avsc) > {code} > ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = > '/tmp/schema-evolution/cx2.avsc'); > {code} > - Print serde columns (top info) and metastore columns (Detailed Table > Information): > {code} > DESCRIBE EXTENDED user_test1 > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)