[jira] [Commented] (HIVE-10324) Hive metatool should take table_param_key to allow for changes to avro serde's schema url key
[
https://issues.apache.org/jira/browse/HIVE-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508270#comment-14508270
]
Ferdinand Xu commented on HIVE-10324:
-
Awesome!! You can add the following information to the use example section.
Thanks [~leftylev]
{noformat}
./hive --service metatool -updateLocation hdfs://localhost:9000
hdfs://namenode2:8020 -tablePropKey avro.schema.url -serdePropKey
avro.schema.url
Initializing HiveMetaTool..
15/04/22 14:18:42 INFO metastore.ObjectStore: ObjectStore, initialize called
15/04/22 14:18:42 INFO DataNucleus.Persistence: Property
hive.metastore.integral.jdo.pushdown unknown - will be ignored
15/04/22 14:18:42 INFO DataNucleus.Persistence: Property
datanucleus.cache.level2 unknown - will be ignored
15/04/22 14:18:43 INFO metastore.ObjectStore: Setting MetaStore object pin
classes with
hive.metastore.cache.pinobjtypes=Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order
15/04/22 14:18:43 INFO DataNucleus.Datastore: The class
org.apache.hadoop.hive.metastore.model.MFieldSchema is tagged as
embedded-only so does not have its own datastore table.
15/04/22 14:18:43 INFO DataNucleus.Datastore: The class
org.apache.hadoop.hive.metastore.model.MOrder is tagged as embedded-only so
does not have its own datastore table.
15/04/22 14:18:44 INFO DataNucleus.Datastore: The class
org.apache.hadoop.hive.metastore.model.MFieldSchema is tagged as
embedded-only so does not have its own datastore table.
15/04/22 14:18:44 INFO DataNucleus.Datastore: The class
org.apache.hadoop.hive.metastore.model.MOrder is tagged as embedded-only so
does not have its own datastore table.
15/04/22 14:18:44 INFO DataNucleus.Query: Reading in results for query
org.datanucleus.store.rdbms.query.SQLQuery@0 since the connection used is
closing
15/04/22 14:18:44 INFO metastore.MetaStoreDirectSql: Using direct SQL,
underlying DB is MYSQL
15/04/22 14:18:44 INFO metastore.ObjectStore: Initialized ObjectStore
Looking for LOCATION_URI field in DBS table to update..
Successfully updated the following locations..
Updated 0 records in DBS table
Looking for LOCATION field in SDS table to update..
Successfully updated the following locations..
Updated 0 records in SDS table
Looking for value of avro.schema.url key in TABLE_PARAMS table to update..
Successfully updated the following locations..
Updated 0 records in TABLE_PARAMS table
Looking for value of avro.schema.url key in SD_PARAMS table to update..
Successfully updated the following locations..
Updated 0 records in SD_PARAMS table
Looking for value of avro.schema.url key in SERDE_PARAMS table to update..
Successfully updated the following locations..
Updated 0 records in SERDE_PARAMS table
{noformat}
Hive metatool should take table_param_key to allow for changes to avro
serde's schema url key
-
Key: HIVE-10324
URL: https://issues.apache.org/jira/browse/HIVE-10324
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Ferdinand Xu
Fix For: 1.2.0
Attachments: HIVE-10324.1.patch, HIVE-10324.patch,
HIVE-10324.patch.WIP
HIVE-3443 added support to change the serdeParams from 'metatool
updateLocation' command.
However, in avro it is possible to specify the schema via the tableParams:
{noformat}
CREATE TABLE `testavro`(
`test` string COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='hdfs://namenode:8020/tmp/test.avsc',
'kite.compression.type'='snappy',
'transient_lastDdlTime'='1427996456')
{noformat}
Hence for those tables the 'metatool updateLocation' will not help.
This is necessary in case like upgrade the namenode to HA where the absolute
paths have changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10324) Hive metatool should take table_param_key to allow for changes to avro serde's schema url key
[
https://issues.apache.org/jira/browse/HIVE-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506693#comment-14506693
]
Lefty Leverenz commented on HIVE-10324:
---
[~Ferd], a first draft of the metatool doc is ready for review in the wiki:
* [Hive MetaTool |
https://cwiki.apache.org/confluence/display/Hive/Hive+MetaTool]
Hive metatool should take table_param_key to allow for changes to avro
serde's schema url key
-
Key: HIVE-10324
URL: https://issues.apache.org/jira/browse/HIVE-10324
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Ferdinand Xu
Fix For: 1.2.0
Attachments: HIVE-10324.1.patch, HIVE-10324.patch,
HIVE-10324.patch.WIP
HIVE-3443 added support to change the serdeParams from 'metatool
updateLocation' command.
However, in avro it is possible to specify the schema via the tableParams:
{noformat}
CREATE TABLE `testavro`(
`test` string COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='hdfs://namenode:8020/tmp/test.avsc',
'kite.compression.type'='snappy',
'transient_lastDdlTime'='1427996456')
{noformat}
Hence for those tables the 'metatool updateLocation' will not help.
This is necessary in case like upgrade the namenode to HA where the absolute
paths have changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10324) Hive metatool should take table_param_key to allow for changes to avro serde's schema url key
[
https://issues.apache.org/jira/browse/HIVE-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504121#comment-14504121
]
Lefty Leverenz commented on HIVE-10324:
---
Super, thanks [~Ferd]!
Hive metatool should take table_param_key to allow for changes to avro
serde's schema url key
-
Key: HIVE-10324
URL: https://issues.apache.org/jira/browse/HIVE-10324
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Ferdinand Xu
Fix For: 1.2.0
Attachments: HIVE-10324.1.patch, HIVE-10324.patch,
HIVE-10324.patch.WIP
HIVE-3443 added support to change the serdeParams from 'metatool
updateLocation' command.
However, in avro it is possible to specify the schema via the tableParams:
{noformat}
CREATE TABLE `testavro`(
`test` string COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='hdfs://namenode:8020/tmp/test.avsc',
'kite.compression.type'='snappy',
'transient_lastDdlTime'='1427996456')
{noformat}
Hence for those tables the 'metatool updateLocation' will not help.
This is necessary in case like upgrade the namenode to HA where the absolute
paths have changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10324) Hive metatool should take table_param_key to allow for changes to avro serde's schema url key
[
https://issues.apache.org/jira/browse/HIVE-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14498516#comment-14498516
]
Hive QA commented on HIVE-10324:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12725773/HIVE-10324.1.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 8702 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3460/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3460/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3460/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12725773 - PreCommit-HIVE-TRUNK-Build
Hive metatool should take table_param_key to allow for changes to avro
serde's schema url key
-
Key: HIVE-10324
URL: https://issues.apache.org/jira/browse/HIVE-10324
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Ferdinand Xu
Attachments: HIVE-10324.1.patch, HIVE-10324.patch,
HIVE-10324.patch.WIP
HIVE-3443 added support to change the serdeParams from 'metatool
updateLocation' command.
However, in avro it is possible to specify the schema via the tableParams:
{noformat}
CREATE TABLE `testavro`(
`test` string COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='hdfs://namenode:8020/tmp/test.avsc',
'kite.compression.type'='snappy',
'transient_lastDdlTime'='1427996456')
{noformat}
Hence for those tables the 'metatool updateLocation' will not help.
This is necessary in case like upgrade the namenode to HA where the absolute
paths have changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10324) Hive metatool should take table_param_key to allow for changes to avro serde's schema url key
[
https://issues.apache.org/jira/browse/HIVE-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499055#comment-14499055
]
Ferdinand Xu commented on HIVE-10324:
-
Thanks [~szehon] for your review. And the failed cases are irrelevant.
Hive metatool should take table_param_key to allow for changes to avro
serde's schema url key
-
Key: HIVE-10324
URL: https://issues.apache.org/jira/browse/HIVE-10324
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Ferdinand Xu
Attachments: HIVE-10324.1.patch, HIVE-10324.patch,
HIVE-10324.patch.WIP
HIVE-3443 added support to change the serdeParams from 'metatool
updateLocation' command.
However, in avro it is possible to specify the schema via the tableParams:
{noformat}
CREATE TABLE `testavro`(
`test` string COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='hdfs://namenode:8020/tmp/test.avsc',
'kite.compression.type'='snappy',
'transient_lastDdlTime'='1427996456')
{noformat}
Hence for those tables the 'metatool updateLocation' will not help.
This is necessary in case like upgrade the namenode to HA where the absolute
paths have changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10324) Hive metatool should take table_param_key to allow for changes to avro serde's schema url key
[
https://issues.apache.org/jira/browse/HIVE-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496425#comment-14496425
]
Hive QA commented on HIVE-10324:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12725504/HIVE-10324.patch
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 8688 tests
executed
*Failed tests:*
{noformat}
TestCustomAuthentication - did not produce a TEST-*.xml file
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.metastore.TestHiveMetaTool.testUpdateFSRootLocation
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3444/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3444/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3444/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12725504 - PreCommit-HIVE-TRUNK-Build
Hive metatool should take table_param_key to allow for changes to avro
serde's schema url key
-
Key: HIVE-10324
URL: https://issues.apache.org/jira/browse/HIVE-10324
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Ferdinand Xu
Attachments: HIVE-10324.patch, HIVE-10324.patch.WIP
HIVE-3443 added support to change the serdeParams from 'metatool
updateLocation' command.
However, in avro it is possible to specify the schema via the tableParams:
{noformat}
CREATE TABLE `testavro`(
`test` string COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='hdfs://namenode:8020/tmp/test.avsc',
'kite.compression.type'='snappy',
'transient_lastDdlTime'='1427996456')
{noformat}
Hence for those tables the 'metatool updateLocation' will not help.
This is necessary in case like upgrade the namenode to HA where the absolute
paths have changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10324) Hive metatool should take table_param_key to allow for changes to avro serde's schema url key
[
https://issues.apache.org/jira/browse/HIVE-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496694#comment-14496694
]
Szehon Ho commented on HIVE-10324:
--
Thanks Ferdinand for taking care of this. Can we keep the update of any
property that match StorageDescriptor property, and just add another method for
Table property? I am afraid that somebody might be using this, unless we can
confirm that that StorageDescriptor property is never used.
Hive metatool should take table_param_key to allow for changes to avro
serde's schema url key
-
Key: HIVE-10324
URL: https://issues.apache.org/jira/browse/HIVE-10324
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Ferdinand Xu
Attachments: HIVE-10324.patch, HIVE-10324.patch.WIP
HIVE-3443 added support to change the serdeParams from 'metatool
updateLocation' command.
However, in avro it is possible to specify the schema via the tableParams:
{noformat}
CREATE TABLE `testavro`(
`test` string COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='hdfs://namenode:8020/tmp/test.avsc',
'kite.compression.type'='snappy',
'transient_lastDdlTime'='1427996456')
{noformat}
Hence for those tables the 'metatool updateLocation' will not help.
This is necessary in case like upgrade the namenode to HA where the absolute
paths have changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10324) Hive metatool should take table_param_key to allow for changes to avro serde's schema url key
[
https://issues.apache.org/jira/browse/HIVE-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497557#comment-14497557
]
Szehon Ho commented on HIVE-10324:
--
Thanks! +1
Hive metatool should take table_param_key to allow for changes to avro
serde's schema url key
-
Key: HIVE-10324
URL: https://issues.apache.org/jira/browse/HIVE-10324
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Ferdinand Xu
Attachments: HIVE-10324.1.patch, HIVE-10324.patch,
HIVE-10324.patch.WIP
HIVE-3443 added support to change the serdeParams from 'metatool
updateLocation' command.
However, in avro it is possible to specify the schema via the tableParams:
{noformat}
CREATE TABLE `testavro`(
`test` string COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='hdfs://namenode:8020/tmp/test.avsc',
'kite.compression.type'='snappy',
'transient_lastDdlTime'='1427996456')
{noformat}
Hence for those tables the 'metatool updateLocation' will not help.
This is necessary in case like upgrade the namenode to HA where the absolute
paths have changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10324) Hive metatool should take table_param_key to allow for changes to avro serde's schema url key
[
https://issues.apache.org/jira/browse/HIVE-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495727#comment-14495727
]
Szehon Ho commented on HIVE-10324:
--
Ah yea it has to be the same db instance (like mysql). It is easy to repro
manually, but might be hard to write a unit test for it. Thanks
Hive metatool should take table_param_key to allow for changes to avro
serde's schema url key
-
Key: HIVE-10324
URL: https://issues.apache.org/jira/browse/HIVE-10324
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Ferdinand Xu
Attachments: HIVE-10324.patch.WIP
HIVE-3443 added support to change the serdeParams from 'metatool
updateLocation' command.
However, in avro it is possible to specify the schema via the tableParams:
{noformat}
CREATE TABLE `testavro`(
`test` string COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='hdfs://namenode:8020/tmp/test.avsc',
'kite.compression.type'='snappy',
'transient_lastDdlTime'='1427996456')
{noformat}
Hence for those tables the 'metatool updateLocation' will not help.
This is necessary in case like upgrade the namenode to HA where the absolute
paths have changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10324) Hive metatool should take table_param_key to allow for changes to avro serde's schema url key
[
https://issues.apache.org/jira/browse/HIVE-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495760#comment-14495760
]
Ferdinand Xu commented on HIVE-10324:
-
Yes, the same result for mysql. I verified my patch manually. Please help me
review it. Thank you!
Hive metatool should take table_param_key to allow for changes to avro
serde's schema url key
-
Key: HIVE-10324
URL: https://issues.apache.org/jira/browse/HIVE-10324
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Ferdinand Xu
Attachments: HIVE-10324.patch, HIVE-10324.patch.WIP
HIVE-3443 added support to change the serdeParams from 'metatool
updateLocation' command.
However, in avro it is possible to specify the schema via the tableParams:
{noformat}
CREATE TABLE `testavro`(
`test` string COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='hdfs://namenode:8020/tmp/test.avsc',
'kite.compression.type'='snappy',
'transient_lastDdlTime'='1427996456')
{noformat}
Hence for those tables the 'metatool updateLocation' will not help.
This is necessary in case like upgrade the namenode to HA where the absolute
paths have changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10324) Hive metatool should take table_param_key to allow for changes to avro serde's schema url key
[
https://issues.apache.org/jira/browse/HIVE-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493628#comment-14493628
]
Szehon Ho commented on HIVE-10324:
--
Basically, after you run the query, run describe extended on the table. The
table should still have the old avro.schema.url (hdfs://namenode:8080).
Hive metatool should take table_param_key to allow for changes to avro
serde's schema url key
-
Key: HIVE-10324
URL: https://issues.apache.org/jira/browse/HIVE-10324
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Ferdinand Xu
HIVE-3443 added support to change the serdeParams from 'metatool
updateLocation' command.
However, in avro it is possible to specify the schema via the tableParams:
{noformat}
CREATE TABLE `testavro`(
`test` string COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='hdfs://namenode:8020/tmp/test.avsc',
'kite.compression.type'='snappy',
'transient_lastDdlTime'='1427996456')
{noformat}
Hence for those tables the 'metatool updateLocation' will not help.
This is necessary in case like upgrade the namenode to HA where the absolute
paths have changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10324) Hive metatool should take table_param_key to allow for changes to avro serde's schema url key
[
https://issues.apache.org/jira/browse/HIVE-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493342#comment-14493342
]
Ferdinand Xu commented on HIVE-10324:
-
Hi [~szehon], I will take a look at it.
Hive metatool should take table_param_key to allow for changes to avro
serde's schema url key
-
Key: HIVE-10324
URL: https://issues.apache.org/jira/browse/HIVE-10324
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Ferdinand Xu
HIVE-3443 added support to change the serdeParams from 'metatool
updateLocation' command.
However, in avro it is possible to specify the schema via the tableParams:
{noformat}
CREATE TABLE `testavro`(
`test` string COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='hdfs://namenode:8020/tmp/test.avsc',
'kite.compression.type'='snappy',
'transient_lastDdlTime'='1427996456')
{noformat}
Hence for those tables the 'metatool updateLocation' will not help.
This is necessary in case like upgrade the namenode to HA where the absolute
paths have changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10324) Hive metatool should take table_param_key to allow for changes to avro serde's schema url key
[
https://issues.apache.org/jira/browse/HIVE-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493298#comment-14493298
]
Szehon Ho commented on HIVE-10324:
--
FYI [~Ferd] or [~dongc] : any interest in looking at this?
Hive metatool should take table_param_key to allow for changes to avro
serde's schema url key
-
Key: HIVE-10324
URL: https://issues.apache.org/jira/browse/HIVE-10324
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 1.1.0
Reporter: Szehon Ho
HIVE-3443 added support to change the serdeParams from 'metatool
updateLocation' command.
However, in avro it is possible to specify the schema via the tableParams:
{noformat}
CREATE TABLE `testavro`(
`test` string COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='hdfs://namenode:8020/tmp/test.avsc',
'kite.compression.type'='snappy',
'transient_lastDdlTime'='1427996456')
{noformat}
Hence for those tables the 'metatool updateLocation' will not help.
This is necessary in case like upgrade the namenode to HA where the absolute
paths have changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10324) Hive metatool should take table_param_key to allow for changes to avro serde's schema url key
[
https://issues.apache.org/jira/browse/HIVE-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493606#comment-14493606
]
Szehon Ho commented on HIVE-10324:
--
Hm, I think its supposed to update the MTable as well? Right now its updating
only MStorageDescriptor.
I am doing something like: metatool -updateLocation hdfs://namenode2:8020
hdfs://namenode:8020 -tablePropKey avro.schema.url -serdePropKey
avro.schema.url , with expectation that afterwards the table avro schema url
has namenode2.
Hive metatool should take table_param_key to allow for changes to avro
serde's schema url key
-
Key: HIVE-10324
URL: https://issues.apache.org/jira/browse/HIVE-10324
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Ferdinand Xu
HIVE-3443 added support to change the serdeParams from 'metatool
updateLocation' command.
However, in avro it is possible to specify the schema via the tableParams:
{noformat}
CREATE TABLE `testavro`(
`test` string COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
'avro.schema.url'='hdfs://namenode:8020/tmp/test.avsc',
'kite.compression.type'='snappy',
'transient_lastDdlTime'='1427996456')
{noformat}
Hence for those tables the 'metatool updateLocation' will not help.
This is necessary in case like upgrade the namenode to HA where the absolute
paths have changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
