wk888 edited a comment on issue #2834:
URL: https://github.com/apache/hudi/issues/2834#issuecomment-822313322
> @wk888 Did you sync table information of hoodie to HMS?
@yanghua yes .tables hudu_test_1_rt and hudu_test_1_ro is create
success by run_sync_tool.sh
./hudi-hive-sync/run_sync_tool.sh \
> --jdbc-url jdbc:hive2://11.11.11.11:30000 \
> --user hive \
> --pass hive \
> --partitioned-by dt \
> --partition-value-extractor
org.apache.hudi.hive.MultiPartKeysValueExtractor \
> --base-path ofs://qqqqqqqqq/user/wangkai/test_hudi_1 \
> --database test \
> --table hudu_test_1
-----------------------------------------logs-------------------------
2021-04-19 16:26:18,871 INFO [main] hive.HoodieHiveClient
(HoodieHiveClient.java:updateHiveSQL(366)) - Executing SQL create database if
not exists test
2021-04-19 16:26:18,995 WARN [main] hive.HiveSyncTool
(HiveSyncTool.java:syncHoodieTable(140)) - Unable to create database
org.apache.hudi.hive.HoodieHiveSyncException: Failed in executing SQL create
database if not exists test
at
org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:369)
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:137)
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:112)
at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:249)
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
compiling statement: No privilege 'Create' found for outputs { }
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253)
at
org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:313)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:253)
at
org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:367)
... 3 more
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
compiling statement: No privilege 'Create' found for outputs { }
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy30.executeStatementAsync(Unknown Source)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:535)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2021-04-19 16:26:19,050 INFO [main] fs.CHDFSInputStream
(CHDFSInputStream.java:close(73)) - close input stream for file:
/user/wangkai/test_hudi_1/.hoodie/20210419160955.deltacommit, inode: 336302421,
fd: 0, time-range-ms:[1618820779011 - 1618820779050], used-time-ms:39
2021-04-19 16:26:19,210 INFO [main] hive.HiveSyncTool
(HiveSyncTool.java:syncSchema(182)) - Hive table hudu_test_1_ro is not found.
Creating it
2021-04-19 16:26:19,215 INFO [main] hive.HoodieHiveClient
(HoodieHiveClient.java:createTable(262)) - Creating table with CREATE EXTERNAL
TABLE IF NOT EXISTS `test`.`hudu_test_1_ro`( `_hoodie_commit_time` string,
`_hoodie_commit_seqno` string, `_hoodie_record_key` string,
`_hoodie_partition_path` string, `_hoodie_file_name` string, `uuid` string,
`name` string, `ts` bigint) PARTITIONED BY (`dt` string) ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS
INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION
'ofs://qqqqqqq/user/wangkai/test_hudi_1'
2021-04-19 16:26:19,215 INFO [main] hive.HoodieHiveClient
(HoodieHiveClient.java:updateHiveSQL(366)) - Executing SQL CREATE EXTERNAL
TABLE IF NOT EXISTS `test`.`hudu_test_1_ro`( `_hoodie_commit_time` string,
`_hoodie_commit_seqno` string, `_hoodie_record_key` string,
`_hoodie_partition_path` string, `_hoodie_file_name` string, `uuid` string,
`name` string, `ts` bigint) PARTITIONED BY (`dt` string) ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS
INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION
'ofs://qqqqqqq/user/wangkai/test_hudi_1'
2021-04-19 16:26:19,864 INFO [main] hive.HiveSyncTool
(HiveSyncTool.java:syncHoodieTable(156)) - Schema sync complete. Syncing
partitions for hudu_test_1_ro
2021-04-19 16:26:19,864 INFO [main] hive.HiveSyncTool
(HiveSyncTool.java:syncHoodieTable(162)) - Last commit time synced was found to
be null
2021-04-19 16:26:19,864 INFO [main] common.AbstractSyncHoodieClient
(AbstractSyncHoodieClient.java:getPartitionsWrittenToSince(128)) - Last commit
time synced is not known, listing all partitions in
ofs://qqqqqqq/user/wangkai/test_hudi_1,FS
:com.qcloud.chdfs.fs.CHDFSHadoopFileSystemAdapter@3adcc812
2021-04-19 16:26:19,900 INFO [main] hive.HiveSyncTool
(HiveSyncTool.java:syncHoodieTable(164)) - Storage partitions scan complete.
Found 2
2021-04-19 16:26:19,921 INFO [main] hive.HiveSyncTool
(HiveSyncTool.java:syncPartitions(223)) - New Partitions [20210415, 20210419]
2021-04-19 16:26:19,921 INFO [main] hive.HoodieHiveClient
(HoodieHiveClient.java:addPartitionsToTable(120)) - Adding partitions 2 to
table hudu_test_1_ro
2021-04-19 16:26:19,922 INFO [main] hive.HoodieHiveClient
(HoodieHiveClient.java:updateHiveSQL(366)) - Executing SQL ALTER TABLE
`test`.`hudu_test_1_ro` ADD IF NOT EXISTS PARTITION (`dt`='20210415')
LOCATION 'ofs://qqqqqqq/user/wangkai/test_hudi_1/20210415' PARTITION
(`dt`='20210419') LOCATION 'ofs://qqqqqqq/user/wangkai/test_hudi_1/20210419'
2021-04-19 16:26:20,285 INFO [main] hive.HiveSyncTool
(HiveSyncTool.java:syncPartitions(226)) - Changed Partitions []
2021-04-19 16:26:20,285 INFO [main] hive.HoodieHiveClient
(HoodieHiveClient.java:updatePartitionsToTable(131)) - No partitions to change
for hudu_test_1_ro
2021-04-19 16:26:20,394 INFO [main] hive.HiveSyncTool
(HiveSyncTool.java:syncHoodieTable(169)) - Sync complete for hudu_test_1_ro
2021-04-19 16:26:20,394 INFO [main] hive.HiveSyncTool
(HiveSyncTool.java:syncHoodieTable(131)) - Trying to sync hoodie table
hudu_test_1_rt with base path ofs://qqqqqqq/user/wangkai/test_hudi_1 of type
MERGE_ON_READ
2021-04-19 16:26:20,394 INFO [main] hive.HoodieHiveClient
(HoodieHiveClient.java:updateHiveSQL(366)) - Executing SQL create database if
not exists test
2021-04-19 16:26:20,472 WARN [main] hive.HiveSyncTool
(HiveSyncTool.java:syncHoodieTable(140)) - Unable to create database
org.apache.hudi.hive.HoodieHiveSyncException: Failed in executing SQL create
database if not exists test
at
org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:369)
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:137)
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:114)
at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:249)
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
compiling statement: No privilege 'Create' found for outputs { }
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253)
at
org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:313)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:253)
at
org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:367)
... 3 more
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
compiling statement: No privilege 'Create' found for outputs { }
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy30.executeStatementAsync(Unknown Source)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:535)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2021-04-19 16:26:20,516 INFO [main] fs.CHDFSInputStream
(CHDFSInputStream.java:close(73)) - close input stream for file:
/user/wangkai/test_hudi_1/.hoodie/20210419160955.deltacommit, inode: 336302421,
fd: 0, time-range-ms:[1618820780481 - 1618820780516], used-time-ms:35
2021-04-19 16:26:20,522 INFO [main] hive.HiveSyncTool
(HiveSyncTool.java:syncSchema(182)) - Hive table hudu_test_1_rt is not found.
Creating it
2021-04-19 16:26:20,522 INFO [main] hive.HoodieHiveClient
(HoodieHiveClient.java:createTable(262)) - Creating table with CREATE EXTERNAL
TABLE IF NOT EXISTS `test`.`hudu_test_1_rt`( `_hoodie_commit_time` string,
`_hoodie_commit_seqno` string, `_hoodie_record_key` string,
`_hoodie_partition_path` string, `_hoodie_file_name` string, `uuid` string,
`name` string, `ts` bigint) PARTITIONED BY (`dt` string) ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS
INPUTFORMAT 'org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION 'ofs://qqqqqqq/user/wangkai/test_hudi_1'
2021-04-19 16:26:20,522 INFO [main] hive.HoodieHiveClient
(HoodieHiveClient.java:updateHiveSQL(366)) - Executing SQL CREATE EXTERNAL
TABLE IF NOT EXISTS `test`.`hudu_test_1_rt`( `_hoodie_commit_time` string,
`_hoodie_commit_seqno` string, `_hoodie_record_key` string,
`_hoodie_partition_path` string, `_hoodie_file_name` string, `uuid` string,
`name` string, `ts` bigint) PARTITIONED BY (`dt` string) ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS
INPUTFORMAT 'org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION 'ofs://qqqqqqq/user/wangkai/test_hudi_1'
2021-04-19 16:26:21,154 INFO [main] hive.HiveSyncTool
(HiveSyncTool.java:syncHoodieTable(156)) - Schema sync complete. Syncing
partitions for hudu_test_1_rt
2021-04-19 16:26:21,154 INFO [main] hive.HiveSyncTool
(HiveSyncTool.java:syncHoodieTable(162)) - Last commit time synced was found to
be null
2021-04-19 16:26:21,154 INFO [main] common.AbstractSyncHoodieClient
(AbstractSyncHoodieClient.java:getPartitionsWrittenToSince(128)) - Last commit
time synced is not known, listing all partitions in
ofs://qqqqqqq/user/wangkai/test_hudi_1,FS
:com.qcloud.chdfs.fs.CHDFSHadoopFileSystemAdapter@3adcc812
2021-04-19 16:26:21,169 INFO [main] hive.HiveSyncTool
(HiveSyncTool.java:syncHoodieTable(164)) - Storage partitions scan complete.
Found 2
2021-04-19 16:26:21,187 INFO [main] hive.HiveSyncTool
(HiveSyncTool.java:syncPartitions(223)) - New Partitions [20210415, 20210419]
2021-04-19 16:26:21,187 INFO [main] hive.HoodieHiveClient
(HoodieHiveClient.java:addPartitionsToTable(120)) - Adding partitions 2 to
table hudu_test_1_rt
2021-04-19 16:26:21,187 INFO [main] hive.HoodieHiveClient
(HoodieHiveClient.java:updateHiveSQL(366)) - Executing SQL ALTER TABLE
`test`.`hudu_test_1_rt` ADD IF NOT EXISTS PARTITION (`dt`='20210415')
LOCATION 'ofs://qqqqqqq/user/wangkai/test_hudi_1/20210415' PARTITION
(`dt`='20210419') LOCATION 'ofs://qqqqqqq/user/wangkai/test_hudi_1/20210419'
2021-04-19 16:26:21,574 INFO [main] hive.HiveSyncTool
(HiveSyncTool.java:syncPartitions(226)) - Changed Partitions []
2021-04-19 16:26:21,574 INFO [main] hive.HoodieHiveClient
(HoodieHiveClient.java:updatePartitionsToTable(131)) - No partitions to change
for hudu_test_1_rt
2021-04-19 16:26:21,648 INFO [main] hive.HiveSyncTool
(HiveSyncTool.java:syncHoodieTable(169)) - Sync complete for hudu_test_1_rt
2021-04-19 16:26:21,680 INFO [main] hive.metastore
(HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore,
current connections: 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
