Archie-selfless commented on issue #6572:
URL: https://github.com/apache/hudi/issues/6572#issuecomment-1235265477
When I switch the sycn_mode to jdbc, I got the same Exception. However, I
can run the HQL below which caught from spark logs to complete the hive table.
```sql
ALTER TABLE `default`.`hudi_mysql_table2`
ADD IF NOT EXISTS
PARTITION (`u_name`='A') LOCATION
'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=A'
PARTITION (`u_name`='B') LOCATION
'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=B'
PARTITION (`u_name`='C') LOCATION
'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=C' ;
```
After the HQL, everything goes smoothly! (i.e. Query the hudi-table by
Presto...)
** ↓↓ The core logs syncing as jdbc mode ↓↓ **
```bash
22/09/02 16:57:27 INFO HiveSyncTool: Storage partitions scan complete. Found
3
22/09/02 16:57:27 INFO HiveSyncTool: New Partitions [u_name=A, u_name=B,
u_name=C]
22/09/02 16:57:27 INFO QueryBasedDDLExecutor: Adding partitions 3 to table
hudi_mysql_table001
22/09/02 16:57:27 INFO QueryBasedDDLExecutor: Executing SQL ALTER TABLE
`default`.`hudi_mysql_table001` ADD IF NOT EXISTS PARTITION (`u_name`='A')
LOCATION 'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=A'
PARTITION (`u_name`='B') LOCATION
'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=B' PARTITION
(`u_name`='C') LOCATION
'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=C'
22/09/02 16:57:28 INFO metastore: Closed a connection to metastore, current
connections: 0
22/09/02 16:57:28 ERROR HoodieDeltaStreamer: Shutting down delta-sync due to
exception
org.apache.hudi.exception.HoodieException: Could not sync using the meta
sync class org.apache.hudi.hive.HiveSyncTool
at
org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:58)
at
org.apache.hudi.utilities.deltastreamer.DeltaSync.runMetaSync(DeltaSync.java:716)
at
org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:634)
at
org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:335)
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:687)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieException: Got runtime exception
when hive syncing hudi_mysql_table001
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:144)
at
org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:56)
... 8 more
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to sync
partitions for table hudi_mysql_table001
at
org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:340)
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:232)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:153)
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:141)
... 9 more
...
...
Caused by: MetaException(message:java.io.IOException: Got exception:
java.io.IOException java.lang.NullPointerException)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result$add_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result$add_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_add_partitions_req(ThriftHiveMetastore.java:1911)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.add_partitions_req(ThriftHiveMetastore.java:1898)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.add_partitions(HiveMetaStoreClient.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
at com.sun.proxy.$Proxy26.add_partitions(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2327)
at com.sun.proxy.$Proxy26.add_partitions(Unknown Source)
at
org.apache.hadoop.hive.ql.metadata.Hive.createPartitions(Hive.java:2103)
... 21 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
