[GitHub] [hudi] galain1994 commented on issue #5690: [SUPPORT] Flink stream write hudi, failed to checkpoint

Wed, 01 Jun 2022 05:10:14 -0700 


galain1994 commented on issue #5690:
URL: https://github.com/apache/hudi/issues/5690#issuecomment-1143525215

   I Got the same problem when using pg-cdc as source and hudi as destionation.
   The records in source table is about 4.3 million.
   flink configuration with hudi is something like:
       , 'table.type'= 'MERGE_ON_READ'
       , 'write.operation' = 'upsert'
       , 'write.precombine.field' = 'create_date'
       , 'write.ignore.failed' = 'true'
       , 'write.tasks' = '1'
       , 'compaction.tasks' = '1'
       , 'compaction.schedule.enabled' = 'true'
       , 'compaction.async.enabled' = 'true'
       , 'compaction.trigger.strategy' = 'num_or_time'
       , 'compaction.delta_commits' = '5'
       , 'compaction.delta_seconds' = '300'
       , 'compaction.max_memory' = '1024'
       , 'hive_sync.enable'= 'true'
       , 'hive_sync.mode'= 'hms'
   
![image](https://user-images.githubusercontent.com/10805116/171400515-a2699db6-fadd-40f7-af90-24f301edbe7e.png)
   
![image](https://user-images.githubusercontent.com/10805116/171400589-9171fb6f-2e9e-4c44-85d0-9d85fb0e7a29.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to