thexiay opened a new issue, #7724: URL: https://github.com/apache/inlong/issues/7724
### Description # Movtivation Nowadays data ingesting into iceberg has some problems， the memory # Proposed Changes 1. 当限流时会影响chk的触发吗，插入checkpoint barrier下游会长时间没法处理导致chk超时吗？ 答：会有一些影响。 a. 下游处理数据被限流，导致buffer积压，导致反压，上游source从split拉取数据会被延缓甚至暂停。 b. barrier的插入是在mailbox中执行CheckpointEvent时，同步的向下游所有节点插入barrier。 c. barrier的发送走的是Event这条路，不会因为buffer内存不够被暂停，但是还是会排在data后面 d. 也就是说当触发chk时，在chk超时之前，下游至少还要处理的数据量为 "网络中积压的buffer / 平均每一条数据量 = 总积压条数"，只要能在限定时间内处理完这么多的积压条数就可以完成chk了 # Example Usage # Rollout/Adoption Plan 1. Impelement rate limit. 2. Rate limit can dynamic adjustment. 3. Replace `insertMap` (determine whether to put it into position delete records in per checkpoint ) from heap map to rocksdb map. ### InLong Component InLong Sort ### Are you willing to submit PR? - [X] Yes, I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
