openinx commented on pull request #2666: URL: https://github.com/apache/iceberg/pull/2666#issuecomment-867636720
> @openinx, sorry for the delay. I'll make some time to review this. > > I also know that while I was out there were a few PRs that I wasn't able to help move along. If you have a list of PRs that are important for you to get in, please send them to me and I'll make time to get them reviewed as I can. Thank you for being patient with me! @rdblue , thanks for your time. All the PRs that were published by me are here : https://github.com/apache/iceberg/pulls/openinx . There are three parts : 1. The first part is about improvement for iceberg+flink module, I think this PR is the highest priority. Others are related to flink CDC write path, let's discuss them in the second parts. 2. Write & analysis the change log events in format v2 iceberg table. After #2410, we could ingest change log events into apache iceberg by pure flink SQL now. But we still have other important issues that need to be addressed: a. How to ensure the stability of streaming jobs ? Currently, the most important PR is : https://github.com/apache/iceberg/pull/2680 . There are some other problems. For example, random change log events may cause a large number of parquet writers to be opened in a checkpoint, and eventually cause OOM of the job. I will open a specific issue for that. b. How to implement the compaction action for v2. As we know we are implementing the minor compaction (I mean translate the equality deletes to pos-deletes ? ) , but I think we may could get this [major compaction](https://github.com/apache/iceberg/pull/2303) merged first (Of course , the current patch still has few concerns that need to be addressed, I will do) . I also had a temporary fix for https://github.com/apache/iceberg/issues/2308, and will publish it to the apache repo for reviewing. 3. Aliyun OSS + DLF integration work. I had an opened PR for [aliyun OSS](https://github.com/apache/iceberg/pull/2230) and an pending PR for[ aliyun DLF integration](https://github.com/openinx/incubator-iceberg/pull/5) (now it's in my personal repo) . Let's put these PRs aside for now, and I will try to split these PRs appropriately so that we can better review them in the future. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
