trushev commented on PR #7626:
URL: https://github.com/apache/hudi/pull/7626#issuecomment-1396468460
@TengHuo
I tried the following workload with MOR table, 2000 partitions and
compaction (checkpoint here triggers compaction)
```java
public class
trushev commented on PR #7626:
URL: https://github.com/apache/hudi/pull/7626#issuecomment-1396449286
> Thanks for great effort @trushev, can we revert the changes/refactoring
for `#performWriteOperation`, it is not related with this issue, and we can
address it in another PR.
>
> We
trushev commented on PR #7626:
URL: https://github.com/apache/hudi/pull/7626#issuecomment-1386555492
@danny0405 Could you pls take a look again. New solution:
- Replaced `Map>` with `Map`
- All handles are definitely closed by finally section with
`closeGracefully()`
--
This is an
trushev commented on PR #7626:
URL: https://github.com/apache/hudi/pull/7626#issuecomment-1386515641
> The thing which I want to share is that caching write handles could take a
lot of memory, because each handle obtains an instance of `HoodieTable`, and
there is a `viewManager` in every
trushev commented on PR #7626:
URL: https://github.com/apache/hudi/pull/7626#issuecomment-1384831292
> May I ask if we can lazy load `HoodieTableFileSystemView` in
`PriorityBasedFileSystemView` when creating `FlinkAppendHandle`? It can also
reduce memory usage for active partitions.
trushev commented on PR #7626:
URL: https://github.com/apache/hudi/pull/7626#issuecomment-1376652385
> Nice catch, @trushev , curious why the closed handle is also taking huge
resource, we may need to figure it out first.
>
> But I still think the change is valid.
Thank you
