`AppenderatorImpl` holds all generated segment files until `abandonSegments()` is called. `IndexTask` works based on `AppenderatorImpl` and calls `abandonSegments()` once it pushes the segments to deep storage. This can cause an issue of too many open files if the indexTask is running with the perfect rollup mode. In the perfect rollup mode, the indexTask can't push segments in the middle of indexing. Instead, they should hold all segments until the very end of the indexing.
The way how `AppenderatorImpl` works makes sense for stream ingestion because the ingested data should be available for query immediately. However, batch tasks don't have to hold them. [ Full content available at: https://github.com/apache/incubator-druid/issues/6286 ] This message was relayed via gitbox.apache.org for [email protected]
