takaaki7 commented on issue #9967:
URL: https://github.com/apache/druid/issues/9967#issuecomment-1794157552
I am hoping for this issue to be resolved...
Looking at past PRs, the implementation and verification are mostly
completed and it appears to be quite effective (though conflicts may occurs).
I encountered challenges with the performance of batch ingestion with a
workload like the following:
Input: 1.3billion rows
Sharding type: hash
Sharding Number: 3000
maxBytesInMemory: 600MB
Servers: 144server (8core / 45GB RAM)
Time to complete: 2.5hours
I have set fine shards, and with maxBytesInMemory being only 600MB, we're
writing partial segments to disk at a very granular level like following log.
```
2023-11-06T05:45:29,443 INFO
[[partial_index_generate_event_xxxx_2022_parq_3_pknobigd_2023-11-06T05:23:36.710Z]-appenderator-persist]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed
in-memory data for
segment[event_xxxx_2022_parq_3_2022-01-01T00:00:00.000Z_2023-01-01T00:00:00.000Z_2023-11-06T04:31:09.889Z_316]
spill[15] to disk in [24] ms (172 rows).
2023-11-06T05:45:29,443 INFO
[[partial_index_generate_event_xxxx_2022_parq_3_pknobigd_2023-11-06T05:23:36.710Z]-appenderator-persist]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl -
Segment[event_xxxx_2022_parq_3_2022-01-01T00:00:00.000Z_2023-01-01T00:00:00.000Z_2023-11-06T04:31:09.889Z_2296]
hydrant[FireHydrant{queryable=null, count=15}] already swapped. Ignoring
request to persist.
2023-11-06T05:45:29,472 INFO
[[partial_index_generate_event_xxxx_2022_parq_3_pknobigd_2023-11-06T05:23:36.710Z]-appenderator-persist]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed
in-memory data for
segment[event_xxxx_2022_parq_3_2022-01-01T00:00:00.000Z_2023-01-01T00:00:00.000Z_2023-11-06T04:31:09.889Z_2296]
spill[16] to disk in [28] ms (785 rows).
2023-11-06T05:45:29,472 INFO
[[partial_index_generate_event_xxxx_2022_parq_3_pknobigd_2023-11-06T05:23:36.710Z]-appenderator-persist]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl -
Segment[event_xxxx_2022_parq_3_2022-01-01T00:00:00.000Z_2023-01-01T00:00:00.000Z_2023-11-06T04:31:09.889Z_2713]
hydrant[FireHydrant{queryable=null, count=15}] already swapped. Ignoring
request to persist.
2023-11-06T05:45:29,494 INFO
[[partial_index_generate_event_xxxx_2022_parq_3_pknobigd_2023-11-06T05:23:36.710Z]-appenderator-persist]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed
in-memory data for
segment[event_xxxx_2022_parq_3_2022-01-01T00:00:00.000Z_2023-01-01T00:00:00.000Z_2023-11-06T04:31:09.889Z_2713]
spill[16] to disk in [22] ms (54 rows).
```
Each time this spill down happens, serialization occurs, and dictionaries
have to be rebuilt.
If I raise the maxBytesInMemory any higher, the overhead for garbage
collection increases, and "GC overhead limit exceeded" occurs. Moreover, even
with current setting, GC threads using cpu heavily so that I can only execute
tasks at about half the parallelism compared to the number of cores.
It's difficult to estimate the effects, but Intuitively I expect that
merging this change could yield an improvement of more than tenfold.
If at least this refactoring PR were to be merged, it would become easier to
add our own extensions.
https://github.com/apache/druid/pull/12122
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
