maytasm commented on a change in pull request #10740:
URL: https://github.com/apache/druid/pull/10740#discussion_r557006534
##########
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java
##########
@@ -617,6 +625,21 @@ public Object call() throws IOException
// NB: The rows are still in memory until they're done persisting, but we
only count rows in active indexes.
rowsCurrentlyInMemory.addAndGet(-numPersistedRows);
bytesCurrentlyInMemory.addAndGet(-bytesPersisted);
+
+ log.info("Persisted rows[%,d] and bytes[%,d]", numPersistedRows,
bytesPersisted);
+
+ // bytesCurrentlyInMemory can change while persisting due to concurrent
ingestion.
+ // Hence, we use bytesInMemoryBeforePersist to determine the change of
this persist
+ if (bytesInMemoryBeforePersist - bytesPersisted > maxBytesTuningConfig) {
Review comment:
I also don't know what an ideal cutoff point would be. I think the
thrashing should only happens for a short duration assuming the data has "many
intervals/sinks" or "many columns" characteristic as the overhead increase
after each persist should increase quickly.
I added more tests to make sure it's well behaved.
I didn't add any config/opt in since most likely anything that will fail
because of this would have fail because of OOM too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
