maytasm commented on a change in pull request #10740:
URL: https://github.com/apache/druid/pull/10740#discussion_r556977817
##########
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java
##########
@@ -617,6 +625,21 @@ public Object call() throws IOException
// NB: The rows are still in memory until they're done persisting, but we
only count rows in active indexes.
rowsCurrentlyInMemory.addAndGet(-numPersistedRows);
bytesCurrentlyInMemory.addAndGet(-bytesPersisted);
+
+ log.info("Persisted rows[%,d] and bytes[%,d]", numPersistedRows,
bytesPersisted);
+
+ // bytesCurrentlyInMemory can change while persisting due to concurrent
ingestion.
+ // Hence, we use bytesInMemoryBeforePersist to determine the change of
this persist
+ if (bytesInMemoryBeforePersist - bytesPersisted > maxBytesTuningConfig) {
+ // We are still over maxBytesTuningConfig even after persisting.
+ // This means that we ran out of all available memory to ingest (due to
overheads created as part of ingestion)
+ log.makeAlert("Persist no longer free up memory")
+ .addData("dataSource", schema.getDataSource())
+ .addData("numSinks", sinks.size())
+ .addData("numHydrantsAcrossAllSinks",
sinks.values().stream().mapToInt(Iterables::size).sum())
+ .emit();
+ throw new RuntimeException("Ingestion run out of available memory as
persist can no longer free up any memory.");
+ }
Review comment:
Done. Added more details to error message. Moved stats into error
message. Added totalRow stat.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
