leventov commented on a change in pull request #8165: [Improvement] DataSegment
intern improvement (reduce 60% memory consume on coordinator)
URL: https://github.com/apache/incubator-druid/pull/8165#discussion_r307830766
##########
File path: core/src/main/java/org/apache/druid/timeline/DataSegment.java
##########
@@ -76,9 +76,22 @@
@Inject(optional = true) @PruneLoadSpec boolean pruneLoadSpec = false;
}
+ public static DataSegment intern(DataSegment dataSegment, boolean
updateLoadSpec)
+ {
+ DataSegment result = DATA_SEGMENT_INTERNER.intern(dataSegment);
+ if (updateLoadSpec) {
+ result.dimensions = dataSegment.dimensions;
+ result.metrics = dataSegment.metrics;
+ result.loadSpec = dataSegment.loadSpec;
+ }
+ return result;
+ }
+
+
private static final Interner<String> STRING_INTERNER =
Interners.newWeakInterner();
private static final Interner<List<String>> DIMENSIONS_INTERNER =
Interners.newWeakInterner();
private static final Interner<List<String>> METRICS_INTERNER =
Interners.newWeakInterner();
+ private static final Interner<DataSegment> DATA_SEGMENT_INTERNER =
Interners.newWeakInterner();
Review comment:
This humongous Map with millions of weak references would be a problem for
GC in itself: see #6357 for context.
You should adopt the design with `BatchServerInverntoryView` and
`SQLSegmentMetadataManager` probing into each other's memory, similarly to what
is explained here:
https://github.com/apache/incubator-druid/issues/7395#issue-427863665
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
