gianm commented on a change in pull request #6677: FileUtils: Sync directory
entry too on writeAtomically.
URL: https://github.com/apache/incubator-druid/pull/6677#discussion_r237203798
##########
File path:
core/src/main/java/org/apache/druid/java/util/common/CompressionUtils.java
##########
@@ -78,16 +78,12 @@ public static long zip(File directory, File outputZipFile,
boolean fsync) throws
log.warn("No .zip suffix[%s], putting files from [%s] into it anyway.",
outputZipFile, directory);
}
- try (final FileOutputStream out = new FileOutputStream(outputZipFile)) {
- long bytes = zip(directory, out);
-
- // For explanation of why fsyncing here is a good practice:
- //
https://github.com/apache/incubator-druid/pull/5187#pullrequestreview-85188984
- if (fsync) {
- out.getChannel().force(true);
+ if (fsync) {
+ return FileUtils.writeAtomically(outputZipFile, out -> zip(directory,
out));
+ } else {
+ try (final FileOutputStream out = new FileOutputStream(outputZipFile)) {
Review comment:
> We are constantly suffering GC and apparent memory leak problems with
pretty much all Druid node types, that only gets worse with Druid updates, but
we cannot understand the reasons. We only try to make incremental improvements
on this front, such as #6370 and other PRs.
Hm interesting. I did a `jmap -histo` on a historical of a fairly active
Druid cluster (~150 queries/sec, ~5000 segments/server) and I see about 900
WeakReferences, 200 SoftReferences, _no_ PhantomReferences, and 1100
Finalizers. It is typically doing 0.5-2sec of GC time per minute (based on the
jvm/gc/time metric). It has been running for a month without being restarted,
and avg heap occupancy / gc time hasn't changed much in that time; so in that
sense, I don't see signs of memory leaks. GC and heap options are `-XX:+UseG1GC
-XX:MaxGCPauseMillis=500 -Xms24g -Xmx24g`. Is your experience different from
this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
