[
https://issues.apache.org/jira/browse/BROOKLYN-526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16157842#comment-16157842
]
ASF GitHub Bot commented on BROOKLYN-526:
-----------------------------------------
Github user ahgittin commented on the issue:
https://github.com/apache/brooklyn-server/pull/791
@geomacy @aledsage I hadn't realized this either; javadocs and online
searches suggest the `fd.sync()` may be needed.
i'm curious whether this really is the issue -- at least on Mac -- as i've
done some testing and the `sync` seems to be very fast whereas `write` is what
takes the time. this is true even under contention: `write`s contend with
each other (8 files written concurrently take 8+ times as long as writing 1
file; if the OS cache is in play i'd expect CPU contention which with 4 cores
we'd expect 8 files to take twice as long, which is what happens if we do a
`System.arraycopy` instead of a `write`) and if there's background disk
activity it also slows down substantially. numbers and code below.
worth having this i agree, based on what i've read, though i have one worry
-- if `sync` is done _after_ the `close`, i always get the
`SyncFailedException`; and it looks like the code in this PR is synching after
close. we might want to understand that before merging?
i also came across the code at
https://github.com/apache/activemq-apollo/blob/trunk/apollo-util/src/main/scala/org/apache/activemq/apollo/util/IOHelper.java#L272
: here to get sync they included some custom JNI calls. this makes me wonder
whether -- although things acted nicely in my tests -- there aren't actually
any strong guarantees with java+mac, and you need that type of low-level
library?
although another part of me wonders if
https://issues.apache.org/jira/browse/BROOKLYN-526 was in fact caused by
something completely different (disk corruption?).
---
one thread:
```
t0 took 1.390 seconds to write 1,048,576,000 bytes to file
t0 took 0.010 seconds to sync
t0 took 0.000 seconds to close
```
8 threads:
```
t7 took 13.180 seconds to write 1,048,576,000 bytes to file
t7 took 0.099 seconds to sync
t7 took 0.000 seconds to close
```
code:
```
public class FileWrite {
public static void main(String[] args) throws Exception {
write("warmup");
int NUM_THREADS = 8;
for (int x=0; x<NUM_THREADS; x++) {
final String id = "t"+x;
new Thread() {
public void run() {
try {
write(id);
} catch (Exception e) {
e.printStackTrace();
}
}
}.start();
}
}
public static void write(String id) throws Exception {
Stopwatch t = Stopwatch.createStarted();
File file = File.createTempFile("deleteme", ".dat"); // put your
file here.
System.out.printf("%s took %.3f seconds to create %s%n", id,
t.elapsed(TimeUnit.NANOSECONDS) / 1e9, file.getAbsolutePath()); t.reset();
t.start();
FileOutputStream fos = new FileOutputStream(file);
byte[] bytes = new byte[32 * 1024];
// byte[] bytes2 = new byte[32 * 1024];
for (long l = 0; l < 1000 * 1024 * 1024; l += bytes.length) {
fos.write(bytes);
// System.arraycopy(bytes, 0, bytes2, 0, 32 * 1024);
}
System.out.printf("%s took %.3f seconds to write %,d bytes to
file%n", id, t.elapsed(TimeUnit.NANOSECONDS) / 1e9, file.length()); t.reset();
t.start();
fos.getFD().sync();
System.out.printf("%s took %.3f seconds to sync%n", id,
t.elapsed(TimeUnit.NANOSECONDS) / 1e9); t.reset(); t.start();
fos.close();
System.out.printf("%s took %.3f seconds to close%n", id,
t.elapsed(TimeUnit.NANOSECONDS) / 1e9); t.reset(); t.start();
File file2 = File.createTempFile("deleteme", ".dat"); // put your
file here.
java.nio.file.Files.move(file.toPath(), file2.toPath(),
StandardCopyOption.ATOMIC_MOVE);
System.out.printf("%s took %.3f seconds to move to %s%n", id,
t.elapsed(TimeUnit.NANOSECONDS) / 1e9, file2.getAbsolutePath()); t.reset();
t.start();
}
}
```
> Laptop crash (?) caused empty files in persisted state
> ------------------------------------------------------
>
> Key: BROOKLYN-526
> URL: https://issues.apache.org/jira/browse/BROOKLYN-526
> Project: Brooklyn
> Issue Type: Bug
> Affects Versions: 0.11.0
> Reporter: Aled Sage
>
> My mac laptop crashed while Brooklyn was running. After restart the mac, and
> restarting Brooklyn, I saw exceptions like that below many times:
> {noformat}
> 2017-08-04 09:23:52,889 WARN 122 o.a.b.c.m.r.RebindExceptionHandlerImpl
> [ooklyn-persister] Rebind: continuing after problem loading memento: memento
> ielugq12x0 loading raw error
> org.apache.brooklyn.util.exceptions.PropagatedRuntimeException:
> SAXParseException: Premature end of file.
> at
> org.apache.brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:129)[143:org.apache.brooklyn.utils-common:0.12.0.SNAPSHOT]
> at
> org.apache.brooklyn.util.core.xstream.XmlUtil.xpath(XmlUtil.java:78)[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
> at
> org.apache.brooklyn.util.core.xstream.XmlUtil.xpathHandlingIllegalChars(XmlUtil.java:100)[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
> at
> org.apache.brooklyn.util.core.xstream.XmlUtil.xpathHandlingIllegalChars(XmlUtil.java:87)[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
> at
> org.apache.brooklyn.core.mgmt.persist.BrooklynMementoPersisterToObjectStore$2.visit(BrooklynMementoPersisterToObjectStore.java:335)[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
> at
> org.apache.brooklyn.core.mgmt.persist.BrooklynMementoPersisterToObjectStore$1VisitorWrapper.run(BrooklynMementoPersisterToObjectStore.java:528)[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_144]
> at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_144]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_144]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_144]
> at java.lang.Thread.run(Thread.java:748)[:1.8.0_144]
> Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1;
> Premature end of file.
> at
> com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)[:1.8.0_144]
> at
> com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)[:1.8.0_144]
> at
> org.apache.brooklyn.util.core.xstream.XmlUtil.xpath(XmlUtil.java:69)[122:org.apache.brooklyn.core:0.12.0.SNAPSHOT]
> ... 9 more
> {noformat}
> Looking in the persisted state directory, there are a bunch of size-zero
> files (for 34 enrichers, 4 feeds, 6 locations and 2 policies)!
> The timestamps for these zero-size files are the newest in the persistence
> directories.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
