Hi,
Has anyone successfully used Java Flight Recorder (JFR) with Spark
Streaming on Oracle Java 8? JFR works for me on batch jobs but not with
Streaming.
I'm running my streaming job on Amazon EMR. I have enabled Java Flight
Recorder (JFR) to profile CPU usage. But at the end of the job, the JFR
output files get deleted and are not available. This is the error in the
JFR trace logs:
[jfr][DEBUG][880.590] Release chunk
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_40_45_8823_0.jfr.part ref
count now 0
[jfr][DEBUG][880.590] Repository chunk
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_40_45_8823_0.jfr.part
deleted.
[jfr][DEBUG][880.590] Repository chunk
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_40_45_8823_0.jfr deleted.
[jfr][INFO ][880.593] Stopped recording [id=1,
name=dump-on-exit-clone-of-0, start=Wed Feb 19 17:26:06 UTC 2020, end=Wed
Feb 19 17:40:45 UTC 2020]
[jfr][ERROR][880.593] Could not copy to
/tmp/hotspot-pid-8823-id-0-2020_02_19_17_40_45.jfr
[jfr][ERROR][880.593] java.io.FileNotFoundException:No chunks
[jfr][ERROR][880.593]
oracle.jrockit.jfr.ChunksChannel.<init>(ChunksChannel.java:26)
[jfr][ERROR][880.593]
oracle.jrockit.jfr.Recording.copyTo(Recording.java:404)
[jfr][ERROR][880.593]
oracle.jrockit.jfr.JFRImpl.dumpOnExit(JFRImpl.java:802)
[jfr][ERROR][880.593]
oracle.jrockit.jfr.JFRImpl.dumpOnExit(JFRImpl.java:789)
[jfr][ERROR][880.593]
oracle.jrockit.jfr.JFRImpl.destroy(JFRImpl.java:767)
[jfr][ERROR][880.593] oracle.jrockit.jfr.JFR$2.run(JFR.java:208)
[jfr][ERROR][880.593] java.lang.Thread.run(Thread.java:748)
[jfr][ERROR][880.593] Could not delete
/tmp/hotspot-pid-8823-id-0-2020_02_19_17_40_45.jfr
[jfr][ERROR][880.593] java.io.FileNotFoundException:No chunks
[jfr][ERROR][880.593] oracle.jrockit.jfr
The rest of the trace can be found in the attachment (stdout.txt).
Also, I've not changed the default value of "
spark.streaming.stopGracefullyOnShutdown" (its default value is false)
The Spark driver and executor JVMs are launched with the following options:
-XX:+UnlockCommercialFeatures -XX:+FlightRecorder
-XX:FlightRecorderOptions=defaultrecording=true,disk=true,repository=/tmp,maxage=2h,dumponexit=true,dumponexitpath=/tmp,loglevel=trace
Pramod
[jfr][INFO ][0.000] JFR log level set. Log level now at [TRACE]
[jfr][TRACE][0.001] Jfr::initialize_stage_one completed
[jfr][TRACE][0.093] Loaded JFR library
[jfr][INFO ][0.125] Using /mnt/tmp/2020_02_19_17_26_04_8823 as Flight Recorder
repository.
[jfr][TRACE][0.157] Default settings loaded.
[jfr][TRACE][0.164] VMJFR created.
[jfr][TRACE][0.239] classFileLoadHook called for java/lang/Throwable
[jfr][TRACE][0.239] instrumenting java.lang.Throwable
[jfr][TRACE][0.245] classFileLoadHook called for java/lang/Error
[jfr][TRACE][0.245] instrumenting java.lang.Error
[jfr][TRACE][0.247] RetransformClasses successful
[jfr][TRACE][0.270] Processing instrumentation class: class
oracle.jrockit.jfr.FileInputStreamInstrumentor
[jfr][TRACE][0.275] Inliner processing method read()I
[jfr][TRACE][0.275] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.FileInputStreamInstrumentor.read()I
[jfr][TRACE][0.275] Inlining call to read()I
[jfr][TRACE][0.275] Inlining done
[jfr][TRACE][0.275] Inlining call to read()I
[jfr][TRACE][0.275] Inlining done
[jfr][TRACE][0.275] Inliner processing method read([B)I
[jfr][TRACE][0.275] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.FileInputStreamInstrumentor.read([B)I
[jfr][TRACE][0.276] Inlining call to read([B)I
[jfr][TRACE][0.276] Inlining done
[jfr][TRACE][0.276] Inlining call to read([B)I
[jfr][TRACE][0.276] Inlining done
[jfr][TRACE][0.276] Inliner processing method read([BII)I
[jfr][TRACE][0.276] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.FileInputStreamInstrumentor.read([BII)I
[jfr][TRACE][0.276] Inlining call to read([BII)I
[jfr][TRACE][0.276] Inlining done
[jfr][TRACE][0.276] Inlining call to read([BII)I
[jfr][TRACE][0.276] Inlining done
[jfr][TRACE][0.279] Deleting read([B)I
[jfr][TRACE][0.279] Deleting read()I
[jfr][TRACE][0.279] Deleting read([BII)I
[jfr][TRACE][0.280] Copying method: read()I
[jfr][TRACE][0.280] with mapper:
{oracle/jrockit/jfr/FileInputStreamInstrumentor=java/io/FileInputStream}
[jfr][TRACE][0.282] Copying method: read([B)I
[jfr][TRACE][0.282] with mapper:
{oracle/jrockit/jfr/FileInputStreamInstrumentor=java/io/FileInputStream}
[jfr][TRACE][0.282] Copying method: read([BII)I
[jfr][TRACE][0.282] with mapper:
{oracle/jrockit/jfr/FileInputStreamInstrumentor=java/io/FileInputStream}
[jfr][TRACE][0.284] Processing instrumentation class: class
oracle.jrockit.jfr.FileOutputStreamInstrumentor
[jfr][TRACE][0.285] Inliner processing method write(I)V
[jfr][TRACE][0.285] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.FileOutputStreamInstrumentor.write(I)V
[jfr][TRACE][0.285] Inlining call to write(I)V
[jfr][TRACE][0.285] Inlining done
[jfr][TRACE][0.285] Inlining call to write(I)V
[jfr][TRACE][0.285] Inlining done
[jfr][TRACE][0.286] Inliner processing method write([B)V
[jfr][TRACE][0.286] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.FileOutputStreamInstrumentor.write([B)V
[jfr][TRACE][0.286] Inlining call to write([B)V
[jfr][TRACE][0.286] Inlining done
[jfr][TRACE][0.286] Inlining call to write([B)V
[jfr][TRACE][0.286] Inlining done
[jfr][TRACE][0.286] Inliner processing method write([BII)V
[jfr][TRACE][0.286] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.FileOutputStreamInstrumentor.write([BII)V
[jfr][TRACE][0.286] Inlining call to write([BII)V
[jfr][TRACE][0.286] Inlining done
[jfr][TRACE][0.286] Inlining call to write([BII)V
[jfr][TRACE][0.286] Inlining done
[jfr][TRACE][0.287] Deleting write([B)V
[jfr][TRACE][0.287] Deleting write([BII)V
[jfr][TRACE][0.287] Deleting write(I)V
[jfr][TRACE][0.288] Copying method: write(I)V
[jfr][TRACE][0.288] with mapper:
{oracle/jrockit/jfr/FileOutputStreamInstrumentor=java/io/FileOutputStream}
[jfr][TRACE][0.289] Copying method: write([B)V
[jfr][TRACE][0.289] with mapper:
{oracle/jrockit/jfr/FileOutputStreamInstrumentor=java/io/FileOutputStream}
[jfr][TRACE][0.289] Copying method: write([BII)V
[jfr][TRACE][0.289] with mapper:
{oracle/jrockit/jfr/FileOutputStreamInstrumentor=java/io/FileOutputStream}
[jfr][TRACE][0.291] Processing instrumentation class: class
oracle.jrockit.jfr.RandomAccessFileInstrumentor
[jfr][TRACE][0.294] Inliner processing method read()I
[jfr][TRACE][0.294] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.RandomAccessFileInstrumentor.read()I
[jfr][TRACE][0.294] Inlining call to read()I
[jfr][TRACE][0.294] Inlining done
[jfr][TRACE][0.294] Inlining call to read()I
[jfr][TRACE][0.294] Inlining done
[jfr][TRACE][0.294] Inliner processing method read([B)I
[jfr][TRACE][0.294] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.RandomAccessFileInstrumentor.read([B)I
[jfr][TRACE][0.294] Inlining call to read([B)I
[jfr][TRACE][0.294] Inlining done
[jfr][TRACE][0.294] Inlining call to read([B)I
[jfr][TRACE][0.294] Inlining done
[jfr][TRACE][0.294] Inliner processing method read([BII)I
[jfr][TRACE][0.294] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.RandomAccessFileInstrumentor.read([BII)I
[jfr][TRACE][0.294] Inlining call to read([BII)I
[jfr][TRACE][0.294] Inlining done
[jfr][TRACE][0.294] Inlining call to read([BII)I
[jfr][TRACE][0.294] Inlining done
[jfr][TRACE][0.295] Inliner processing method write(I)V
[jfr][TRACE][0.295] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.RandomAccessFileInstrumentor.write(I)V
[jfr][TRACE][0.295] Inlining call to write(I)V
[jfr][TRACE][0.295] Inlining done
[jfr][TRACE][0.295] Inlining call to write(I)V
[jfr][TRACE][0.295] Inlining done
[jfr][TRACE][0.295] Inliner processing method write([B)V
[jfr][TRACE][0.295] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.RandomAccessFileInstrumentor.write([B)V
[jfr][TRACE][0.295] Inlining call to write([B)V
[jfr][TRACE][0.295] Inlining done
[jfr][TRACE][0.295] Inlining call to write([B)V
[jfr][TRACE][0.295] Inlining done
[jfr][TRACE][0.295] Inliner processing method write([BII)V
[jfr][TRACE][0.295] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.RandomAccessFileInstrumentor.write([BII)V
[jfr][TRACE][0.295] Inlining call to write([BII)V
[jfr][TRACE][0.295] Inlining done
[jfr][TRACE][0.295] Inlining call to write([BII)V
[jfr][TRACE][0.295] Inlining done
[jfr][TRACE][0.296] Deleting write([BII)V
[jfr][TRACE][0.296] Deleting write(I)V
[jfr][TRACE][0.296] Deleting write([B)V
[jfr][TRACE][0.296] Deleting read([BII)I
[jfr][TRACE][0.296] Deleting read([B)I
[jfr][TRACE][0.296] Deleting read()I
[jfr][TRACE][0.299] Copying method: read()I
[jfr][TRACE][0.299] with mapper:
{oracle/jrockit/jfr/RandomAccessFileInstrumentor=java/io/RandomAccessFile}
[jfr][TRACE][0.300] Copying method: read([B)I
[jfr][TRACE][0.300] with mapper:
{oracle/jrockit/jfr/RandomAccessFileInstrumentor=java/io/RandomAccessFile}
[jfr][TRACE][0.300] Copying method: read([BII)I
[jfr][TRACE][0.300] with mapper:
{oracle/jrockit/jfr/RandomAccessFileInstrumentor=java/io/RandomAccessFile}
[jfr][TRACE][0.301] Copying method: write(I)V
[jfr][TRACE][0.301] with mapper:
{oracle/jrockit/jfr/RandomAccessFileInstrumentor=java/io/RandomAccessFile}
[jfr][TRACE][0.302] Copying method: write([B)V
[jfr][TRACE][0.302] with mapper:
{oracle/jrockit/jfr/RandomAccessFileInstrumentor=java/io/RandomAccessFile}
[jfr][TRACE][0.303] Copying method: write([BII)V
[jfr][TRACE][0.303] with mapper:
{oracle/jrockit/jfr/RandomAccessFileInstrumentor=java/io/RandomAccessFile}
[jfr][TRACE][0.305] Processing instrumentation class: class
oracle.jrockit.jfr.FileChannelImplInstrumentor
[jfr][TRACE][0.312] Inliner processing method read(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.312] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.FileChannelImplInstrumentor.read(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.312] Inlining call to read(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.313] Inlining done
[jfr][TRACE][0.313] Inlining call to read(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.313] Inlining done
[jfr][TRACE][0.313] Inliner processing method read(Ljava/nio/ByteBuffer;J)I
[jfr][TRACE][0.313] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.FileChannelImplInstrumentor.read(Ljava/nio/ByteBuffer;J)I
[jfr][TRACE][0.313] Inlining call to read(Ljava/nio/ByteBuffer;J)I
[jfr][TRACE][0.313] Inlining done
[jfr][TRACE][0.313] Inlining call to read(Ljava/nio/ByteBuffer;J)I
[jfr][TRACE][0.313] Inlining done
[jfr][TRACE][0.313] Inliner processing method read([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.313] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.FileChannelImplInstrumentor.read([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.313] Inlining call to read([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.313] Inlining done
[jfr][TRACE][0.314] Inlining call to read([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.314] Inlining done
[jfr][TRACE][0.314] Inliner processing method write(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.314] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.FileChannelImplInstrumentor.write(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.314] Inlining call to write(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.314] Inlining done
[jfr][TRACE][0.314] Inlining call to write(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.314] Inlining done
[jfr][TRACE][0.314] Inliner processing method write(Ljava/nio/ByteBuffer;J)I
[jfr][TRACE][0.314] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.FileChannelImplInstrumentor.write(Ljava/nio/ByteBuffer;J)I
[jfr][TRACE][0.314] Inlining call to write(Ljava/nio/ByteBuffer;J)I
[jfr][TRACE][0.315] Inlining done
[jfr][TRACE][0.315] Inlining call to write(Ljava/nio/ByteBuffer;J)I
[jfr][TRACE][0.315] Inlining done
[jfr][TRACE][0.315] Inliner processing method write([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.315] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.FileChannelImplInstrumentor.write([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.315] Inlining call to write([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.315] Inlining done
[jfr][TRACE][0.315] Inlining call to write([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.315] Inlining done
[jfr][TRACE][0.319] Deleting write(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.319] Deleting write(Ljava/nio/ByteBuffer;J)I
[jfr][TRACE][0.319] Deleting write([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.319] Deleting read(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.319] Deleting read([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.319] Deleting read(Ljava/nio/ByteBuffer;J)I
[jfr][TRACE][0.330] Copying method: read(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.330] with mapper:
{oracle/jrockit/jfr/FileChannelImplInstrumentor=sun/nio/ch/FileChannelImpl}
[jfr][TRACE][0.333] Copying method: read(Ljava/nio/ByteBuffer;J)I
[jfr][TRACE][0.333] with mapper:
{oracle/jrockit/jfr/FileChannelImplInstrumentor=sun/nio/ch/FileChannelImpl}
[jfr][TRACE][0.334] Copying method: read([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.334] with mapper:
{oracle/jrockit/jfr/FileChannelImplInstrumentor=sun/nio/ch/FileChannelImpl}
[jfr][TRACE][0.338] Copying method: write(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.338] with mapper:
{oracle/jrockit/jfr/FileChannelImplInstrumentor=sun/nio/ch/FileChannelImpl}
[jfr][TRACE][0.341] Copying method: write(Ljava/nio/ByteBuffer;J)I
[jfr][TRACE][0.341] with mapper:
{oracle/jrockit/jfr/FileChannelImplInstrumentor=sun/nio/ch/FileChannelImpl}
[jfr][TRACE][0.343] Copying method: write([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.343] with mapper:
{oracle/jrockit/jfr/FileChannelImplInstrumentor=sun/nio/ch/FileChannelImpl}
[jfr][TRACE][0.351] Processing instrumentation class: class
oracle.jrockit.jfr.SocketInputStreamInstrumentor
[jfr][TRACE][0.353] Inliner processing method read([BIII)I
[jfr][TRACE][0.353] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.SocketInputStreamInstrumentor.read([BIII)I
[jfr][TRACE][0.353] Inlining call to read([BIII)I
[jfr][TRACE][0.353] Inlining done
[jfr][TRACE][0.353] Inlining call to read([BIII)I
[jfr][TRACE][0.353] Inlining done
[jfr][TRACE][0.354] Deleting read([BIII)I
[jfr][TRACE][0.354] Copying method: read([BIII)I
[jfr][TRACE][0.354] with mapper:
{oracle/jrockit/jfr/SocketInputStreamInstrumentor=java/net/SocketInputStream,
oracle/jrockit/jfr/SocketInputStreamInstrumentor$InetAddressHolder=java/net/InetAddress$InetAddressHolder,
oracle/jrockit/jfr/SocketInputStreamInstrumentor$AbstractPlainSocketImpl=java/net/AbstractPlainSocketImpl,
oracle/jrockit/jfr/SocketInputStreamInstrumentor$InetAddress=java/net/InetAddress}
[jfr][TRACE][0.361] Processing instrumentation class: class
oracle.jrockit.jfr.SocketOutputStreamInstrumentor
[jfr][TRACE][0.362] Inliner processing method socketWrite([BII)V
[jfr][TRACE][0.362] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.SocketOutputStreamInstrumentor.socketWrite([BII)V
[jfr][TRACE][0.362] Inlining call to socketWrite([BII)V
[jfr][TRACE][0.362] Inlining done
[jfr][TRACE][0.362] Inlining call to socketWrite([BII)V
[jfr][TRACE][0.363] Inlining done
[jfr][TRACE][0.363] Deleting socketWrite([BII)V
[jfr][TRACE][0.363] Copying method: socketWrite([BII)V
[jfr][TRACE][0.363] with mapper:
{oracle/jrockit/jfr/SocketOutputStreamInstrumentor$InetAddressHolder=java/net/InetAddress$InetAddressHolder,
oracle/jrockit/jfr/SocketOutputStreamInstrumentor=java/net/SocketOutputStream,
oracle/jrockit/jfr/SocketOutputStreamInstrumentor$AbstractPlainSocketImpl=java/net/AbstractPlainSocketImpl,
oracle/jrockit/jfr/SocketOutputStreamInstrumentor$InetAddress=java/net/InetAddress}
[jfr][TRACE][0.368] Processing instrumentation class: class
oracle.jrockit.jfr.SocketChannelImplInstrumentor
[jfr][TRACE][0.373] Inliner processing method read(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.373] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.SocketChannelImplInstrumentor.read(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.373] Inlining call to read(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.374] Inlining done
[jfr][TRACE][0.374] Inlining call to read(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.374] Inlining done
[jfr][TRACE][0.374] Inliner processing method read([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.374] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.SocketChannelImplInstrumentor.read([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.374] Inlining call to read([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.375] Inlining done
[jfr][TRACE][0.375] Inlining call to read([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.375] Inlining done
[jfr][TRACE][0.375] Inliner processing method write(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.375] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.SocketChannelImplInstrumentor.write(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.375] Inlining call to write(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.376] Inlining done
[jfr][TRACE][0.376] Inlining call to write(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.376] Inlining done
[jfr][TRACE][0.376] Inliner processing method write([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.376] MethodCallInliner:
targetMethod=oracle.jrockit.jfr.SocketChannelImplInstrumentor.write([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.376] Inlining call to write([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.376] Inlining done
[jfr][TRACE][0.376] Inlining call to write([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.377] Inlining done
[jfr][TRACE][0.378] Deleting write(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.378] Deleting write([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.378] Deleting read(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.378] Deleting read([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.390] Copying method: read(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.390] with mapper:
{oracle/jrockit/jfr/SocketChannelImplInstrumentor=sun/nio/ch/SocketChannelImpl}
[jfr][TRACE][0.396] Copying method: read([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.396] with mapper:
{oracle/jrockit/jfr/SocketChannelImplInstrumentor=sun/nio/ch/SocketChannelImpl}
[jfr][TRACE][0.403] Copying method: write(Ljava/nio/ByteBuffer;)I
[jfr][TRACE][0.403] with mapper:
{oracle/jrockit/jfr/SocketChannelImplInstrumentor=sun/nio/ch/SocketChannelImpl}
[jfr][TRACE][0.409] Copying method: write([Ljava/nio/ByteBuffer;II)J
[jfr][TRACE][0.409] with mapper:
{oracle/jrockit/jfr/SocketChannelImplInstrumentor=sun/nio/ch/SocketChannelImpl}
[jfr][TRACE][0.425] Shutdown hook registered
[jfr][TRACE][0.425] Java started. Took 334 ms
[jfr][TRACE][0.426] Starting up default recording
[jfr][DEBUG][0.426] Bufferthread started (0x228c)
[jfr][INFO ][1.935] Starting recording [id=0, name=HotSpot default, start=Wed
Feb 19 17:26:06 UTC 2020, maxAge=7200000ms]
[jfr][DEBUG][1.936] First recording starting...
[jfr][TRACE][1.936] Created chunk path
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_26_06_8823_0.jfr
[jfr][TRACE][1.941] Binding recording [id=0, name=HotSpot default, start=Wed
Feb 19 17:26:06 UTC 2020, maxAge=7200000ms]
[jfr][TRACE][1.941] Default recording started
[jfr][TRACE][1.941] Finished starting default recording
[jfr][TRACE][1.941] Jfr::initialize_stage_two completed
[jfr][TRACE][155.630] Filled global buffer. Total full global buffers 1
[jfr][DEBUG][155.640] Wrote 1 full global buffer(s) of 458576 B of data to
chunk.
[jfr][TRACE][287.089] Filled global buffer. Total full global buffers 1
[jfr][DEBUG][287.090] Wrote 1 full global buffer(s) of 459678 B of data to
chunk.
[jfr][TRACE][421.275] Filled global buffer. Total full global buffers 1
[jfr][DEBUG][421.275] Wrote 1 full global buffer(s) of 460393 B of data to
chunk.
[jfr][TRACE][575.360] Filled global buffer. Total full global buffers 1
[jfr][DEBUG][575.360] Wrote 1 full global buffer(s) of 460354 B of data to
chunk.
[jfr][TRACE][717.271] Filled global buffer. Total full global buffers 1
[jfr][DEBUG][717.573] Wrote 1 full global buffer(s) of 462769 B of data to
chunk.
[jfr][DEBUG][880.473] Shutdown hook: destroy java
[jfr][INFO ][880.473] Dump on exit: recording 0 'HotSpot default'
[jfr][INFO ][880.475] Starting recording [id=1, name=dump-on-exit-clone-of-0,
start=Wed Feb 19 17:26:06 UTC 2020]
[jfr][DEBUG][880.475] Recording starting, issuing buffer rotation...
[jfr][TRACE][880.484] Filled global buffer. Total full global buffers 1
[jfr][DEBUG][880.485] Wrote 1 full global buffer(s) of 458055 B of data to
chunk.
[jfr][TRACE][880.564] Created chunk path
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_40_45_8823_0.jfr
[jfr][WARN ][880.565] VMJFR.finishChunk ignored an exception
[jfr][WARN ][880.565] java.io.IOException:No such file or directory
[jfr][WARN ][880.565] java.io.UnixFileSystem.createFileExclusively(Native
Method)
[jfr][WARN ][880.566] java.io.File.createTempFile(File.java:2024)
[jfr][WARN ][880.566] java.io.File.createTempFile(File.java:2070)
[jfr][WARN ][880.566]
oracle.jrockit.jfr.RepositoryChunk$1.run(RepositoryChunk.java:76)
[jfr][WARN ][880.566]
oracle.jrockit.jfr.RepositoryChunk$1.run(RepositoryChunk.java:73)
[jfr][WARN ][880.566] java.security.AccessController.doPrivileged(Native
Method)
[jfr][WARN ][880.566]
java.security.AccessController.doPrivileged(AccessController.java:713)
[jfr][WARN ][880.566]
oracle.jrockit.jfr.RepositoryChunk.finish(RepositoryChunk.java:73)
[jfr][WARN ][880.566]
oracle.jrockit.jfr.RepositoryChunk.finish(RepositoryChunk.java:57)
[jfr][WARN ][880.566] oracle.jrockit.jfr.VMJFR.finishChunk(VMJFR.java:357)
[jfr][DEBUG][880.566] Release chunk
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_26_06_8823_0.jfr.part ref count
now 0
[jfr][DEBUG][880.568] Repository chunk
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_26_06_8823_0.jfr.part deleted.
[jfr][DEBUG][880.568] Repository chunk
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_26_06_8823_0.jfr deleted.
[jfr][DEBUG][880.570] Recording stopping, issuing buffer rotation...
[jfr][TRACE][880.589] Created chunk path
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_40_45_8823_1.jfr
[jfr][WARN ][880.590] VMJFR.finishChunk ignored an exception
[jfr][WARN ][880.590] java.io.IOException:No such file or directory
[jfr][WARN ][880.590] java.io.UnixFileSystem.createFileExclusively(Native
Method)
[jfr][WARN ][880.590] java.io.File.createTempFile(File.java:2024)
[jfr][WARN ][880.590] java.io.File.createTempFile(File.java:2070)
[jfr][WARN ][880.590]
oracle.jrockit.jfr.RepositoryChunk$1.run(RepositoryChunk.java:76)
[jfr][WARN ][880.590]
oracle.jrockit.jfr.RepositoryChunk$1.run(RepositoryChunk.java:73)
[jfr][WARN ][880.590] java.security.AccessController.doPrivileged(Native
Method)
[jfr][WARN ][880.590]
java.security.AccessController.doPrivileged(AccessController.java:713)
[jfr][WARN ][880.590]
oracle.jrockit.jfr.RepositoryChunk.finish(RepositoryChunk.java:73)
[jfr][WARN ][880.590]
oracle.jrockit.jfr.RepositoryChunk.finish(RepositoryChunk.java:57)
[jfr][WARN ][880.590] oracle.jrockit.jfr.VMJFR.finishChunk(VMJFR.java:357)
[jfr][DEBUG][880.590] Release chunk
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_40_45_8823_0.jfr.part ref count
now 0
[jfr][DEBUG][880.590] Repository chunk
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_40_45_8823_0.jfr.part deleted.
[jfr][DEBUG][880.590] Repository chunk
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_40_45_8823_0.jfr deleted.
[jfr][INFO ][880.593] Stopped recording [id=1, name=dump-on-exit-clone-of-0,
start=Wed Feb 19 17:26:06 UTC 2020, end=Wed Feb 19 17:40:45 UTC 2020]
[jfr][ERROR][880.593] Could not copy to
/tmp/hotspot-pid-8823-id-0-2020_02_19_17_40_45.jfr
[jfr][ERROR][880.593] java.io.FileNotFoundException:No chunks
[jfr][ERROR][880.593]
oracle.jrockit.jfr.ChunksChannel.<init>(ChunksChannel.java:26)
[jfr][ERROR][880.593] oracle.jrockit.jfr.Recording.copyTo(Recording.java:404)
[jfr][ERROR][880.593] oracle.jrockit.jfr.JFRImpl.dumpOnExit(JFRImpl.java:802)
[jfr][ERROR][880.593] oracle.jrockit.jfr.JFRImpl.dumpOnExit(JFRImpl.java:789)
[jfr][ERROR][880.593] oracle.jrockit.jfr.JFRImpl.destroy(JFRImpl.java:767)
[jfr][ERROR][880.593] oracle.jrockit.jfr.JFR$2.run(JFR.java:208)
[jfr][ERROR][880.593] java.lang.Thread.run(Thread.java:748)
[jfr][ERROR][880.593] Could not delete
/tmp/hotspot-pid-8823-id-0-2020_02_19_17_40_45.jfr
[jfr][ERROR][880.593] java.io.FileNotFoundException:No chunks
[jfr][ERROR][880.593]
oracle.jrockit.jfr.ChunksChannel.<init>(ChunksChannel.java:26)
[jfr][ERROR][880.593] oracle.jrockit.jfr.Recording.copyTo(Recording.java:404)
[jfr][ERROR][880.593] oracle.jrockit.jfr.JFRImpl.dumpOnExit(JFRImpl.java:802)
[jfr][ERROR][880.593] oracle.jrockit.jfr.JFRImpl.dumpOnExit(JFRImpl.java:789)
[jfr][ERROR][880.593] oracle.jrockit.jfr.JFRImpl.destroy(JFRImpl.java:767)
[jfr][ERROR][880.593] oracle.jrockit.jfr.JFR$2.run(JFR.java:208)
[jfr][ERROR][880.593] java.lang.Thread.run(Thread.java:748)
[jfr][WARN ][880.593] Could not dump recording 0 'HotSpot default' on exit
[jfr][WARN ][880.593] java.io.FileNotFoundException:No chunks
[jfr][WARN ][880.593]
oracle.jrockit.jfr.ChunksChannel.<init>(ChunksChannel.java:26)
[jfr][WARN ][880.593] oracle.jrockit.jfr.Recording.copyTo(Recording.java:404)
[jfr][WARN ][880.593] oracle.jrockit.jfr.JFRImpl.dumpOnExit(JFRImpl.java:802)
[jfr][WARN ][880.593] oracle.jrockit.jfr.JFRImpl.dumpOnExit(JFRImpl.java:789)
[jfr][WARN ][880.593] oracle.jrockit.jfr.JFRImpl.destroy(JFRImpl.java:767)
[jfr][WARN ][880.593] oracle.jrockit.jfr.JFR$2.run(JFR.java:208)
[jfr][WARN ][880.593] java.lang.Thread.run(Thread.java:748)
[jfr][DEBUG][880.593] Last recording stopping...
[jfr][WARN ][880.608] VMJFR.finishChunk ignored an exception
[jfr][WARN ][880.608] java.io.IOException:No such file or directory
[jfr][WARN ][880.608] java.io.UnixFileSystem.createFileExclusively(Native
Method)
[jfr][WARN ][880.608] java.io.File.createTempFile(File.java:2024)
[jfr][WARN ][880.608] java.io.File.createTempFile(File.java:2070)
[jfr][WARN ][880.608]
oracle.jrockit.jfr.RepositoryChunk$1.run(RepositoryChunk.java:76)
[jfr][WARN ][880.608]
oracle.jrockit.jfr.RepositoryChunk$1.run(RepositoryChunk.java:73)
[jfr][WARN ][880.608] java.security.AccessController.doPrivileged(Native
Method)
[jfr][WARN ][880.608]
java.security.AccessController.doPrivileged(AccessController.java:713)
[jfr][WARN ][880.608]
oracle.jrockit.jfr.RepositoryChunk.finish(RepositoryChunk.java:73)
[jfr][WARN ][880.608]
oracle.jrockit.jfr.RepositoryChunk.finish(RepositoryChunk.java:57)
[jfr][WARN ][880.608] oracle.jrockit.jfr.VMJFR.finishChunk(VMJFR.java:357)
[jfr][DEBUG][880.608] Release chunk
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_40_45_8823_1.jfr.part ref count
now 0
[jfr][DEBUG][880.608] Repository chunk
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_40_45_8823_1.jfr.part deleted.
[jfr][DEBUG][880.608] Repository chunk
/mnt/tmp/2020_02_19_17_26_04_8823/2020_02_19_17_40_45_8823_1.jfr deleted.
[jfr][INFO ][880.609] Stopped recording [id=0, name=HotSpot default, start=Wed
Feb 19 17:26:06 UTC 2020, end=Wed Feb 19 17:40:45 UTC 2020, maxAge=7200000ms]
[jfr][INFO ][880.609] Flight Recorder repository deleted
[jfr][DEBUG][880.609] Shutdown hook: shutdown native
[jfr][DEBUG][880.609] Bufferthread stopped.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
