[
https://issues.apache.org/jira/browse/FLINK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14703425#comment-14703425
]
Min Jiang commented on FLINK-2089:
----------------------------------
I re-ran today and directed to a result log file. On the screen I got the
message stack as below
org.apache.flink.client.program.ProgramInvocationException: The program
execution failed: Job execution failed.
at org.apache.flink.client.program.Client.run(Client.java:413)
at org.apache.flink.client.program.Client.run(Client.java:356)
at org.apache.flink.client.program.Client.run(Client.java:349)
at
org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:63)
at min.play.RWA.main(RWA.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
at org.apache.flink.client.program.Client.run(Client.java:315)
at
org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:584)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:290)
at
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:880)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:922)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution
failed.
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$receiveWithLogMessages$1.applyOrElse(JobManager.scala:314)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at
org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:36)
at
org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:29)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
at
org.apache.flink.runtime.ActorLogMessages$$anon$1.applyOrElse(ActorLogMessages.scala:29)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at
org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:92)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.IllegalStateException: Buffer has already been recycled.
at
org.apache.flink.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:173)
at
org.apache.flink.runtime.io.network.buffer.Buffer.ensureNotRecycled(Buffer.java:142)
at
org.apache.flink.runtime.io.network.buffer.Buffer.setSize(Buffer.java:105)
at
org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.getCurrentBuffer(SpanningRecordSerializer.java:151)
at
org.apache.flink.runtime.io.network.api.writer.RecordWriter.clearBuffers(RecordWriter.java:166)
at
org.apache.flink.runtime.operators.RegularPactTask.clearWriters(RegularPactTask.java:1532)
at
org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:220)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
at java.lang.Thread.run(Thread.java:745)
Below is the log message from result log:
08/19/2015 13:06:58 Job execution switched to status RUNNING.
08/19/2015 13:06:58 CHAIN DataSource (at main(RWA.java:40)
(org.apache.flink.api.java.io.TextInputFormat)) -> Filter (Filter at
main(RWA.java:40))(1/1) switched to SCHEDULED
08/19/2015 13:06:58 CHAIN DataSource (at main(RWA.java:40)
(org.apache.flink.api.java.io.TextInputFormat)) -> Filter (Filter at
main(RWA.java:40))(1/1) switched to DEPLOYING
08/19/2015 13:06:58 CHAIN DataSource (at main(RWA.java:43)
(org.apache.flink.api.java.io.TextInputFormat)) -> Filter (Filter at
main(RWA.java:43))(1/1) switched to SCHEDULED
08/19/2015 13:06:58 CHAIN DataSource (at main(RWA.java:43)
(org.apache.flink.api.java.io.TextInputFormat)) -> Filter (Filter at
main(RWA.java:43))(1/1) switched to DEPLOYING
08/19/2015 13:06:58 CHAIN DataSource (at main(RWA.java:43)
(org.apache.flink.api.java.io.TextInputFormat)) -> Filter (Filter at
main(RWA.java:43))(1/1) switched to RUNNING
08/19/2015 13:06:58 CHAIN DataSource (at main(RWA.java:40)
(org.apache.flink.api.java.io.TextInputFormat)) -> Filter (Filter at
main(RWA.java:40))(1/1) switched to RUNNING
08/19/2015 13:06:59 DataSink (TextOutputFormat
(/home/min/flink_data/bad_data.csv) - UTF-8)(1/1) switched to SCHEDULED
08/19/2015 13:06:59 DataSink (TextOutputFormat
(/home/min/flink_data/bad_data.csv) - UTF-8)(1/1) switched to DEPLOYING
08/19/2015 13:06:59 DataSink (TextOutputFormat
(/home/min/flink_data/bad_data.csv) - UTF-8)(1/1) switched to RUNNING
08/19/2015 13:08:08 CHAIN DataSource (at main(RWA.java:43)
(org.apache.flink.api.java.io.TextInputFormat)) -> Filter (Filter at
main(RWA.java:43))(1/1) switched to FINISHED
08/19/2015 13:08:08 DataSink (TextOutputFormat
(/home/min/flink_data/bad_data.csv) - UTF-8)(1/1) switched to FINISHED
08/19/2015 13:08:10 CHAIN DataSource (at main(RWA.java:40)
(org.apache.flink.api.java.io.TextInputFormat)) -> Filter (Filter at
main(RWA.java:40))(1/1) switched to FAILED
java.lang.IllegalStateException: Buffer has already been recycled.
at
org.apache.flink.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:173)
at
org.apache.flink.runtime.io.network.buffer.Buffer.ensureNotRecycled(Buffer.java:142)
at
org.apache.flink.runtime.io.network.buffer.Buffer.setSize(Buffer.java:105)
at
org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.getCurrentBuffer(SpanningRecordSerializer.java:151)
at
org.apache.flink.runtime.io.network.api.writer.RecordWriter.clearBuffers(RecordWriter.java:166)
at
org.apache.flink.runtime.operators.RegularPactTask.clearWriters(RegularPactTask.java:1532)
at
org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:220)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
at java.lang.Thread.run(Thread.java:745)
08/19/2015 13:08:10 Job execution switched to status FAILING.
08/19/2015 13:08:10 FlatMap (FlatMap at main(RWA.java:62))(1/1) switched to
CANCELED
08/19/2015 13:08:10 CHAIN FlatMap (FlatMap at main(RWA.java:46)) -> FlatMap
(FlatMap at main(RWA.java:59))(1/1) switched to CANCELED
08/19/2015 13:08:10 Join(Join at
projectTupleX(JoinOperator.java:1341))(1/1) switched to CANCELED
08/19/2015 13:08:10 DataSink (CsvOutputFormat (path:
/home/min/flink_data/result.csv, delimiter: ¶))(1/1) switched to CANCELED
08/19/2015 13:08:10 Job execution switched to status FAILED.
> "Buffer recycled" IllegalStateException during cancelling
> ---------------------------------------------------------
>
> Key: FLINK-2089
> URL: https://issues.apache.org/jira/browse/FLINK-2089
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
> Affects Versions: master
> Reporter: Ufuk Celebi
> Assignee: Ufuk Celebi
> Fix For: 0.9
>
>
> [~rmetzger] reported the following stack trace during cancelling of high
> parallelism jobs:
> {code}
> Error: java.lang.IllegalStateException: Buffer has already been recycled.
> at
> org.apache.flink.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> at
> org.apache.flink.runtime.io.network.buffer.Buffer.ensureNotRecycled(Buffer.java:142)
> at
> org.apache.flink.runtime.io.network.buffer.Buffer.getMemorySegment(Buffer.java:78)
> at
> org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.setNextBuffer(SpillingAdaptiveSpanningRecordDeserializer.java:72)
> at
> org.apache.flink.runtime.io.network.api.reader.AbstractRecordReader.getNextRecord(AbstractRecordReader.java:80)
> at
> org.apache.flink.runtime.io.network.api.reader.MutableRecordReader.next(MutableRecordReader.java:34)
> at
> org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:73)
> at org.apache.flink.runtime.operators.MapDriver.run(MapDriver.java:96)
> at
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:496)
> at
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:362)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> This looks like a concurrent buffer pool release/buffer usage error. I'm
> investing this today.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
