[jira] [Commented] (FLINK-1650) Suppress Akka's Netty Shutdown Errors through the log config
[ https://issues.apache.org/jira/browse/FLINK-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386792#comment-14386792 ] Robert Metzger commented on FLINK-1650: --- I'm going to set the Akka version back to 2.3.7 in https://github.com/apache/flink/pull/542 because the error detection in Akka is much slower in 2.3.9. I've asked on the Akka mailing list for help. Suppress Akka's Netty Shutdown Errors through the log config Key: FLINK-1650 URL: https://issues.apache.org/jira/browse/FLINK-1650 Project: Flink Issue Type: Bug Components: other Affects Versions: 0.9 Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 0.9 I suggest to set the logging for `org.jboss.netty.channel.DefaultChannelPipeline` to error, in order to get rid of the misleading stack trace caused by an akka/netty hickup on shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1650) Suppress Akka's Netty Shutdown Errors through the log config
[ https://issues.apache.org/jira/browse/FLINK-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384008#comment-14384008 ] ASF GitHub Bot commented on FLINK-1650: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/518 Suppress Akka's Netty Shutdown Errors through the log config Key: FLINK-1650 URL: https://issues.apache.org/jira/browse/FLINK-1650 Project: Flink Issue Type: Bug Components: other Affects Versions: 0.9 Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 0.9 I suggest to set the logging for `org.jboss.netty.channel.DefaultChannelPipeline` to error, in order to get rid of the misleading stack trace caused by an akka/netty hickup on shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1650) Suppress Akka's Netty Shutdown Errors through the log config
[ https://issues.apache.org/jira/browse/FLINK-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377587#comment-14377587 ] ASF GitHub Bot commented on FLINK-1650: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/518#issuecomment-85426018 Good find, but just calling it in the beginning of every main() method seems hacky. I think the code should go to the AkkaUtils.createActorSystem() method. Then we also need it only once. Suppress Akka's Netty Shutdown Errors through the log config Key: FLINK-1650 URL: https://issues.apache.org/jira/browse/FLINK-1650 Project: Flink Issue Type: Bug Components: other Affects Versions: 0.9 Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 0.9 I suggest to set the logging for `org.jboss.netty.channel.DefaultChannelPipeline` to error, in order to get rid of the misleading stack trace caused by an akka/netty hickup on shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1650) Suppress Akka's Netty Shutdown Errors through the log config
[ https://issues.apache.org/jira/browse/FLINK-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378523#comment-14378523 ] ASF GitHub Bot commented on FLINK-1650: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/518#issuecomment-85678768 I moved the call to the Akka utils. Suppress Akka's Netty Shutdown Errors through the log config Key: FLINK-1650 URL: https://issues.apache.org/jira/browse/FLINK-1650 Project: Flink Issue Type: Bug Components: other Affects Versions: 0.9 Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 0.9 I suggest to set the logging for `org.jboss.netty.channel.DefaultChannelPipeline` to error, in order to get rid of the misleading stack trace caused by an akka/netty hickup on shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1650) Suppress Akka's Netty Shutdown Errors through the log config
[ https://issues.apache.org/jira/browse/FLINK-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375907#comment-14375907 ] ASF GitHub Bot commented on FLINK-1650: --- GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/518 [FLINK-1650] Configure Netty (akka) to use Slf4j. It seems that Netty 3.8.0 used by Akka was using `java.util.logging` for its internal logging, that's why this entry was without effect: https://github.com/apache/flink/blob/master/flink-dist/src/main/flink-bin/conf/log4j.properties#L29. The change sets the Netty logging factory to Slf4j. This means that Netty is now using Sfl4j. It'll also respect our logging settings (`org.jboss.netty.channel.DefaultChannelPipeline` to lvl ERROR). Lets see if the exception now disappears. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink flink1650 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/518.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #518 commit d4778d4d51cd20db2861b3632e64aeac1f5f6f7a Author: Robert Metzger rmetz...@apache.org Date: 2015-03-23T10:42:21Z [FLINK-1650] Set akka version to 2.3.9 commit 69b6a9b497955544e0e5c8707293c580e1ba3d5e Author: Robert Metzger rmetz...@apache.org Date: 2015-03-23T13:44:19Z [FLINK-1650] Let Netty(Akka) use Slf4j Suppress Akka's Netty Shutdown Errors through the log config Key: FLINK-1650 URL: https://issues.apache.org/jira/browse/FLINK-1650 Project: Flink Issue Type: Bug Components: other Affects Versions: 0.9 Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 0.9 I suggest to set the logging for `org.jboss.netty.channel.DefaultChannelPipeline` to error, in order to get rid of the misleading stack trace caused by an akka/netty hickup on shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1650) Suppress Akka's Netty Shutdown Errors through the log config
[ https://issues.apache.org/jira/browse/FLINK-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376463#comment-14376463 ] ASF GitHub Bot commented on FLINK-1650: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/518#issuecomment-85160071 The build passed in my travis: https://travis-ci.org/rmetzger/flink/builds/55487319 Suppress Akka's Netty Shutdown Errors through the log config Key: FLINK-1650 URL: https://issues.apache.org/jira/browse/FLINK-1650 Project: Flink Issue Type: Bug Components: other Affects Versions: 0.9 Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 0.9 I suggest to set the logging for `org.jboss.netty.channel.DefaultChannelPipeline` to error, in order to get rid of the misleading stack trace caused by an akka/netty hickup on shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1650) Suppress Akka's Netty Shutdown Errors through the log config
[
https://issues.apache.org/jira/browse/FLINK-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375668#comment-14375668
]
Robert Metzger commented on FLINK-1650:
---
I got a response from the Akka mailing list. They've asked me to open an issue
if we're still seeing the issue in {{2.3.9}}.
We are currently on Akka version {{2.3.7}}.
The changelogs:
http://akka.io/news/2014/12/17/akka-2.3.8-released.html
http://akka.io/news/2015/01/19/akka-2.3.9-released.html
Any objections against bumping our Akka dependency to 2.3.9 ?
Suppress Akka's Netty Shutdown Errors through the log config
Key: FLINK-1650
URL: https://issues.apache.org/jira/browse/FLINK-1650
Project: Flink
Issue Type: Bug
Components: other
Affects Versions: 0.9
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 0.9
I suggest to set the logging for
`org.jboss.netty.channel.DefaultChannelPipeline` to error, in order to get
rid of the misleading stack trace caused by an akka/netty hickup on shutdown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1650) Suppress Akka's Netty Shutdown Errors through the log config
[ https://issues.apache.org/jira/browse/FLINK-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376848#comment-14376848 ] ASF GitHub Bot commented on FLINK-1650: --- Github user tillrohrmann commented on the pull request: https://github.com/apache/flink/pull/518#issuecomment-85239319 Great to hear that you figured out why we could not suppress this error :-) On Mon, Mar 23, 2015 at 8:33 PM, Robert Metzger notificati...@github.com wrote: The build passed in my travis: https://travis-ci.org/rmetzger/flink/builds/55487319 — Reply to this email directly or view it on GitHub https://github.com/apache/flink/pull/518#issuecomment-85160071. Suppress Akka's Netty Shutdown Errors through the log config Key: FLINK-1650 URL: https://issues.apache.org/jira/browse/FLINK-1650 Project: Flink Issue Type: Bug Components: other Affects Versions: 0.9 Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 0.9 I suggest to set the logging for `org.jboss.netty.channel.DefaultChannelPipeline` to error, in order to get rid of the misleading stack trace caused by an akka/netty hickup on shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1650) Suppress Akka's Netty Shutdown Errors through the log config
[ https://issues.apache.org/jira/browse/FLINK-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365953#comment-14365953 ] Robert Metzger commented on FLINK-1650: --- I've asked on the Akka userlist regarding this issue. Suppress Akka's Netty Shutdown Errors through the log config Key: FLINK-1650 URL: https://issues.apache.org/jira/browse/FLINK-1650 Project: Flink Issue Type: Bug Components: other Affects Versions: 0.9 Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 0.9 I suggest to set the logging for `org.jboss.netty.channel.DefaultChannelPipeline` to error, in order to get rid of the misleading stack trace caused by an akka/netty hickup on shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1650) Suppress Akka's Netty Shutdown Errors through the log config
[
https://issues.apache.org/jira/browse/FLINK-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360287#comment-14360287
]
Robert Metzger commented on FLINK-1650:
---
Right now, we only shade away Hadoop's netty, so that we can use whatever netty
version we want (in particular because hadoop has a very old netty in hadoop
2.0.0-alpha).
Our users still see our netty dependency.
The log also shows that the exception hasn't been thrown from a relocated class.
{code}
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- Found a file with an exception. Printing contents:
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE: Mar 12, 2015 10:46:35 PM org.jboss.netty.channel.DefaultChannelPipeline
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE: WARNING: An exception was thrown by an exception handler.
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE: java.util.concurrent.RejectedExecutionException: Worker has already
been shutdown
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
org.jboss.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:54)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
org.jboss.netty.channel.Channels.disconnect(Channels.java:781)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
org.jboss.netty.channel.AbstractChannel.disconnect(AbstractChannel.java:211)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
akka.remote.transport.netty.NettyTransport$$anonfun$gracefulClose$1.apply(NettyTransport.scala:223)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
akka.remote.transport.netty.NettyTransport$$anonfun$gracefulClose$1.apply(NettyTransport.scala:222)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at scala.util.Success.foreach(Try.scala:205)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
scala.concurrent.Future$$anonfun$foreach$1.apply(Future.scala:204)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
scala.concurrent.Future$$anonfun$foreach$1.apply(Future.scala:204)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
22:46:49,202 WARN org.apache.flink.yarn.YarnTestBase
- LINE:at
[jira] [Commented] (FLINK-1650) Suppress Akka's Netty Shutdown Errors through the log config
[ https://issues.apache.org/jira/browse/FLINK-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360230#comment-14360230 ] Stephan Ewen commented on FLINK-1650: - I this YARN specific? Does the shading affect the netty classes and hence change the classnames and the logger config? Suppress Akka's Netty Shutdown Errors through the log config Key: FLINK-1650 URL: https://issues.apache.org/jira/browse/FLINK-1650 Project: Flink Issue Type: Bug Components: other Affects Versions: 0.9 Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 0.9 I suggest to set the logging for `org.jboss.netty.channel.DefaultChannelPipeline` to error, in order to get rid of the misleading stack trace caused by an akka/netty hickup on shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1650) Suppress Akka's Netty Shutdown Errors through the log config
[ https://issues.apache.org/jira/browse/FLINK-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348709#comment-14348709 ] ASF GitHub Bot commented on FLINK-1650: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/455 Suppress Akka's Netty Shutdown Errors through the log config Key: FLINK-1650 URL: https://issues.apache.org/jira/browse/FLINK-1650 Project: Flink Issue Type: Bug Components: other Affects Versions: 0.9 Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 0.9 I suggest to set the logging for `org.jboss.netty.channel.DefaultChannelPipeline` to error, in order to get rid of the misleading stack trace caused by an akka/netty hickup on shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1650) Suppress Akka's Netty Shutdown Errors through the log config
[
https://issues.apache.org/jira/browse/FLINK-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347450#comment-14347450
]
Stephan Ewen commented on FLINK-1650:
-
The logged error is below.
Setting the log level to ERROR should allow us to see critical messages and
suppress the confusing warnings that seem to be an akka/netty bug.
{code}
Feb 18, 2015 5:25:18 PM org.jboss.netty.channel.DefaultChannelPipeline
WARNING: An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been
shutdown
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
at
org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
at
org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at
org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
at
org.jboss.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
at
org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
at
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:54)
at org.jboss.netty.channel.Channels.disconnect(Channels.java:781)
at
org.jboss.netty.channel.AbstractChannel.disconnect(AbstractChannel.java:211)
at
akka.remote.transport.netty.NettyTransport$$anonfun$gracefulClose$1.apply(NettyTransport.scala:223)
at
akka.remote.transport.netty.NettyTransport$$anonfun$gracefulClose$1.apply(NettyTransport.scala:222)
at scala.util.Success.foreach(Try.scala:205)
at scala.concurrent.Future$$anonfun$foreach$1.apply(Future.scala:204)
at scala.concurrent.Future$$anonfun$foreach$1.apply(Future.scala:204)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
at
scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
{code}
Suppress Akka's Netty Shutdown Errors through the log config
Key: FLINK-1650
URL: https://issues.apache.org/jira/browse/FLINK-1650
Project: Flink
Issue Type: Bug
Components: other
Affects Versions: 0.9
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 0.9
I suggest to set the logging for
`org.jboss.netty.channel.DefaultChannelPipeline` to error, in order to get
rid of the misleading stack trace caused by an akka/netty hickup on shutdown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
