[
https://issues.apache.org/jira/browse/BEAM-9399?focusedWorklogId=500822&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500822
]
ASF GitHub Bot logged work on BEAM-9399:
----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Oct/20 19:41
Start Date: 14/Oct/20 19:41
Worklog Time Spent: 10m
Work Description: scwhittle commented on pull request #11351:
URL: https://github.com/apache/beam/pull/11351#issuecomment-708619117
On Wed, Oct 14, 2020, 7:06 PM Lukasz Cwik <[email protected]> wrote:
> That would work.
>
> Hi @lukecwik <https://github.com/lukecwik> - Interestingly enough, I'm
> having it failing on the 2.23.0 version ?
>
> Code : catch (Exception e) {
> System.out.println("Failed to filter user data " + c.element());
> e.printStackTrace();
>
> I assume to work around would be to change it from send it to StackTrace
> to another print out ?
>
> Errror :
> java.lang.IllegalStateException: BEAM-9399
> <https://issues.apache.org/jira/browse/BEAM-9399>: publish should not be
> called with the lock as it may cause deadlock
> at
>
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState(Preconditions.java:507)
> at
>
org.apache.beam.runners.dataflow.worker.logging.JulHandlerPrintStreamAdapterFactory$JulHandlerPrintStream.publishIfNonEmpty(JulHandlerPrintStreamAdapterFactory.java:380)
> at
>
org.apache.beam.runners.dataflow.worker.logging.JulHandlerPrintStreamAdapterFactory$JulHandlerPrintStream.println(JulHandlerPrintStreamAdapterFactory.java:332)
> at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)
> at java.lang.Throwable.printStackTrace(Throwable.java:655)
> at java.lang.Throwable.printStackTrace(Throwable.java:643)
> at java.lang.Throwable.printStackTrace(Throwable.java:634)
> at
>
com.metix.historic.transforms.FilterAndMapPatientsByAge.processElement(FilterAndMapPatientsByAge.java:25)
>
> Yes, using another printstream would work around the issue.
>
Could also avoid e.printStackTrace() and instead print the stack trace to a
string and then print, for example using
https://guava.dev/releases/23.0/api/docs/com/google/common/base/Throwables.html#getStackTraceAsString-java.lang.Throwable-
System.out.println("Failed to filter user data " + c.element() + " " +
Throwables.getStackTraceAsString(e));
> @scwhittle <https://github.com/scwhittle> Any ideas as to why it would be
> broken in 2.23?
>
It is broken because https://github.com/apache/beam/pull/11096 changed a
potential deadlock into consistent IllegalStateException. This was
unintended, I didn't consider that other things might synchronize on the
overridden System.err PrintStream. Throwable.printStackTrace does this.
This was fixed by https://github.com/apache/beam/pull/12825 which was
cherrypicked to the 2.25 release. The commit comment there has more details.
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/11351#issuecomment-708536481>, or
> unsubscribe
>
<https://github.com/notifications/unsubscribe-auth/ABBZZTC4FDINTANLXPZTNUDSKXLCLANCNFSM4MEFYJFQ>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 500822)
Time Spent: 12h (was: 11h 50m)
> Possible deadlock between DataflowWorkerLoggingHandler and overridden
> System.err PrintStream
> --------------------------------------------------------------------------------------------
>
> Key: BEAM-9399
> URL: https://issues.apache.org/jira/browse/BEAM-9399
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
> Reporter: Sam Whittle
> Assignee: Sam Whittle
> Priority: P0
> Fix For: 2.25.0
>
> Time Spent: 12h
> Remaining Estimate: 0h
>
> When an exception is encountered in DataflowWorkerLoggingHandler the
> ErrorManager is used to log the exception. ErrorManager uses System.err
> which is overridden to be a PrintStream that writes back into
> DataflowWorkerLoggingHandler.
> This has the lock ordering DataflowWorkerLoggingHandler -> PrintStream.
> Other logging of System.err has the inverse lock ordering
> PrintStream->DataflowWorkerLoggingHandler so there is potential for deadlock.
> This is one known cause of the inversion, but any other System.err logs from
> inside DataflowWorkerLoggingHandler could cause the same issue.
> Proposed fix is to address low-hanging fruit of having ErrorManager output to
> the original System.err. A full fix would be to improve our override of
> System.err to a PrintStream that can detect the locking inversion or possibly
> we could use the PrintStream mutex in both cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
