[jira] [Commented] (NIFI-7035) Don't spam logs with Apache Curator errors on Zookeeper Disconnect
[
https://issues.apache.org/jira/browse/NIFI-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17130751#comment-17130751
]
Makarov Vasiliy Nicolaevich commented on NIFI-7035:
---
[~markap14]
Good point, Mark, I have tried it out and it seems to fix the problem of
excessive logging. These exceptions are no longer spammed in the NiFi log.
I will create PR and attach steps to reproduce the issue and also logs
references.
> Don't spam logs with Apache Curator errors on Zookeeper Disconnect
> --
>
> Key: NIFI-7035
> URL: https://issues.apache.org/jira/browse/NIFI-7035
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Shawn Weeks
>Assignee: Makarov Vasiliy Nicolaevich
>Priority: Trivial
>
> Sometimes when you startup NiFi and not all of the Zookeeper nodes aren't up
> it fills the logs with Apache Curator errors like this one. We should be able
> to improve this a bit as logging the same error repeatedly isn't helpful.
> {code:java}
> 2020-01-16 08:58:28,767 ERROR [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up
> org.apache.curator.CuratorConnectionLossException: KeeperErrorCode =
> ConnectionLoss
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7035) Don't spam logs with Apache Curator errors on Zookeeper Disconnect
[
https://issues.apache.org/jira/browse/NIFI-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17130694#comment-17130694
]
Mark Payne commented on NIFI-7035:
--
I'd recommend trying that flag -
"-Dcurator-log-only-first-connection-issue-as-error-level=true" to see if it
resolves the issue first. That can be added to `bootstrap.conf` such as
{code:java}
java.arg.curator.supress.excessive.logs=-Dcurator-log-only-first-connection-issue-as-error-level=true{code}
>From the note that Jordan Zimmerman left it sounds like that would address the
>issue.
> Don't spam logs with Apache Curator errors on Zookeeper Disconnect
> --
>
> Key: NIFI-7035
> URL: https://issues.apache.org/jira/browse/NIFI-7035
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Shawn Weeks
>Assignee: Makarov Vasiliy Nicolaevich
>Priority: Trivial
>
> Sometimes when you startup NiFi and not all of the Zookeeper nodes aren't up
> it fills the logs with Apache Curator errors like this one. We should be able
> to improve this a bit as logging the same error repeatedly isn't helpful.
> {code:java}
> 2020-01-16 08:58:28,767 ERROR [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up
> org.apache.curator.CuratorConnectionLossException: KeeperErrorCode =
> ConnectionLoss
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7035) Don't spam logs with Apache Curator errors on Zookeeper Disconnect
[
https://issues.apache.org/jira/browse/NIFI-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17129295#comment-17129295
]
Makarov Vasiliy Nicolaevich commented on NIFI-7035:
---
[~Absolutesantaja] [~rabikumar.kc]
Ok, so I have reproduced the issue. It occurs when NiFi is in cluster mode but
has no access to one of the Zookeper nodes (embedded in my case).
I will try to limit it / filter excessive logs out it on Logbase config part
because as I see we don't explicitly log this error in NiFi code.
Also, this task is related it seems to:
[https://issues.apache.org/jira/browse/NIFI-3731
] [~randgalt] indicated there that Curator has a switch to log only the first
error as an ERROR level and others as DEBUG level. Don't know if it helps in
the situation with zookeeper only.
Any advice would be appreciated though.
> Don't spam logs with Apache Curator errors on Zookeeper Disconnect
> --
>
> Key: NIFI-7035
> URL: https://issues.apache.org/jira/browse/NIFI-7035
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Shawn Weeks
>Assignee: Makarov Vasiliy Nicolaevich
>Priority: Trivial
>
> Sometimes when you startup NiFi and not all of the Zookeeper nodes aren't up
> it fills the logs with Apache Curator errors like this one. We should be able
> to improve this a bit as logging the same error repeatedly isn't helpful.
> {code:java}
> 2020-01-16 08:58:28,767 ERROR [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up
> org.apache.curator.CuratorConnectionLossException: KeeperErrorCode =
> ConnectionLoss
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7035) Don't spam logs with Apache Curator errors on Zookeeper Disconnect
[
https://issues.apache.org/jira/browse/NIFI-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128375#comment-17128375
]
Makarov Vasiliy Nicolaevich commented on NIFI-7035:
---
I would like to participate in this issue too. Looking forward to being added
to the contributors.
> Don't spam logs with Apache Curator errors on Zookeeper Disconnect
> --
>
> Key: NIFI-7035
> URL: https://issues.apache.org/jira/browse/NIFI-7035
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Shawn Weeks
>Priority: Trivial
>
> Sometimes when you startup NiFi and not all of the Zookeeper nodes aren't up
> it fills the logs with Apache Curator errors like this one. We should be able
> to improve this a bit as logging the same error repeatedly isn't helpful.
> {code:java}
> 2020-01-16 08:58:28,767 ERROR [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up
> org.apache.curator.CuratorConnectionLossException: KeeperErrorCode =
> ConnectionLoss
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7035) Don't spam logs with Apache Curator errors on Zookeeper Disconnect
[
https://issues.apache.org/jira/browse/NIFI-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017057#comment-17017057
]
Shawn Weeks commented on NIFI-7035:
---
I'll try and track down the exact scenario that causes this but it's happen to
me several times and some other folks on Slack.
> Don't spam logs with Apache Curator errors on Zookeeper Disconnect
> --
>
> Key: NIFI-7035
> URL: https://issues.apache.org/jira/browse/NIFI-7035
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Shawn Weeks
>Priority: Trivial
>
> Sometimes when you startup NiFi and not all of the Zookeeper nodes aren't up
> it fills the logs with Apache Curator errors like this one. We should be able
> to improve this a bit as logging the same error repeatedly isn't helpful.
> {code:java}
> 2020-01-16 08:58:28,767 ERROR [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up
> org.apache.curator.CuratorConnectionLossException: KeeperErrorCode =
> ConnectionLoss
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
