Maybe something interesting nifi-bootstrap.log when this happens? On Wed, Sep 25, 2019 at 11:51 AM Jean-Sebastien Vachon < [email protected]> wrote:
> I could if the Nifi was up and running but when I loose the node, the Nifi > process itself dies so I can hardly take a heap dump of something that is > not running 😉 > > It happened twice before I sent my first email...nothing since then. > ------------------------------ > *From:* DEHAY Aurelien <[email protected]> > *Sent:* Wednesday, September 25, 2019 10:29 AM > *To:* [email protected] <[email protected]> > *Subject:* RE: node died unexpectedly > > > Classification\_- *INTERNAL & PARTNERS* > > > > Hello. > > > > By any chance, did you take thread dumps? > > > > If it happens again, take thread dumps of the java nifi process (with > jstack: jstack -l <pid> >> threaddumps.log). Take 3 or 4 every 10/15 > seconds . > > > > Regards. > > > > *From:* Jean-Sebastien Vachon <[email protected]> > *Sent:* lundi 23 septembre 2019 15:08 > *To:* [email protected] > *Subject:* node died unexpectedly > > > > Hi, > > > > I am running a cluster of five nodes and one of them just quit for no > apparent reason... at least I could not find anything wrong on the system. > > Here is a section of the logs. You can clearly see that there is a 8 > minutes gap before I manually had to restart Nifi. I've looked in > /var/log/messages and Nifi's log files but could > > not find any reason for this to happen... > > > > any thoughts/ideas? > > > > ----------- > > > > 2019-09-23 12:51:54,567 ERROR [Timer-Driven Process Thread-40] > o.a.n.p.standard.ExecuteStreamCommand > ExecuteStreamCommand[id=8b748c9a-7d7d-30cd-86a3-4ea3c4ccd3b4] Transferring > flow file > StandardFlowFileRecord[uuid=60d29ce2-088a-4829-a0ae-17672f1d86e6,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id > > =1569243036019-4497, container=default, section=401], offset=591157, > length=-1],offset=0,name=186373,size=0] to nonzero status. Executable > command /usr/bin/python3 ended in an error: > > 2019-09-23 12:51:54,570 ERROR [Timer-Driven Process Thread-4] > o.a.n.p.standard.ExecuteStreamCommand > ExecuteStreamCommand[id=272623bc-a371-3ff9-adaf-d8363b91a5ed] Transferring > flow file > StandardFlowFileRecord[uuid=a18485c8-9b03-43a8-8c13-e76803df9290,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1569242553862-4458, > container=default, section=362], offset=780733, > length=21229],offset=0,name=33287,size=21229] to nonzero status. Executable > command /usr/bin/python3 ended in an error: > /usr/local/lib/python3.7/site-packages/requests/__init__.py:80: > RequestsDependencyWarning: urllib3 (1.23) or chardet (3.0.4) doesn't match > a supported version! > > RequestsDependencyWarning) > > > > 2019-09-23 12:51:54,570 ERROR [Thread-130947] > o.a.n.p.standard.ExecuteStreamCommand > ExecuteStreamCommand[id=a7cb604a-465d-3440-beab-b9554e1ba4bb] Failed to > write flow file to stdin due to java.io.IOException: Broken pipe: > java.io.IOException: Broken pipe > > java.io.IOException: Broken pipe > > at java.io.FileOutputStream.writeBytes(Native Method) > > at java.io.FileOutputStream.write(FileOutputStream.java:326) > > at > java.io.BufferedOutputStream.write(BufferedOutputStream.java:122) > > at > java.io.BufferedOutputStream.write(BufferedOutputStream.java:122) > > at org.apache.nifi.stream.io.StreamUtils.copy(StreamUtils.java:36) > > at > org.apache.nifi.processors.standard.ExecuteStreamCommand$2.run(ExecuteStreamCommand.java:519) > > at java.lang.Thread.run(Thread.java:748) > > 2019-09-23 12:51:54,570 ERROR [Timer-Driven Process Thread-27] > o.a.n.p.standard.ExecuteStreamCommand > ExecuteStreamCommand[id=a7cb604a-465d-3440-beab-b9554e1ba4bb] Transferring > flow file > StandardFlowFileRecord[uuid=e5a82bb6-23a9-4a16-825a-1d976cd114cc,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1569242062433-4406, > container=default, section=310], offset=680454, > length=-1],offset=0,name=164088,size=0] to nonzero status. Executable > command /usr/bin/python3 ended in an error: > > > > 2019-09-23 12:59:56,012 INFO [main] org.apache.nifi.NiFi Launching NiFi... > > 2019-09-23 12:59:56,179 INFO [main] > o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties > path to be '/opt/nifi/./conf/nifi.properties' > > 2019-09-23 12:59:56,181 INFO [main] > o.a.nifi.properties.NiFiPropertiesLoader Loaded 149 properties from > /opt/nifi/./conf/nifi.properties > > 2019-09-23 12:59:56,186 INFO [main] org.apache.nifi.NiFi Loaded 149 > properties > > This electronic transmission (and any attachments thereto) is intended > solely for the use of the addressee(s). It may contain confidential or > legally privileged information. If you are not the intended recipient of > this message, you must delete it immediately and notify the sender. Any > unauthorized use or disclosure of this message is strictly prohibited. > Faurecia does not guarantee the integrity of this transmission and shall > therefore never be liable if the message is altered or falsified nor for > any virus, interception or damage to your system. >
