[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361974#comment-17361974 ] ASF subversion and git services commented on NIFI-8625: --- Commit 3e774bc5beadd62a646794cc659db9902c5ab1fa in nifi's branch refs/heads/main from Matt Burgess [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=3e774bc ] NIFI-8625: Refactor scripted components to use ScriptRunner to fix concurrency issues (#5116) NIFI-8625: Refactor scripted components to use ScriptRunner to fix concurrency issues > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Assignee: Matt Burgess >Priority: Critical > Fix For: 1.14.0 > > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > Time Spent: 2h 20m > Remaining Estimate: 0h > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361975#comment-17361975 ] ASF subversion and git services commented on NIFI-8625: --- Commit 3e774bc5beadd62a646794cc659db9902c5ab1fa in nifi's branch refs/heads/main from Matt Burgess [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=3e774bc ] NIFI-8625: Refactor scripted components to use ScriptRunner to fix concurrency issues (#5116) NIFI-8625: Refactor scripted components to use ScriptRunner to fix concurrency issues > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Assignee: Matt Burgess >Priority: Critical > Fix For: 1.14.0 > > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > Time Spent: 2h 20m > Remaining Estimate: 0h > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355739#comment-17355739 ] Matt Burgess commented on NIFI-8625: The script configurators have become cumbersome, they were originally designed just to add whatever imports were needed and to find libraries/modules from the module paths such as JARs and Jython modules. The compilation stuff has made them unnecessarily complex and awkward, I think we should move only the relevant code into a ScriptRunner interface and impls and the scripted components just need to set up a runner (passing in the appropriate ScriptEngine) for each task, then set the bindings when evaluating the script. Will post a PR shortly for review > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Assignee: Matt Burgess >Priority: Critical > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354167#comment-17354167 ] KevinSky commented on NIFI-8625: no,8461 only describes part of the situation > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351886#comment-17351886 ] Otto Fowler commented on NIFI-8625: --- so this is a duplicate? > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351419#comment-17351419 ] KevinSky commented on NIFI-8625: hi [#Otto Fowler], Here is a better solution: nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/script/impl/JythonScriptEngineConfigurator.java !image-2021-05-26-09-13-01-784.png|width=965,height=298! !image-2021-05-26-09-13-25-543.png! nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/script/ScriptingComponentHelper.java !image-2021-05-26-09-13-55-074.png|width=814,height=104! > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351225#comment-17351225 ] Otto Fowler commented on NIFI-8625: --- I'm going to try to get some feedback or tips from [~mburgess149] > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351179#comment-17351179 ] Otto Fowler commented on NIFI-8625: --- also seeing : e -> {PyBaseExceptionDerived@19457} "AttributeError("'NoneType' object has no attribute 'getAttribute'",) > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351176#comment-17351176 ] Otto Fowler commented on NIFI-8625: --- It looks like once I get the 'not known to session' exception, every new file fails with that and the queue is not being processed > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351171#comment-17351171 ] Otto Fowler commented on NIFI-8625: --- What I'm doing: I have 4 files in the input directory after processing the first four, I do a `cp` like from 1.txt to 5.txt etc to add a file at a time Sometimes it works, sometimes I get the exception above > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351169#comment-17351169 ] Otto Fowler commented on NIFI-8625: --- also seeing this at times {noformat} xecuteScript[id=bb911eff-0bbc-346e-6a34-3875ae197ff0] org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=4cc02d23-3763-4b09-99e2-725d67924ba4,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1621958091080-1, container=default, section=1], offset=43, length=4],offset=0,name=5.txt,size=4] transfer relationship not specified at org.apache.nifi.controller.repository.StandardProcessSession.validateCommitState(StandardProcessSession.java:245) at org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:259) at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:406) at org.apache.nifi.controller.repository.StandardProcessSession.commitAsync(StandardProcessSession.java:362) at org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:273) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1180) at org.apache.nifi.controller.tasks.Con {noformat} > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351162#comment-17351162 ] Otto Fowler commented on NIFI-8625: --- ok, i'm seeing mixed results: newest error detail is {noformat} org.apache.nifi.processor.exception.FlowFileHandlingException: org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=29345e45-3fc6-4859-9449-0ce12f640e6f,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1621958091080-1, container=default, section=1], offset=9, length=4],offset=0,name=2.txt,size=4] is already marked for transfer {noformat} > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351132#comment-17351132 ] Otto Fowler commented on NIFI-8625: --- Strangely enough, I try to reproduce in the debugger, and it works: {noformat} 2021-05-25 11:24:14,368 INFO [Timer-Driven Process Thread-10] o.a.n.processors.standard.LogAttribute LogAttribute[id=a402624a-0179-1000-3c1d-e423f684707b] logging for flow file StandardFlowFileRecord[uuid=5a7bc46c-5a63-4a32-9fac-35913fa0c087,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1621954150437-1, container=default, section=1], offset=38, length=4],offset=0,name=1.txt,size=4] -- Standard FlowFile Attributes Key: 'entryDate' Value: 'Tue May 25 11:24:00 EDT 2021' Key: 'lineageStartDate' Value: 'Tue May 25 11:24:00 EDT 2021' Key: 'fileSize' Value: '4' FlowFile Attribute Map Content Key: 'absolute.path' Value: '/Users/ottofowler/tmp/nifi-input/' Key: 'file.creationTime' Value: '2021-05-25T10:45:26-0400' Key: 'file.group' Value: 'staff' Key: 'file.lastAccessTime' Value: '2021-05-25T10:49:10-0400' Key: 'file.lastModifiedTime' Value: '2021-05-25T10:45:34-0400' Key: 'file.owner' Value: 'ottofowler' Key: 'file.permissions' Value: 'rw-r--r--' Key: 'file.size' Value: '4' Key: 'filename' Value: '1.txt' Key: 'path' Value: './' Key: 'userCountName' Value: 'D:\99_TestCase\4_MultiFile_Stuck\B5_ALL_CELL_.txt' Key: 'uuid' Value: '5a7bc46c-5a63-4a32-9fac-35913fa0c087' -- one {noformat} > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351112#comment-17351112 ] Otto Fowler commented on NIFI-8625: --- Thanks! I just ran the template against main: * change input directory * remove filter * input has 4 text files * change log message ( success ) to logattributes I am seeing exceptions in the nifi-app.log that are rolling back the session. Can you see if you see those as well? {noformat} 2021-05-25 10:53:01,402 ERROR [Timer-Driven Process Thread-9] o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=bb911eff-0bbc-346e-6a34-3875ae197ff0] ExecuteScript[id=bb911eff-0bbc-346e-6a34-3875ae197ff0] failed to process due to org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: java.lang.NullPointerException: java.lang.NullPointerException in
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350710#comment-17350710 ] KevinSky commented on NIFI-8625: [^executeScript_nifi_8625.xml] ^Here is my template, you might modify some configurations in this,like filename.^ ^It's easy to reproduce,just write any python code in executeScript,and execute like this:^ {code:java} In single thread(concurrent task=1), executeScript just stop and start, flow file always stuck in connections. In multi thread(concurrent task=10).just execute it,executeScript always throw exception like is already marked for transfer in or is not known in this session. {code} > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > Attachments: executeScript_nifi_8625.xml, > image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350560#comment-17350560 ] Otto Fowler commented on NIFI-8625: --- Can you provide a template or steps to reproduce? > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > Attachments: image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw exception like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17349074#comment-17349074 ] KevinSky commented on NIFI-8625: remove compiledScriptRef will works fine. but I'm not sure it can resulting in up to 10x performance improvements by NIFI-8080 !image-2021-05-21-16-22-34-775.png! > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > Attachments: image-2021-05-21-16-22-34-775.png > > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw excetion like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8625) ExecuteScript processor always stuck after restart or multi thread
[ https://issues.apache.org/jira/browse/NIFI-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17349072#comment-17349072 ] KevinSky commented on NIFI-8625: NIFI-8080caused this, in sigle thread,if it is not first in,compiledScriptRef save the first time compiled value,and when executeScript restart, it only clear ScriptingComponentHelper.engineQ,but scriptEngineConfiguratorMap also has the reference of compiledScriptRef. so that is why we see stuck in connections. in multi thread, compiledScriptRef only save first engine compiled CompiledScript. eval() method don't use the true engine to eval,so it always throw exception NIFI-8461 reveals this problem, but the scene description is incomplete. > ExecuteScript processor always stuck after restart or multi thread > -- > > Key: NIFI-8625 > URL: https://issues.apache.org/jira/browse/NIFI-8625 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.2 >Reporter: KevinSky >Priority: Critical > Labels: ExecuteScript,stcuk > > In single thread, executeScript just stop and start, flow file always stuck > in connections. > In multi thread.executeScript always throw excetion like is already marked > for transfer in or is not known in this session. > -- This message was sent by Atlassian Jira (v8.3.4#803005)