date:20170803

[jira] [Commented] (NIFI-3329) MergeContent throws FlowFileHandlingException: not the most recent version of this FlowFile within this session

2017-08-03 Thread Joseph Witt (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113823#comment-16113823
 ] 

Joseph Witt commented on NIFI-3329:
---

after that exception provenance summary page comes back but viewing lineage is 
not available.

> MergeContent throws FlowFileHandlingException: not the most recent version of 
> this FlowFile within this session
> ---
>
> Key: NIFI-3329
> URL: https://issues.apache.org/jira/browse/NIFI-3329
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.1.1
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
> Attachments: simple_flow.xml
>
>
> The following error was reported on the dev mailing list:
> {code}
> This error occurred in NiFi 1.1.1 (single node) right after an upgrade from
> 1.1.0, so I thought it was due to the upgrade, but now I see that it happens
> if I restart as well and there is data in the flow. 
> This MergeContent processor is sorting incoming FlowFiles into bins based on
> a correlation attribute and dumping out the accumulated uber-FlowFile after
> one hour or 500K messages are accumulated.
> ...
> 2017-01-11 10:53:39,561 WARN [Timer-Driven Process Thread-9]
> o.a.n.processors.standard.MergeContent
> MergeContent[id=b9898788-1c11-45d9-8646-44b1dceb92d5] Processor
> Administratively Yielded for 1 sec due to processing failure
> 2017-01-11 10:53:39,561 WARN [Timer-Driven Process Thread-9]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> MergeContent[id=b9898788-1c11-45d9-8646-44b1dceb92d5] due to uncaught
> Exception: org.apache.nifi.processor.exception.FlowFileHandlingException:
> StandardFlowFileRecord[uuid=07f3d75a-3347-45da-8dd4-fecf775d1dcf,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1484151577298-1971,
> container=default, section=947], offset=566526,
> length=64],offset=0,name=8551433629757608.tsv,size=64] is not the most
> recent version of this FlowFile within this session
> (StandardProcessSession[id=133767])
> 2017-01-11 10:53:39,563 WARN [Timer-Driven Process Thread-9]
> o.a.n.c.t.ContinuallyRunProcessorTask
> org.apache.nifi.processor.exception.FlowFileHandlingException:
> StandardFlowFileRecord[uuid=07f3d75a-3347-45da-8dd4-fecf775d1dcf,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1484151577298-1971,
> container=default, section=947], offset=566526,
> length=64],offset=0,name=8551433629757608.tsv,size=64] is not the most
> recent version of this FlowFile within this session
> (StandardProcessSession[id=133767])
>at
> org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1121)
> ~[nifi-framework-core-1.1.0.jar:1.1.0]
>at
> org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1102)
> ~[nifi-framework-core-1.1.0.jar:1.1.0]
>at org.apache.nifi.processor.util.bin.Bin.offer(Bin.java:142)
> ~[na:na]
>at
> org.apache.nifi.processor.util.bin.BinManager.offer(BinManager.java:194)
> ~[na:na]
>at
> org.apache.nifi.processor.util.bin.BinFiles.binFlowFiles(BinFiles.java:279)
> ~[na:na]
>at
> org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:178)
> ~[na:na]
>at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
> ~[nifi-framework-core-1.1.0.jar:1.1.0]
>at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.1.0.jar:1.1.0]
>at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.1.0.jar:1.1.0]
>at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> [nifi-framework-core-1.1.0.jar:1.1.0]
>at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_65]
>at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_65]
>at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_65]
>at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_65]
>at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_65]
>at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_65]
>at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3329) MergeContent throws FlowFileHandlingException: not the most recent version of this FlowFile within this session

2017-08-03 Thread Joseph Witt (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113822#comment-16113822
 ] 

Joseph Witt commented on NIFI-3329:
---

without any swapping possible i was able to get this with just stable operation

2017-08-03 19:24:49,991 ERROR [Index Provenance Events-1] 
o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
NativeFSLock@/Users/jwitt/Development/apachenifi/nifi.git/nifi-assembly/target/nifi-1.4.0-SNAPSHOT-bin/nifi-1.4.0-SNAPSHOT/provenance_repository/index-1501813052257/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:89)
at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
at 
org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
at 
org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
at 
org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
at 
org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

> MergeContent throws FlowFileHandlingException: not the most recent version of 
> this FlowFile within this session
> ---
>
> Key: NIFI-3329
> URL: https://issues.apache.org/jira/browse/NIFI-3329
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.1.1
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
> Attachments: simple_flow.xml
>
>
> The following error was reported on the dev mailing list:
> {code}
> This error occurred in NiFi 1.1.1 (single node) right after an upgrade from
> 1.1.0, so I thought it was due to the upgrade, but now I see that it happens
> if I restart as well and there is data in the flow. 
> This MergeContent processor is sorting incoming FlowFiles into bins based on
> a correlation attribute and dumping out the accumulated uber-FlowFile after
> one hour or 500K messages are accumulated.
> ...
> 2017-01-11 10:53:39,561 WARN [Timer-Driven Process Thread-9]
> o.a.n.processors.standard.MergeContent
> MergeContent[id=b9898788-1c11-45d9-8646-44b1dceb92d5] Processor
> Administratively Yielded for 1 sec due to processing failure
> 2017-01-11 10:53:39,561 WARN [Timer-Driven Process Thread-9]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> MergeContent[id=b9898788-1c11-45d9-8646-44b1dceb92d5] due to uncaught
> Exception: org.apache.nifi.processor.exception.FlowFileHandlingException:
> StandardFlowFileRecord[uuid=07f3d75a-3347-45da-8dd4-fecf775d1dcf,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1484151577298-1971,
> container=default, section=947], offset=566526,
> length=64],offset=0,name=8551433629757608.tsv,size=64] is not the most
> recent version of this FlowFile within this session
> (StandardProcessSession[id=133767])
> 2017-01-11 10:53:39,563 WARN [Timer-Driven Process Thread-9]
> o.a.n.c.t.ContinuallyRunProcessorTask
> org.apache.nifi.processor.exception.FlowFileHandlingException:
> StandardFlowFileRecord[uuid=07f3d75a-3347-45da-8dd4-fecf775d1dcf,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1484151577298-1971,
> container=default, section=947], offset=566526,
> length=64],offset=0,name=8551433629757608.tsv,size=64] is not the most
> recent version of this FlowFile within this session
> (StandardProcessSession[id=133767])
>at
> org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1121)
> ~[nifi-framework-core-1.1.0.jar:1.1.0]
>at
> org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1102)
> ~[nifi-framework-core-1.1.0.jar:1.1.0]
>at org.apache.nifi.processor.util.bin.Bin.offer(Bin.java:142)
> ~[na:na]
>at
> org.apache.nifi.processor.util.bin.BinManager.offer(BinManager.java:194)
> ~[na:na]
>at
> org.apache.nifi.processor.util.bin.BinFiles.binFlowFiles(BinFiles.java:279)
> ~[na:na]
>at
> org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:178)
> ~[na:na]
>at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
> ~[nifi-framework-core-1.1.0.jar:1.1.0]
>at
>

[jira] [Updated] (NIFI-3329) MergeContent throws FlowFileHandlingException: not the most recent version of this FlowFile within this session

2017-08-03 Thread Joseph Witt (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-3329:
--
Attachment: simple_flow.xml

> MergeContent throws FlowFileHandlingException: not the most recent version of 
> this FlowFile within this session
> ---
>
> Key: NIFI-3329
> URL: https://issues.apache.org/jira/browse/NIFI-3329
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.1.1
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
> Attachments: simple_flow.xml
>
>
> The following error was reported on the dev mailing list:
> {code}
> This error occurred in NiFi 1.1.1 (single node) right after an upgrade from
> 1.1.0, so I thought it was due to the upgrade, but now I see that it happens
> if I restart as well and there is data in the flow. 
> This MergeContent processor is sorting incoming FlowFiles into bins based on
> a correlation attribute and dumping out the accumulated uber-FlowFile after
> one hour or 500K messages are accumulated.
> ...
> 2017-01-11 10:53:39,561 WARN [Timer-Driven Process Thread-9]
> o.a.n.processors.standard.MergeContent
> MergeContent[id=b9898788-1c11-45d9-8646-44b1dceb92d5] Processor
> Administratively Yielded for 1 sec due to processing failure
> 2017-01-11 10:53:39,561 WARN [Timer-Driven Process Thread-9]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> MergeContent[id=b9898788-1c11-45d9-8646-44b1dceb92d5] due to uncaught
> Exception: org.apache.nifi.processor.exception.FlowFileHandlingException:
> StandardFlowFileRecord[uuid=07f3d75a-3347-45da-8dd4-fecf775d1dcf,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1484151577298-1971,
> container=default, section=947], offset=566526,
> length=64],offset=0,name=8551433629757608.tsv,size=64] is not the most
> recent version of this FlowFile within this session
> (StandardProcessSession[id=133767])
> 2017-01-11 10:53:39,563 WARN [Timer-Driven Process Thread-9]
> o.a.n.c.t.ContinuallyRunProcessorTask
> org.apache.nifi.processor.exception.FlowFileHandlingException:
> StandardFlowFileRecord[uuid=07f3d75a-3347-45da-8dd4-fecf775d1dcf,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1484151577298-1971,
> container=default, section=947], offset=566526,
> length=64],offset=0,name=8551433629757608.tsv,size=64] is not the most
> recent version of this FlowFile within this session
> (StandardProcessSession[id=133767])
>at
> org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1121)
> ~[nifi-framework-core-1.1.0.jar:1.1.0]
>at
> org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1102)
> ~[nifi-framework-core-1.1.0.jar:1.1.0]
>at org.apache.nifi.processor.util.bin.Bin.offer(Bin.java:142)
> ~[na:na]
>at
> org.apache.nifi.processor.util.bin.BinManager.offer(BinManager.java:194)
> ~[na:na]
>at
> org.apache.nifi.processor.util.bin.BinFiles.binFlowFiles(BinFiles.java:279)
> ~[na:na]
>at
> org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:178)
> ~[na:na]
>at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
> ~[nifi-framework-core-1.1.0.jar:1.1.0]
>at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.1.0.jar:1.1.0]
>at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.1.0.jar:1.1.0]
>at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> [nifi-framework-core-1.1.0.jar:1.1.0]
>at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_65]
>at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_65]
>at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_65]
>at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_65]
>at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_65]
>at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_65]
>at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3329) MergeContent throws FlowFileHandlingException: not the most recent version of this FlowFile within this session

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113757#comment-16113757
 ] 

ASF GitHub Bot commented on NIFI-3329:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2053
  
i cleared out all the repos.  Dropped the queue threshold to 5000 (from 
4) to avoid swapping and i'm not seeing that issue.  I cant reason over 
how/if they're related but just an FYI


> MergeContent throws FlowFileHandlingException: not the most recent version of 
> this FlowFile within this session
> ---
>
> Key: NIFI-3329
> URL: https://issues.apache.org/jira/browse/NIFI-3329
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.1.1
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> The following error was reported on the dev mailing list:
> {code}
> This error occurred in NiFi 1.1.1 (single node) right after an upgrade from
> 1.1.0, so I thought it was due to the upgrade, but now I see that it happens
> if I restart as well and there is data in the flow. 
> This MergeContent processor is sorting incoming FlowFiles into bins based on
> a correlation attribute and dumping out the accumulated uber-FlowFile after
> one hour or 500K messages are accumulated.
> ...
> 2017-01-11 10:53:39,561 WARN [Timer-Driven Process Thread-9]
> o.a.n.processors.standard.MergeContent
> MergeContent[id=b9898788-1c11-45d9-8646-44b1dceb92d5] Processor
> Administratively Yielded for 1 sec due to processing failure
> 2017-01-11 10:53:39,561 WARN [Timer-Driven Process Thread-9]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> MergeContent[id=b9898788-1c11-45d9-8646-44b1dceb92d5] due to uncaught
> Exception: org.apache.nifi.processor.exception.FlowFileHandlingException:
> StandardFlowFileRecord[uuid=07f3d75a-3347-45da-8dd4-fecf775d1dcf,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1484151577298-1971,
> container=default, section=947], offset=566526,
> length=64],offset=0,name=8551433629757608.tsv,size=64] is not the most
> recent version of this FlowFile within this session
> (StandardProcessSession[id=133767])
> 2017-01-11 10:53:39,563 WARN [Timer-Driven Process Thread-9]
> o.a.n.c.t.ContinuallyRunProcessorTask
> org.apache.nifi.processor.exception.FlowFileHandlingException:
> StandardFlowFileRecord[uuid=07f3d75a-3347-45da-8dd4-fecf775d1dcf,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1484151577298-1971,
> container=default, section=947], offset=566526,
> length=64],offset=0,name=8551433629757608.tsv,size=64] is not the most
> recent version of this FlowFile within this session
> (StandardProcessSession[id=133767])
>at
> org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1121)
> ~[nifi-framework-core-1.1.0.jar:1.1.0]
>at
> org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1102)
> ~[nifi-framework-core-1.1.0.jar:1.1.0]
>at org.apache.nifi.processor.util.bin.Bin.offer(Bin.java:142)
> ~[na:na]
>at
> org.apache.nifi.processor.util.bin.BinManager.offer(BinManager.java:194)
> ~[na:na]
>at
> org.apache.nifi.processor.util.bin.BinFiles.binFlowFiles(BinFiles.java:279)
> ~[na:na]
>at
> org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:178)
> ~[na:na]
>at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
> ~[nifi-framework-core-1.1.0.jar:1.1.0]
>at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.1.0.jar:1.1.0]
>at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.1.0.jar:1.1.0]
>at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> [nifi-framework-core-1.1.0.jar:1.1.0]
>at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_65]
>at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_65]
>at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_65]
>at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_65]
>at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_65]
>at
>

[GitHub] nifi issue #2053: NIFI-3329: Removed check for latest version of flowfile wh...

2017-08-03 Thread joewitt

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2053
  
i cleared out all the repos.  Dropped the queue threshold to 5000 (from 
4) to avoid swapping and i'm not seeing that issue.  I cant reason over 
how/if they're related but just an FYI


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #2053: NIFI-3329: Removed check for latest version of flowfile wh...

2017-08-03 Thread joewitt

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2053
  
in reviewing i keep seeing issues with provenance.  Probably unrelated but 
does it make sense?

2017-08-03 17:48:04,182 ERROR [Compress Provenance Logs-1-thread-1] 
o.a.n.p.s.EventFileCompressor Failed to read TOC File 
./provenance_repository/toc/167962066.toc
java.io.EOFException: null
at 
org.apache.nifi.provenance.toc.StandardTocReader.(StandardTocReader.java:48)
at 
org.apache.nifi.provenance.serialization.EventFileCompressor.run(EventFileCompressor.java:93)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3329) MergeContent throws FlowFileHandlingException: not the most recent version of this FlowFile within this session

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113640#comment-16113640
 ] 

ASF GitHub Bot commented on NIFI-3329:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2053
  
good catch.  all the automated builds seem to have tanked but doing one 
last local confirmation.


> MergeContent throws FlowFileHandlingException: not the most recent version of 
> this FlowFile within this session
> ---
>
> Key: NIFI-3329
> URL: https://issues.apache.org/jira/browse/NIFI-3329
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.1.1
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> The following error was reported on the dev mailing list:
> {code}
> This error occurred in NiFi 1.1.1 (single node) right after an upgrade from
> 1.1.0, so I thought it was due to the upgrade, but now I see that it happens
> if I restart as well and there is data in the flow. 
> This MergeContent processor is sorting incoming FlowFiles into bins based on
> a correlation attribute and dumping out the accumulated uber-FlowFile after
> one hour or 500K messages are accumulated.
> ...
> 2017-01-11 10:53:39,561 WARN [Timer-Driven Process Thread-9]
> o.a.n.processors.standard.MergeContent
> MergeContent[id=b9898788-1c11-45d9-8646-44b1dceb92d5] Processor
> Administratively Yielded for 1 sec due to processing failure
> 2017-01-11 10:53:39,561 WARN [Timer-Driven Process Thread-9]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> MergeContent[id=b9898788-1c11-45d9-8646-44b1dceb92d5] due to uncaught
> Exception: org.apache.nifi.processor.exception.FlowFileHandlingException:
> StandardFlowFileRecord[uuid=07f3d75a-3347-45da-8dd4-fecf775d1dcf,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1484151577298-1971,
> container=default, section=947], offset=566526,
> length=64],offset=0,name=8551433629757608.tsv,size=64] is not the most
> recent version of this FlowFile within this session
> (StandardProcessSession[id=133767])
> 2017-01-11 10:53:39,563 WARN [Timer-Driven Process Thread-9]
> o.a.n.c.t.ContinuallyRunProcessorTask
> org.apache.nifi.processor.exception.FlowFileHandlingException:
> StandardFlowFileRecord[uuid=07f3d75a-3347-45da-8dd4-fecf775d1dcf,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1484151577298-1971,
> container=default, section=947], offset=566526,
> length=64],offset=0,name=8551433629757608.tsv,size=64] is not the most
> recent version of this FlowFile within this session
> (StandardProcessSession[id=133767])
>at
> org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1121)
> ~[nifi-framework-core-1.1.0.jar:1.1.0]
>at
> org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1102)
> ~[nifi-framework-core-1.1.0.jar:1.1.0]
>at org.apache.nifi.processor.util.bin.Bin.offer(Bin.java:142)
> ~[na:na]
>at
> org.apache.nifi.processor.util.bin.BinManager.offer(BinManager.java:194)
> ~[na:na]
>at
> org.apache.nifi.processor.util.bin.BinFiles.binFlowFiles(BinFiles.java:279)
> ~[na:na]
>at
> org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:178)
> ~[na:na]
>at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
> ~[nifi-framework-core-1.1.0.jar:1.1.0]
>at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.1.0.jar:1.1.0]
>at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.1.0.jar:1.1.0]
>at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> [nifi-framework-core-1.1.0.jar:1.1.0]
>at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_65]
>at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_65]
>at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_65]
>at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_65]
>at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_65]
>at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_65]
>at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
>

[GitHub] nifi issue #2053: NIFI-3329: Removed check for latest version of flowfile wh...

2017-08-03 Thread joewitt

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2053
  
good catch.  all the automated builds seem to have tanked but doing one 
last local confirmation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4262) MergeContent - option to add merged uuid in original flow files

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113544#comment-16113544
 ] 

ASF GitHub Bot commented on NIFI-4262:
--

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/2056

NIFI-4262 - MergeContent - option to add merged uuid in original flow…

… files

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-4262

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2056.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2056


commit bf77df17e8e00d32e1f71922774cf68dd6ba8c2d
Author: Pierre Villard 
Date:   2017-08-03T21:40:48Z

NIFI-4262 - MergeContent - option to add merged uuid in original flow files




> MergeContent - option to add merged uuid in original flow files
> ---
>
> Key: NIFI-4262
> URL: https://issues.apache.org/jira/browse/NIFI-4262
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> With the apparition of Wait/Notify processors it is now possible tackle new 
> challenges when it comes to synchronize the execution of different parts of 
> the workflow.
> The objective here is the following:
> Flow files are sent to a MergeContent processor. Merged flow files are then 
> sent to a processor A while original flow files are sent to a processor B. I 
> want to trigger processor B when and only when processor A has completed.
> To use the Wait/Notify approach, a common attribute must be available to be 
> used as a signal in the distributed cache. This JIRA is about adding a 
> processor property allowing a user to add the UUID of the merged flow file as 
> a new attribute of all the original flow files that are constituting the 
> merged flow file.
> The template attached to NIFI-4028 can be used for this use case. Note that 
> the fix for NIFI-4028 is needed to solve the use case described in this JIRA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4262) MergeContent - option to add merged uuid in original flow files

2017-08-03 Thread Pierre Villard (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4262:
-
Status: Patch Available  (was: Open)

> MergeContent - option to add merged uuid in original flow files
> ---
>
> Key: NIFI-4262
> URL: https://issues.apache.org/jira/browse/NIFI-4262
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> With the apparition of Wait/Notify processors it is now possible tackle new 
> challenges when it comes to synchronize the execution of different parts of 
> the workflow.
> The objective here is the following:
> Flow files are sent to a MergeContent processor. Merged flow files are then 
> sent to a processor A while original flow files are sent to a processor B. I 
> want to trigger processor B when and only when processor A has completed.
> To use the Wait/Notify approach, a common attribute must be available to be 
> used as a signal in the distributed cache. This JIRA is about adding a 
> processor property allowing a user to add the UUID of the merged flow file as 
> a new attribute of all the original flow files that are constituting the 
> merged flow file.
> The template attached to NIFI-4028 can be used for this use case. Note that 
> the fix for NIFI-4028 is needed to solve the use case described in this JIRA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2056: NIFI-4262 - MergeContent - option to add merged uui...

2017-08-03 Thread pvillard31

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/2056

NIFI-4262 - MergeContent - option to add merged uuid in original flowâ¦

â¦ files

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-4262

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2056.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2056


commit bf77df17e8e00d32e1f71922774cf68dd6ba8c2d
Author: Pierre Villard 
Date:   2017-08-03T21:40:48Z

NIFI-4262 - MergeContent - option to add merged uuid in original flow files




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-4262) MergeContent - option to add merged uuid in original flow files

2017-08-03 Thread Pierre Villard (JIRA)

Pierre Villard created NIFI-4262:


 Summary: MergeContent - option to add merged uuid in original flow 
files
 Key: NIFI-4262
 URL: https://issues.apache.org/jira/browse/NIFI-4262
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Reporter: Pierre Villard
Assignee: Pierre Villard


With the apparition of Wait/Notify processors it is now possible tackle new 
challenges when it comes to synchronize the execution of different parts of the 
workflow.

The objective here is the following:
Flow files are sent to a MergeContent processor. Merged flow files are then 
sent to a processor A while original flow files are sent to a processor B. I 
want to trigger processor B when and only when processor A has completed.

To use the Wait/Notify approach, a common attribute must be available to be 
used as a signal in the distributed cache. This JIRA is about adding a 
processor property allowing a user to add the UUID of the merged flow file as a 
new attribute of all the original flow files that are constituting the merged 
flow file.

The template attached to NIFI-4028 can be used for this use case. Note that the 
fix for NIFI-4028 is needed to solve the use case described in this JIRA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2020: [NiFi-3973] Add PutKudu Processor for ingesting dat...

2017-08-03 Thread cammachusa

Github user cammachusa commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2020#discussion_r131258877
  
--- Diff: 
nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-processors/src/main/java/org/apache/nifi/processors/kudu/AbstractKudu.java
 ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.kudu;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.kudu.client.KuduClient;
+import org.apache.kudu.client.KuduException;
+import org.apache.kudu.client.KuduSession;
+import org.apache.kudu.client.KuduTable;
+import org.apache.kudu.client.Insert;
+import org.apache.kudu.client.Upsert;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.FlowFileAccessException;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.hadoop.exception.RecordReaderFactoryException;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.RecordSet;
+import org.apache.nifi.serialization.record.Record;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+
+public abstract class AbstractKudu extends AbstractProcessor {
+
+protected static final PropertyDescriptor KUDU_MASTERS = new 
PropertyDescriptor.Builder()
+.name("KUDU Masters")
+.description("List all kudu masters's ip with port (e.g. 
7051), comma separated")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+protected static final PropertyDescriptor TABLE_NAME = new 
PropertyDescriptor.Builder()
+.name("Table Name")
+.description("The name of the Kudu Table to put data into")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor RECORD_READER = new 
PropertyDescriptor.Builder()
+.name("record-reader")
+.displayName("Record Reader")
+.description("The service for reading records from incoming 
flow files.")
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+protected static final PropertyDescriptor SKIP_HEAD_LINE = new 
PropertyDescriptor.Builder()
+.name("Skip head line")
+.description("Set it to true if your first line is the header 
line e.g. column names")
+.allowableValues("true", "false")
+.defaultValue("true")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+protected static final PropertyDescriptor INSERT_OPERATION = new 
PropertyDescriptor.Builder()
+.name("Insert Operation")
+.description("Specify operation for this processor. 
Insert-Ignore will ignore duplicated rows")
+.allowableValues(OperationType.INSERT.toString(), 
OperationType.INSERT_IGNORE.toString(), OperationType.UPSERT.toString())
--- End diff --

:+1: It makes the code clean. Why not :-)


---
If your project

[GitHub] nifi pull request #2020: [NiFi-3973] Add PutKudu Processor for ingesting dat...

2017-08-03 Thread cammachusa

Github user cammachusa commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2020#discussion_r131258450
  
--- Diff: 
nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-processors/src/main/java/org/apache/nifi/processors/kudu/PutKudu.java
 ---
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.kudu;
+
+import org.apache.kudu.Schema;
+import org.apache.kudu.Type;
+import org.apache.kudu.client.Insert;
+import org.apache.kudu.client.Upsert;
+import org.apache.kudu.client.PartialRow;
+import org.apache.kudu.client.KuduTable;
+import org.apache.kudu.client.Operation;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.serialization.record.Record;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@EventDriven
+@SupportsBatching
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"put", "database", "NoSQL", "kudu", "HDFS"})
+@CapabilityDescription("Reads records from an incoming FlowFile using the 
provided Record Reader, and writes those records " +
+"to the specified Kudu's table. The schema for the table must be 
provided in the processor properties or from your source." +
+" If any error occurs while reading records from the input, or 
writing records to Kudu, the FlowFile will be routed to failure")
+
+public class PutKudu extends AbstractKudu {
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+final List properties = new ArrayList<>();
+properties.add(KUDU_MASTERS);
+properties.add(TABLE_NAME);
+properties.add(SKIP_HEAD_LINE);
+properties.add(RECORD_READER);
+properties.add(INSERT_OPERATION);
+
+return properties;
+}
+
+@Override
+public Set getRelationships() {
+final Set rels = new HashSet<>();
+rels.add(REL_SUCCESS);
+rels.add(REL_FAILURE);
+return rels;
+}
+
+@Override
+protected Upsert upsertRecordToKudu(KuduTable kuduTable, Record 
record, List fieldNames) throws IllegalStateException, Exception {
+Upsert upsert = kuduTable.newUpsert();
+this.insert(kuduTable, upsert, record, fieldNames);
+return upsert;
+}
+
+@Override
+protected Insert insertRecordToKudu(KuduTable kuduTable, Record 
record, List fieldNames) throws IllegalStateException, Exception {
+Insert insert = kuduTable.newInsert();
+this.insert(kuduTable, insert, record, fieldNames);
+return insert;
+}
+
+private void insert(KuduTable kuduTable, Operation operation, Record 
record, List fieldNames){
+PartialRow row = operation.getRow();
+Schema colSchema = kuduTable.getSchema();
+
+for (String colName : fieldNames) {
+int colIdx = this.getColumnIndex(colSchema, colName);
+if (colIdx != -1) {
+Type colType = 
colSchema.getColumnByIndex(colIdx).getType();
+
+switch (colType.getDataType()) {
+case BOOL:
+row.addBoolean(colIdx, 
record.getAsBoolean(colName));
+break;
+case FLOAT:
+row.addFloat(colIdx, record.getAsFloat(colName));
+break;
+case DOUBLE:
+row.addDouble(colIdx,

[GitHub] nifi pull request #2020: [NiFi-3973] Add PutKudu Processor for ingesting dat...

2017-08-03 Thread cammachusa

Github user cammachusa commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2020#discussion_r131257922
  
--- Diff: 
nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-processors/src/main/java/org/apache/nifi/processors/kudu/PutKudu.java
 ---
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.kudu;
+
+import org.apache.kudu.Schema;
+import org.apache.kudu.Type;
+import org.apache.kudu.client.Insert;
+import org.apache.kudu.client.Upsert;
+import org.apache.kudu.client.PartialRow;
+import org.apache.kudu.client.KuduTable;
+import org.apache.kudu.client.Operation;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.serialization.record.Record;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@EventDriven
+@SupportsBatching
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"put", "database", "NoSQL", "kudu", "HDFS"})
+@CapabilityDescription("Reads records from an incoming FlowFile using the 
provided Record Reader, and writes those records " +
+"to the specified Kudu's table. The schema for the table must be 
provided in the processor properties or from your source." +
+" If any error occurs while reading records from the input, or 
writing records to Kudu, the FlowFile will be routed to failure")
+
+public class PutKudu extends AbstractKudu {
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+final List properties = new ArrayList<>();
+properties.add(KUDU_MASTERS);
+properties.add(TABLE_NAME);
+properties.add(SKIP_HEAD_LINE);
+properties.add(RECORD_READER);
+properties.add(INSERT_OPERATION);
+
+return properties;
+}
+
+@Override
+public Set getRelationships() {
+final Set rels = new HashSet<>();
+rels.add(REL_SUCCESS);
+rels.add(REL_FAILURE);
+return rels;
+}
+
+@Override
+protected Upsert upsertRecordToKudu(KuduTable kuduTable, Record 
record, List fieldNames) throws IllegalStateException, Exception {
+Upsert upsert = kuduTable.newUpsert();
+this.insert(kuduTable, upsert, record, fieldNames);
+return upsert;
+}
+
+@Override
+protected Insert insertRecordToKudu(KuduTable kuduTable, Record 
record, List fieldNames) throws IllegalStateException, Exception {
+Insert insert = kuduTable.newInsert();
+this.insert(kuduTable, insert, record, fieldNames);
+return insert;
+}
+
+private void insert(KuduTable kuduTable, Operation operation, Record 
record, List fieldNames){
+PartialRow row = operation.getRow();
+Schema colSchema = kuduTable.getSchema();
+
+for (String colName : fieldNames) {
+int colIdx = this.getColumnIndex(colSchema, colName);
+if (colIdx != -1) {
+Type colType = 
colSchema.getColumnByIndex(colIdx).getType();
+
+switch (colType.getDataType()) {
+case BOOL:
+row.addBoolean(colIdx, 
record.getAsBoolean(colName));
+break;
+case FLOAT:
+row.addFloat(colIdx, record.getAsFloat(colName));
+break;
+case DOUBLE:
+row.addDouble(colIdx,

[jira] [Commented] (NIFI-4028) Wait processor removes cache regardless of the remaing counter

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113471#comment-16113471
 ] 

ASF GitHub Bot commented on NIFI-4028:
--

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/2055

NIFI-4028 - fix cache update when Wait releases flow files

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-4028

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2055.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2055


commit 5adcb0b2db68eef18fb7bcc84634436d36c97be7
Author: Pierre Villard 
Date:   2017-08-03T20:56:27Z

NIFI-4028 - fix cache update when Wait releases flow files




> Wait processor removes cache regardless of the remaing counter
> --
>
> Key: NIFI-4028
> URL: https://issues.apache.org/jira/browse/NIFI-4028
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Pierre Villard
> Attachments: WaitNotify_NIFI-4028.xml
>
>
> Current [Wait processor implementation removes cache 
> entry|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/Wait.java#L438]
>  if it's configured with 'Releasable FlowFile Count' is one, when it finds a 
> wait signal reaches to 'Target Signal Count' no matter how large the current 
> count is.
> This behavior can be problematic, when an user expects the same Signal 
> Identifier to be released multiple FlowFiles one by one while multiple counts 
> to be accumulated by Notify.
> For example this scenario does not work:
> {code}
> 1. Notify notifies to a 'signal-a' + 1 => 'signal-a' = 1
> 2. Notify notifies to a 'signal-a' again, +1 => 'signal-a' = 2
> 3. Wait releases 1 FlowFile, then remove the 'signal-a' => 'signal-a' is 
> removed
> 4. Wait will not be able to release another FlowFile
> {code}
> This should be fixed as follows:
> {code}
> 1. Notify notifies to a 'signal-a' + 1 => 'signal-a' = 1
> 2. Notify notifies to a 'signal-a' again, +1 => 'signal-a' = 2
> 3. Wait releases 1 FlowFile, then update the 'signal-a', -1 => 'signal-a' = 1
> 4. Wait releases another FlowFile, then 'signal-a' becomes 0, remove 
> 'signal-a'
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4028) Wait processor removes cache regardless of the remaing counter

2017-08-03 Thread Pierre Villard (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4028:
-
Status: Patch Available  (was: Open)

> Wait processor removes cache regardless of the remaing counter
> --
>
> Key: NIFI-4028
> URL: https://issues.apache.org/jira/browse/NIFI-4028
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Pierre Villard
> Attachments: WaitNotify_NIFI-4028.xml
>
>
> Current [Wait processor implementation removes cache 
> entry|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/Wait.java#L438]
>  if it's configured with 'Releasable FlowFile Count' is one, when it finds a 
> wait signal reaches to 'Target Signal Count' no matter how large the current 
> count is.
> This behavior can be problematic, when an user expects the same Signal 
> Identifier to be released multiple FlowFiles one by one while multiple counts 
> to be accumulated by Notify.
> For example this scenario does not work:
> {code}
> 1. Notify notifies to a 'signal-a' + 1 => 'signal-a' = 1
> 2. Notify notifies to a 'signal-a' again, +1 => 'signal-a' = 2
> 3. Wait releases 1 FlowFile, then remove the 'signal-a' => 'signal-a' is 
> removed
> 4. Wait will not be able to release another FlowFile
> {code}
> This should be fixed as follows:
> {code}
> 1. Notify notifies to a 'signal-a' + 1 => 'signal-a' = 1
> 2. Notify notifies to a 'signal-a' again, +1 => 'signal-a' = 2
> 3. Wait releases 1 FlowFile, then update the 'signal-a', -1 => 'signal-a' = 1
> 4. Wait releases another FlowFile, then 'signal-a' becomes 0, remove 
> 'signal-a'
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2055: NIFI-4028 - fix cache update when Wait releases flo...

2017-08-03 Thread pvillard31

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/2055

NIFI-4028 - fix cache update when Wait releases flow files

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-4028

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2055.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2055


commit 5adcb0b2db68eef18fb7bcc84634436d36c97be7
Author: Pierre Villard 
Date:   2017-08-03T20:56:27Z

NIFI-4028 - fix cache update when Wait releases flow files




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #2054: NIFI-4051: Ensure that we use the appropriate Lexic...

2017-08-03 Thread markap14

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2054

NIFI-4051: Ensure that we use the appropriate Lexical syntax for SQL â¦

â¦query validation as we do when running

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4051

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2054.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2054


commit 1e7aa95ad6abd177069507883d84e3380f879207
Author: Mark Payne 
Date:   2017-08-03T20:56:10Z

NIFI-4051: Ensure that we use the appropriate Lexical syntax for SQL query 
validation as we do when running




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-4028) Wait processor removes cache regardless of the remaing counter

2017-08-03 Thread Pierre Villard (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4028:
-
Attachment: WaitNotify_NIFI-4028.xml

Attached a template to reproduce the context / test the patch.

> Wait processor removes cache regardless of the remaing counter
> --
>
> Key: NIFI-4028
> URL: https://issues.apache.org/jira/browse/NIFI-4028
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Attachments: WaitNotify_NIFI-4028.xml
>
>
> Current [Wait processor implementation removes cache 
> entry|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/Wait.java#L438]
>  if it's configured with 'Releasable FlowFile Count' is one, when it finds a 
> wait signal reaches to 'Target Signal Count' no matter how large the current 
> count is.
> This behavior can be problematic, when an user expects the same Signal 
> Identifier to be released multiple FlowFiles one by one while multiple counts 
> to be accumulated by Notify.
> For example this scenario does not work:
> {code}
> 1. Notify notifies to a 'signal-a' + 1 => 'signal-a' = 1
> 2. Notify notifies to a 'signal-a' again, +1 => 'signal-a' = 2
> 3. Wait releases 1 FlowFile, then remove the 'signal-a' => 'signal-a' is 
> removed
> 4. Wait will not be able to release another FlowFile
> {code}
> This should be fixed as follows:
> {code}
> 1. Notify notifies to a 'signal-a' + 1 => 'signal-a' = 1
> 2. Notify notifies to a 'signal-a' again, +1 => 'signal-a' = 2
> 3. Wait releases 1 FlowFile, then update the 'signal-a', -1 => 'signal-a' = 1
> 4. Wait releases another FlowFile, then 'signal-a' becomes 0, remove 
> 'signal-a'
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2020: [NiFi-3973] Add PutKudu Processor for ingesting dat...

2017-08-03 Thread cammachusa

Github user cammachusa commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2020#discussion_r131257343
  
--- Diff: 
nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-processors/src/main/java/org/apache/nifi/processors/kudu/AbstractKudu.java
 ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.kudu;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.kudu.client.KuduClient;
+import org.apache.kudu.client.KuduException;
+import org.apache.kudu.client.KuduSession;
+import org.apache.kudu.client.KuduTable;
+import org.apache.kudu.client.Insert;
+import org.apache.kudu.client.Upsert;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.FlowFileAccessException;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.hadoop.exception.RecordReaderFactoryException;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.RecordSet;
+import org.apache.nifi.serialization.record.Record;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+
+public abstract class AbstractKudu extends AbstractProcessor {
+
+protected static final PropertyDescriptor KUDU_MASTERS = new 
PropertyDescriptor.Builder()
+.name("KUDU Masters")
+.description("List all kudu masters's ip with port (e.g. 
7051), comma separated")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+protected static final PropertyDescriptor TABLE_NAME = new 
PropertyDescriptor.Builder()
+.name("Table Name")
+.description("The name of the Kudu Table to put data into")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor RECORD_READER = new 
PropertyDescriptor.Builder()
+.name("record-reader")
+.displayName("Record Reader")
+.description("The service for reading records from incoming 
flow files.")
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+protected static final PropertyDescriptor SKIP_HEAD_LINE = new 
PropertyDescriptor.Builder()
+.name("Skip head line")
+.description("Set it to true if your first line is the header 
line e.g. column names")
+.allowableValues("true", "false")
+.defaultValue("true")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+protected static final PropertyDescriptor INSERT_OPERATION = new 
PropertyDescriptor.Builder()
+.name("Insert Operation")
+.description("Specify operation for this processor. 
Insert-Ignore will ignore duplicated rows")
+.allowableValues(OperationType.INSERT.toString(), 
OperationType.INSERT_IGNORE.toString(), OperationType.UPSERT.toString())
+.defaultValue("Insert")
+

[jira] [Commented] (NIFI-3196) Provide MiNiFi FlowStatus Insights processor that integrates MiNiFi Flow Status Report with Azure Application Insights

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113436#comment-16113436
 ] 

ASF GitHub Bot commented on NIFI-3196:
--

Github user jzonthemtn commented on the issue:

https://github.com/apache/nifi/pull/1331
  
@apsaltis Just found this and I'm glad I did. It looks very nice. I'm 
facing the same type of task except with AWS CloudWatch. I wanted to ask if you 
are still continuing development toward what was discussed earlier in this 
conversation? Thanks!


> Provide MiNiFi FlowStatus Insights processor that integrates MiNiFi Flow 
> Status Report with Azure Application Insights
> --
>
> Key: NIFI-3196
> URL: https://issues.apache.org/jira/browse/NIFI-3196
> Project: Apache NiFi
>  Issue Type: New Feature
>Affects Versions: 1.1.0
>Reporter: Andrew Psaltis
>Assignee: Andrew Psaltis
>
> As a user of both NiFi and MiNiFi there are many times I would like to have 
> an operational dashboard for MiNiFi. Although, there is a discussion and 
> desire for [Centralized management | 
> https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Community+Driven+Requirements]
>  there are many times where as a user I just want to be able to use the 
> operational tools I already have. In many cases those may be products such 
> as:  Nagios, Zenoos, Zabix, Graphite, Grafana and in cloud environments 
> perhaps Azure Application Insights and AWS Cloudwatch. 
> This JIRA and the related P/R provides a processor that ingests the MiNiFi 
> Flow Status report and sends it to Azure Application Insights as custom 
> metrics. This then allows a user to set alerts in Azure Application Insights, 
> thus providing the ability to monitor MiNiFi agents. As such, this has a 
> dependency on the following MiNiFi JIRA: [Change FlowStatusReport to be JSON 
> | https://issues.apache.org/jira/browse/MINIFI-170]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1331: NIFI-3196 Provide MiNiFi FlowStatus Insights processor tha...

2017-08-03 Thread jzonthemtn

Github user jzonthemtn commented on the issue:

https://github.com/apache/nifi/pull/1331
  
@apsaltis Just found this and I'm glad I did. It looks very nice. I'm 
facing the same type of task except with AWS CloudWatch. I wanted to ask if you 
are still continuing development toward what was discussed earlier in this 
conversation? Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #2020: [NiFi-3973] Add PutKudu Processor for ingesting dat...

2017-08-03 Thread rickysaltzer

Github user rickysaltzer commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2020#discussion_r131244333
  
--- Diff: 
nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-processors/src/main/java/org/apache/nifi/processors/kudu/AbstractKudu.java
 ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.kudu;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.kudu.client.KuduClient;
+import org.apache.kudu.client.KuduException;
+import org.apache.kudu.client.KuduSession;
+import org.apache.kudu.client.KuduTable;
+import org.apache.kudu.client.Insert;
+import org.apache.kudu.client.Upsert;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.FlowFileAccessException;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.hadoop.exception.RecordReaderFactoryException;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.RecordSet;
+import org.apache.nifi.serialization.record.Record;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+
+public abstract class AbstractKudu extends AbstractProcessor {
+
+protected static final PropertyDescriptor KUDU_MASTERS = new 
PropertyDescriptor.Builder()
+.name("KUDU Masters")
+.description("List all kudu masters's ip with port (e.g. 
7051), comma separated")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+protected static final PropertyDescriptor TABLE_NAME = new 
PropertyDescriptor.Builder()
+.name("Table Name")
+.description("The name of the Kudu Table to put data into")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor RECORD_READER = new 
PropertyDescriptor.Builder()
+.name("record-reader")
+.displayName("Record Reader")
+.description("The service for reading records from incoming 
flow files.")
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+protected static final PropertyDescriptor SKIP_HEAD_LINE = new 
PropertyDescriptor.Builder()
+.name("Skip head line")
+.description("Set it to true if your first line is the header 
line e.g. column names")
+.allowableValues("true", "false")
+.defaultValue("true")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+protected static final PropertyDescriptor INSERT_OPERATION = new 
PropertyDescriptor.Builder()
+.name("Insert Operation")
+.description("Specify operation for this processor. 
Insert-Ignore will ignore duplicated rows")
+.allowableValues(OperationType.INSERT.toString(), 
OperationType.INSERT_IGNORE.toString(), OperationType.UPSERT.toString())
+.defaultValue("Insert")
+

[jira] [Commented] (NIFI-3329) MergeContent throws FlowFileHandlingException: not the most recent version of this FlowFile within this session

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113382#comment-16113382
 ] 

ASF GitHub Bot commented on NIFI-3329:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2053

NIFI-3329: Removed check for latest version of flowfile when migratin…

…g flowfile from 1 session to another because we now longer are requiring 
latest version of flowfile be passed to session anywhere else. It was intended 
to be removed from StandardProcessSession.migrate when it was removed from 
StandardProcessSession.validateRecordState but it was overlooked.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-3329

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2053.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2053


commit ef0785f37a6bd410b7dd66d1ce6107ed9b5d9c07
Author: Mark Payne 
Date:   2017-08-03T19:55:19Z

NIFI-3329: Removed check for latest version of flowfile when migrating 
flowfile from 1 session to another because we now longer are requiring latest 
version of flowfile be passed to session anywhere else. It was intended to be 
removed from StandardProcessSession.migrate when it was removed from 
StandardProcessSession.validateRecordState but it was overlooked.




> MergeContent throws FlowFileHandlingException: not the most recent version of 
> this FlowFile within this session
> ---
>
> Key: NIFI-3329
> URL: https://issues.apache.org/jira/browse/NIFI-3329
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.1.1
>Reporter: Mark Payne
> Fix For: 1.4.0
>
>
> The following error was reported on the dev mailing list:
> {code}
> This error occurred in NiFi 1.1.1 (single node) right after an upgrade from
> 1.1.0, so I thought it was due to the upgrade, but now I see that it happens
> if I restart as well and there is data in the flow. 
> This MergeContent processor is sorting incoming FlowFiles into bins based on
> a correlation attribute and dumping out the accumulated uber-FlowFile after
> one hour or 500K messages are accumulated.
> ...
> 2017-01-11 10:53:39,561 WARN [Timer-Driven Process Thread-9]
> o.a.n.processors.standard.MergeContent
> MergeContent[id=b9898788-1c11-45d9-8646-44b1dceb92d5] Processor
> Administratively Yielded for 1 sec due to processing failure
> 2017-01-11 10:53:39,561 WARN [Timer-Driven Process Thread-9]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> MergeContent[id=b9898788-1c11-45d9-8646-44b1dceb92d5] due to uncaught
> Exception: org.apache.nifi.processor.exception.FlowFileHandlingException:
>

[GitHub] nifi pull request #2053: NIFI-3329: Removed check for latest version of flow...

2017-08-03 Thread markap14

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2053

NIFI-3329: Removed check for latest version of flowfile when migratinâ¦

â¦g flowfile from 1 session to another because we now longer are requiring 
latest version of flowfile be passed to session anywhere else. It was intended 
to be removed from StandardProcessSession.migrate when it was removed from 
StandardProcessSession.validateRecordState but it was overlooked.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-3329

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2053.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2053


commit ef0785f37a6bd410b7dd66d1ce6107ed9b5d9c07
Author: Mark Payne 
Date:   2017-08-03T19:55:19Z

NIFI-3329: Removed check for latest version of flowfile when migrating 
flowfile from 1 session to another because we now longer are requiring latest 
version of flowfile be passed to session anywhere else. It was intended to be 
removed from StandardProcessSession.migrate when it was removed from 
StandardProcessSession.validateRecordState but it was overlooked.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4024) Create EvaluateRecordPath processor

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113377#comment-16113377
 ] 

ASF GitHub Bot commented on NIFI-4024:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/1961
  
Looks good to me.


> Create EvaluateRecordPath processor
> ---
>
> Key: NIFI-4024
> URL: https://issues.apache.org/jira/browse/NIFI-4024
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Steve Champagne
>Priority: Minor
>
> With the new RecordPath DSL, it would be nice if there was a processor that 
> could pull fields into attributes of the flowfile based on a RecordPath. This 
> would be similar to the EvaluateJsonPath processor that currently exists, 
> except it could be used to pull fields from arbitrary record formats. My 
> current use case for it would be pulling fields out of Avro records while 
> skipping the steps of having to convert Avro to JSON, evaluate JsonPath, and 
> then converting back to Avro. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1961: NIFI-4024 Added org.apache.nifi.hbase.PutHBaseRecord

2017-08-03 Thread MikeThomsen

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/1961
  
Looks good to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #2020: [NiFi-3973] Add PutKudu Processor for ingesting dat...

2017-08-03 Thread rickysaltzer

Github user rickysaltzer commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2020#discussion_r131242503
  
--- Diff: 
nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-processors/src/main/java/org/apache/nifi/processors/kudu/AbstractKudu.java
 ---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.kudu;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.kudu.client.KuduClient;
+import org.apache.kudu.client.KuduException;
+import org.apache.kudu.client.KuduSession;
+import org.apache.kudu.client.KuduTable;
+import org.apache.kudu.client.Insert;
+import org.apache.kudu.client.Upsert;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.FlowFileAccessException;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.hadoop.exception.RecordReaderFactoryException;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.RecordSet;
+import org.apache.nifi.serialization.record.Record;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+
+public abstract class AbstractKudu extends AbstractProcessor {
+
+protected static final PropertyDescriptor KUDU_MASTERS = new 
PropertyDescriptor.Builder()
+.name("KUDU Masters")
+.description("List all kudu masters's ip with port (e.g. 
7051), comma separated")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+protected static final PropertyDescriptor TABLE_NAME = new 
PropertyDescriptor.Builder()
+.name("Table Name")
+.description("The name of the Kudu Table to put data into")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor RECORD_READER = new 
PropertyDescriptor.Builder()
+.name("record-reader")
+.displayName("Record Reader")
+.description("The service for reading records from incoming 
flow files.")
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+protected static final PropertyDescriptor SKIP_HEAD_LINE = new 
PropertyDescriptor.Builder()
+.name("Skip head line")
+.description("Set it to true if your first line is the header 
line e.g. column names")
+.allowableValues("true", "false")
+.defaultValue("true")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+protected static final PropertyDescriptor INSERT_OPERATION = new 
PropertyDescriptor.Builder()
+.name("Insert Operation")
+.description("Specify operation for this processor. 
Insert-Ignore will ignore duplicated rows")
+.allowableValues(OperationType.INSERT.toString(), 
OperationType.INSERT_IGNORE.toString(), OperationType.UPSERT.toString())
--- End diff --

Little trick to note, you can use `OperationType.values()`. You

[jira] [Commented] (NIFI-4215) Avro schemas with records that have a field of themselves fail to parse, causing stackoverflow exception

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113321#comment-16113321
 ] 

ASF GitHub Bot commented on NIFI-4215:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2052


> Avro schemas with records that have a field of themselves fail to parse, 
> causing stackoverflow exception
> 
>
> Key: NIFI-4215
> URL: https://issues.apache.org/jira/browse/NIFI-4215
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Wesley L Lawrence
>Priority: Minor
> Fix For: 1.4.0
>
> Attachments: nifi-4215.patch
>
>
> Noticed this while attempting to use the AvroSchemaRegsitry with some complex 
> schema. Boiled down, Avro lets you define a schema such as;
> {code}
> { 
>   "namespace": "org.apache.nifi.testing", 
>   "name": "CompositRecord", 
>   "type": "record", 
>   "fields": [ 
> { 
>   "name": "id", 
>   "type": "int" 
> }, 
> { 
>   "name": "value", 
>   "type": "string" 
> }, 
> { 
>   "name": "parent", 
>   "type": [
> "null",
> "CompositRecord"
>   ]
> } 
>   ] 
> }
> {code}
> The AvroSchemaRegistry (AvroTypeUtil specifically) will fail to parse, and 
> generate a stackoverflow exception.
> I've whipped up a fix, tested it out in 1.4.0, and am just running through 
> the contrib build before I submit a patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-4215) Avro schemas with records that have a field of themselves fail to parse, causing stackoverflow exception

2017-08-03 Thread Mark Payne (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne resolved NIFI-4215.
--
   Resolution: Fixed
Fix Version/s: 1.4.0

> Avro schemas with records that have a field of themselves fail to parse, 
> causing stackoverflow exception
> 
>
> Key: NIFI-4215
> URL: https://issues.apache.org/jira/browse/NIFI-4215
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Wesley L Lawrence
>Priority: Minor
> Fix For: 1.4.0
>
> Attachments: nifi-4215.patch
>
>
> Noticed this while attempting to use the AvroSchemaRegsitry with some complex 
> schema. Boiled down, Avro lets you define a schema such as;
> {code}
> { 
>   "namespace": "org.apache.nifi.testing", 
>   "name": "CompositRecord", 
>   "type": "record", 
>   "fields": [ 
> { 
>   "name": "id", 
>   "type": "int" 
> }, 
> { 
>   "name": "value", 
>   "type": "string" 
> }, 
> { 
>   "name": "parent", 
>   "type": [
> "null",
> "CompositRecord"
>   ]
> } 
>   ] 
> }
> {code}
> The AvroSchemaRegistry (AvroTypeUtil specifically) will fail to parse, and 
> generate a stackoverflow exception.
> I've whipped up a fix, tested it out in 1.4.0, and am just running through 
> the contrib build before I submit a patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2052: NIFI-4215 NiFi can now parse an Avro schema of a re...

2017-08-03 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2052


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4215) Avro schemas with records that have a field of themselves fail to parse, causing stackoverflow exception

2017-08-03 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113319#comment-16113319
 ] 

ASF subversion and git services commented on NIFI-4215:
---

Commit 40cde0466a31e7378553dea2e9a9101a057f083a in nifi's branch 
refs/heads/master from [~WesleyLawrence]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=40cde04 ]

NIFI-4215 NiFi can now parse an Avro schema of a record that references an 
already defined record, including itself.

This closes #2052.


> Avro schemas with records that have a field of themselves fail to parse, 
> causing stackoverflow exception
> 
>
> Key: NIFI-4215
> URL: https://issues.apache.org/jira/browse/NIFI-4215
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Wesley L Lawrence
>Priority: Minor
> Attachments: nifi-4215.patch
>
>
> Noticed this while attempting to use the AvroSchemaRegsitry with some complex 
> schema. Boiled down, Avro lets you define a schema such as;
> {code}
> { 
>   "namespace": "org.apache.nifi.testing", 
>   "name": "CompositRecord", 
>   "type": "record", 
>   "fields": [ 
> { 
>   "name": "id", 
>   "type": "int" 
> }, 
> { 
>   "name": "value", 
>   "type": "string" 
> }, 
> { 
>   "name": "parent", 
>   "type": [
> "null",
> "CompositRecord"
>   ]
> } 
>   ] 
> }
> {code}
> The AvroSchemaRegistry (AvroTypeUtil specifically) will fail to parse, and 
> generate a stackoverflow exception.
> I've whipped up a fix, tested it out in 1.4.0, and am just running through 
> the contrib build before I submit a patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-minifi pull request #88: MINIFI-347: Adding tests for C2 file system ca...

2017-08-03 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi/pull/88


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #2020: [NiFi-3973] Add PutKudu Processor for ingesting data to Ku...

2017-08-03 Thread rickysaltzer

Github user rickysaltzer commented on the issue:

https://github.com/apache/nifi/pull/2020
  
I will continue reviewing later today or tomorrow, thanks for your 
patience. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4224) Add Variable Registry at Process Group level

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113249#comment-16113249
 ] 

ASF GitHub Bot commented on NIFI-4224:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2051
  
Will review...


> Add Variable Registry at Process Group level
> 
>
> Key: NIFI-4224
> URL: https://issues.apache.org/jira/browse/NIFI-4224
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>
> Currently, NiFi exposes a variable registry that is configurable by adding 
> the name of a properties file to nifi.properties and then treating the 
> referenced properties file as key/value pairs for the variable registry. 
> This, however, is very limiting, as it provides a global scope for all 
> variables, and it requires a restart of NiFi in order to pick up any updates 
> to the file. We should expose a Process Group-level Variable Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2051: NIFI-4224: Initial implementation of Process Group level V...

2017-08-03 Thread mcgilman

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2051
  
Will review...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3747) Add link to context sensitive Help within configuration dialogs

2017-08-03 Thread Yuri (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113229#comment-16113229
 ] 

Yuri commented on NIFI-3747:


>From my point of view, this can be solved by adding existing _Usage_ button + 
>link also to processor/service/task _Configure_ dialog.

> Add link to context sensitive Help within configuration dialogs
> ---
>
> Key: NIFI-3747
> URL: https://issues.apache.org/jira/browse/NIFI-3747
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Andrew Lim
>Priority: Minor
>
> When the user is in the middle of configuring a processor, a controller 
> service, a reporting task, etc. it is done within a dialog.  But that dialog 
> does not provide any way for the user to get to the context specific help for 
> that processor, controller service, or reporting task while within that 
> dialog.  The user has to close the dialog to access that help.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4215) Avro schemas with records that have a field of themselves fail to parse, causing stackoverflow exception

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113222#comment-16113222
 ] 

ASF GitHub Bot commented on NIFI-4215:
--

GitHub user Wesley-Lawrence opened a pull request:

https://github.com/apache/nifi/pull/2052

NIFI-4215 NiFi can now parse an Avro schema of a record that referenc…

…es an already defined record, including itself.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [✓] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [✓] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [✓] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [✓] Is your initial contribution a single, squashed commit?

### For code changes:
- [✓] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [✓] Have you written or updated unit tests to verify your changes?
- [N/A] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [N/A] If applicable, have you updated the LICENSE file, including the 
main LICENSE file under nifi-assembly?
- [N/A] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [N/A] If adding new Properties, have you added .displayName in addition 
to .name (programmatic access) for each of the new properties?

### For documentation related changes:
- [N/A] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Wesley-Lawrence/nifi NIFI-4215

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2052.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2052


commit 2423fd346031bbb66124863250080b356d029d90
Author: Wesley-Lawrence 
Date:   2017-08-03T17:48:31Z

NIFI-4215 NiFi can now parse an Avro schema of a record that references an 
already defined record, including itself.




> Avro schemas with records that have a field of themselves fail to parse, 
> causing stackoverflow exception
> 
>
> Key: NIFI-4215
> URL: https://issues.apache.org/jira/browse/NIFI-4215
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Wesley L Lawrence
>Priority: Minor
> Attachments: nifi-4215.patch
>
>
> Noticed this while attempting to use the AvroSchemaRegsitry with some complex 
> schema. Boiled down, Avro lets you define a schema such as;
> {code}
> { 
>   "namespace": "org.apache.nifi.testing", 
>   "name": "CompositRecord", 
>   "type": "record", 
>   "fields": [ 
> { 
>   "name": "id", 
>   "type": "int" 
> }, 
> { 
>   "name": "value", 
>   "type": "string" 
> }, 
> { 
>   "name": "parent", 
>   "type": [
> "null",
> "CompositRecord"
>   ]
> } 
>   ] 
> }
> {code}
> The AvroSchemaRegistry (AvroTypeUtil specifically) will fail to parse, and 
> generate a stackoverflow exception.
> I've whipped up a fix, tested it out in 1.4.0, and am just running through 
> the contrib build before I submit a patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2052: NIFI-4215 NiFi can now parse an Avro schema of a re...

2017-08-03 Thread Wesley-Lawrence

GitHub user Wesley-Lawrence opened a pull request:

https://github.com/apache/nifi/pull/2052

NIFI-4215 NiFi can now parse an Avro schema of a record that referencâ¦

â¦es an already defined record, including itself.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [â] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [â] Does your PR title start with NIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [â] Has your PR been rebased against the latest commit within the 
target branch (typically master)?

- [â] Is your initial contribution a single, squashed commit?

### For code changes:
- [â] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [â] Have you written or updated unit tests to verify your changes?
- [N/A] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [N/A] If applicable, have you updated the LICENSE file, including the 
main LICENSE file under nifi-assembly?
- [N/A] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [N/A] If adding new Properties, have you added .displayName in addition 
to .name (programmatic access) for each of the new properties?

### For documentation related changes:
- [N/A] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Wesley-Lawrence/nifi NIFI-4215

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2052.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2052


commit 2423fd346031bbb66124863250080b356d029d90
Author: Wesley-Lawrence 
Date:   2017-08-03T17:48:31Z

NIFI-4215 NiFi can now parse an Avro schema of a record that references an 
already defined record, including itself.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4215) Avro schemas with records that have a field of themselves fail to parse, causing stackoverflow exception

2017-08-03 Thread Wesley L Lawrence (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113158#comment-16113158
 ] 

Wesley L Lawrence commented on NIFI-4215:
-

[~markap14] No worries, it's always better to be safe then sorry.

I'll re-submit the patch, with the schema identifier issues fixed. Thanks again 
for taking a look at this!

> Avro schemas with records that have a field of themselves fail to parse, 
> causing stackoverflow exception
> 
>
> Key: NIFI-4215
> URL: https://issues.apache.org/jira/browse/NIFI-4215
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Wesley L Lawrence
>Priority: Minor
> Attachments: nifi-4215.patch
>
>
> Noticed this while attempting to use the AvroSchemaRegsitry with some complex 
> schema. Boiled down, Avro lets you define a schema such as;
> {code}
> { 
>   "namespace": "org.apache.nifi.testing", 
>   "name": "CompositRecord", 
>   "type": "record", 
>   "fields": [ 
> { 
>   "name": "id", 
>   "type": "int" 
> }, 
> { 
>   "name": "value", 
>   "type": "string" 
> }, 
> { 
>   "name": "parent", 
>   "type": [
> "null",
> "CompositRecord"
>   ]
> } 
>   ] 
> }
> {code}
> The AvroSchemaRegistry (AvroTypeUtil specifically) will fail to parse, and 
> generate a stackoverflow exception.
> I've whipped up a fix, tested it out in 1.4.0, and am just running through 
> the contrib build before I submit a patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4224) Add Variable Registry at Process Group level

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113154#comment-16113154
 ] 

ASF GitHub Bot commented on NIFI-4224:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2051

NIFI-4224: Initial implementation of Process Group level Variable Reg…

…istry

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4224

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2051.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2051


commit 3c43dedb2038a3c3e674c0be876c43010d672938
Author: Mark Payne 
Date:   2017-07-25T17:46:10Z

NIFI-4224: Initial implementation of Process Group level Variable Registry




> Add Variable Registry at Process Group level
> 
>
> Key: NIFI-4224
> URL: https://issues.apache.org/jira/browse/NIFI-4224
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>
> Currently, NiFi exposes a variable registry that is configurable by adding 
> the name of a properties file to nifi.properties and then treating the 
> referenced properties file as key/value pairs for the variable registry. 
> This, however, is very limiting, as it provides a global scope for all 
> variables, and it requires a restart of NiFi in order to pick up any updates 
> to the file. We should expose a Process Group-level Variable Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2051: NIFI-4224: Initial implementation of Process Group ...

2017-08-03 Thread markap14

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2051

NIFI-4224: Initial implementation of Process Group level Variable Regâ¦

â¦istry

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4224

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2051.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2051


commit 3c43dedb2038a3c3e674c0be876c43010d672938
Author: Mark Payne 
Date:   2017-07-25T17:46:10Z

NIFI-4224: Initial implementation of Process Group level Variable Registry




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4022) Use SASL Auth Scheme For Secured Zookeeper Client Interaction

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113136#comment-16113136
 ] 

ASF GitHub Bot commented on NIFI-4022:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/2046
  
Reviewing...


> Use SASL Auth Scheme For Secured Zookeeper Client Interaction
> -
>
> Key: NIFI-4022
> URL: https://issues.apache.org/jira/browse/NIFI-4022
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Yolanda M. Davis
>Assignee: Yolanda M. Davis
>
> NiFi uses Zookeeper to assist in cluster orchestration including leader 
> elections for Primary Node and Cluster Coordinator and to store state for 
> various processors (such as MonitorActivity). In secured Zookeeper 
> environments (supported by SASL + Kerberos) NiFi should protect the zNodes it 
> creates to prevent users or hosts, outside of a NiFi cluster, from accessing 
> or modifying entries.  In its current implementation security can be enforced 
> for processors that store state information in Zookeeper, however zNodes used 
> for managing Primary Node and Cluster Coordinator data are left open and 
> susceptible to change from any user.  Also when zNodes are secured for 
> processor state, a “Creator Only” policy is used which allows the system to 
> determine the identification of the NiFi node and protect any zNodes created 
> with that node id using Zookeeper’s “auth” scheme. The challenge with this 
> scheme is that it limits the ability for other NiFi nodes in the cluster to 
> access that zNode if needed (since it is specifically binds that zNode to the 
> unique id of its creator).
>  
> To best protect zNodes created in Zookeeper by NiFi while maximizing NiFi’s 
> ability to share information across the cluster I propose that we move to 
> using Zookeeper’s SASL authentication scheme, which will allow the use of 
> Kerberos principals for securing zNode with the appropriate permissions.  For 
> maximum flexibility, these principals can be mapped appropriately in 
> Zookeeper, using auth-to-local rules, to ensure that nodes across the cluster 
> can share zNodes as needed. 
>  
> Potential Concerns/Challenges for Discussion:
>  
> 1)  For existing NiFi users how will we migrate Zookeeper entries from 
> the old security scheme to the new scheme?
> 2)  How should zNodes be reverted to open if kerberos is disabled?
> 3)  What will the performance impact be on the cluster once SASL scheme 
> is enabled (since we’d be moving from open to protected)? Would require 
> investigation
> 4)  Currently users can control authentication scheme via state 
> management configuration for processors yet not for clusters.  Should we 
> still maintain the practice of allowing schemes to be configurable for 
> processors (with SASL being the new default)?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2046: NIFI-4022 - Enabled SASL auth scheme/ACL support for Curat...

2017-08-03 Thread bbende

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/2046
  
Reviewing...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4215) Avro schemas with records that have a field of themselves fail to parse, causing stackoverflow exception

2017-08-03 Thread Mark Payne (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113129#comment-16113129
 ] 

Mark Payne commented on NIFI-4215:
--

[~WesleyLawrence] [~jameswing] - hey guys sorry, just finished reviewing a bit 
deeper so that i understand exactly how this is working. I think when I read 
through it the first time, I didn't fully appreciate the approach that was 
being taken. In terms of the concern about making it mutable, I am just a fan 
of making objects immutable when we can because it simplifies things down the 
line and guarantees thread safety. That said, sometimes we can't have 
immutability. I think immutability is a great thing to strive for, but it's not 
always required. In this case, I agree that it's probably okay to make it 
mutable. Specifically, the fact that it is the SimpleRecordReader that 
introduces this method, not the interface, makes me feel better. Also, as it is 
being used currently, the fields are always being set before the object is ever 
provided to another thread, and I believe this means that we are okay in terms 
of thread safety (other threads won't get a stale value because the correct 
value will always be set before being handed over to another thread). So yes, 
I'm okay with this change to make the object mutable, as long as we fix the bug 
with the Schema Identifier. Sorry for the alarm there :)

> Avro schemas with records that have a field of themselves fail to parse, 
> causing stackoverflow exception
> 
>
> Key: NIFI-4215
> URL: https://issues.apache.org/jira/browse/NIFI-4215
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Wesley L Lawrence
>Priority: Minor
> Attachments: nifi-4215.patch
>
>
> Noticed this while attempting to use the AvroSchemaRegsitry with some complex 
> schema. Boiled down, Avro lets you define a schema such as;
> {code}
> { 
>   "namespace": "org.apache.nifi.testing", 
>   "name": "CompositRecord", 
>   "type": "record", 
>   "fields": [ 
> { 
>   "name": "id", 
>   "type": "int" 
> }, 
> { 
>   "name": "value", 
>   "type": "string" 
> }, 
> { 
>   "name": "parent", 
>   "type": [
> "null",
> "CompositRecord"
>   ]
> } 
>   ] 
> }
> {code}
> The AvroSchemaRegistry (AvroTypeUtil specifically) will fail to parse, and 
> generate a stackoverflow exception.
> I've whipped up a fix, tested it out in 1.4.0, and am just running through 
> the contrib build before I submit a patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4139) Refactor KeyProvider interface from provenance module to framework-level service

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113111#comment-16113111
 ] 

ASF GitHub Bot commented on NIFI-4139:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2044
  
Will review...


> Refactor KeyProvider interface from provenance module to framework-level 
> service
> 
>
> Key: NIFI-4139
> URL: https://issues.apache.org/jira/browse/NIFI-4139
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>  Labels: encryption, key-management, security
>
> The {{KeyProvider}} interface introduced in NIFI-3388 to allow the encrypted 
> provenance repository should be refactored to a framework-level service which 
> is accessible to the encrypted content repository and encrypted flowfile 
> repository as well. Exposing this common functionality will reduce code & 
> logic duplication and consolidate sensitive behavior in a single location. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2044: NIFI-4139 Extract key provider to framework-level service

2017-08-03 Thread mcgilman

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2044
  
Will review...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4024) Create EvaluateRecordPath processor

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113096#comment-16113096
 ] 

ASF GitHub Bot commented on NIFI-4024:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/1961
  
Thanks @MikeThomsen , two minor issues while testing this...

1) There is a try-with-resource block that creates the RecordReader, and as 
part of that line it also creates an InputStream for the flow file, but since 
the InputStream is created inline it won't be auto-closed which can cause a 
problem later on.

2) The complex field handling was only being used in the case where the 
field encoding strategy was "Bytes", which is different from how PutHBaseJson 
works.

I went ahead and addressed these issues in my own branch:

https://github.com/bbende/nifi/commit/a9d8d038fe40b50bab1a8b59e11ad926ed07cc78

If you are on-board with my changes then I'll go ahead and merge everything 
to master.

Let me know.


> Create EvaluateRecordPath processor
> ---
>
> Key: NIFI-4024
> URL: https://issues.apache.org/jira/browse/NIFI-4024
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Steve Champagne
>Priority: Minor
>
> With the new RecordPath DSL, it would be nice if there was a processor that 
> could pull fields into attributes of the flowfile based on a RecordPath. This 
> would be similar to the EvaluateJsonPath processor that currently exists, 
> except it could be used to pull fields from arbitrary record formats. My 
> current use case for it would be pulling fields out of Avro records while 
> skipping the steps of having to convert Avro to JSON, evaluate JsonPath, and 
> then converting back to Avro. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1961: NIFI-4024 Added org.apache.nifi.hbase.PutHBaseRecord

2017-08-03 Thread bbende

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/1961
  
Thanks @MikeThomsen , two minor issues while testing this...

1) There is a try-with-resource block that creates the RecordReader, and as 
part of that line it also creates an InputStream for the flow file, but since 
the InputStream is created inline it won't be auto-closed which can cause a 
problem later on.

2) The complex field handling was only being used in the case where the 
field encoding strategy was "Bytes", which is different from how PutHBaseJson 
works.

I went ahead and addressed these issues in my own branch:

https://github.com/bbende/nifi/commit/a9d8d038fe40b50bab1a8b59e11ad926ed07cc78

If you are on-board with my changes then I'll go ahead and merge everything 
to master.

Let me know.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (NIFI-4239) Implement a CaptureChangePostgreSQL processor

2017-08-03 Thread Gerdan Santos (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gerdan Santos reassigned NIFI-4239:
---

Assignee: Gerdan Santos

> Implement a CaptureChangePostgreSQL processor
> -
>
> Key: NIFI-4239
> URL: https://issues.apache.org/jira/browse/NIFI-4239
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Gerdan Santos
>Assignee: Gerdan Santos
>   Original Estimate: 2,688h
>  Remaining Estimate: 2,688h
>
> Inspired on CaptureChangeMySQL, this processor can use one Streaming 
> Replication PostgreSQL Connection to allow access to their transactional logs 
> and such, in order for external clients to have a "change data capture" (CDC) 
> capability.
> The processor would include properties needed for PostgreSQL connectivity 
> PostgreSQL Streaming Replication. It would also need to keep a "sequence ID" 
> such that an EnforceOrder processor (NIFI-3414) for example could guarantee 
> the order of CDC events for use cases such as replication. 
> It will likely need State Management for that, and may need other facilities 
> such as a DistributedMapCache in order to keep information (column names and 
> types, e.g.) that enrich the raw CDC events.
> The processor would accept no incoming connections (it is a "get" or source 
> processor), would be intended to run on the primary node only as a single 
> threaded processor, and would generate a flow file for each operation 
> (INSERT, UPDATE, DELETE, e.g.) in one or some number of formats (JSON, e.g.).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4261) User should be made aware if Controller Services are not working even if enabled

2017-08-03 Thread Andrew Lim (JIRA)

Andrew Lim created NIFI-4261:


 Summary: User should be made aware if Controller Services are not 
working even if enabled
 Key: NIFI-4261
 URL: https://issues.apache.org/jira/browse/NIFI-4261
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core UI
Reporter: Andrew Lim
Priority: Minor


A controller service can be enabled but not working.  For example, say your 
flow uses a ConvertRecord processor.  Say one or both of the record 
reader/writers used in the processor references the HortonworksSchemaRegistry 
controller service.  The Hortonworks Registry that the controller service 
points to could go down.  But in this situation the controller service itself 
is still enabled and there are no warnings given to the user.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-1580) Allow double-click to display config of processor

2017-08-03 Thread Pierre Villard (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-1580:
-
Fix Version/s: 1.4.0

Setting version to 1.4.0 to be sure this is fixed before the next release.

> Allow double-click to display config of processor
> -
>
> Key: NIFI-1580
> URL: https://issues.apache.org/jira/browse/NIFI-1580
> Project: Apache NiFi
>  Issue Type: Wish
>  Components: Core UI
>Affects Versions: 0.4.1
> Environment: all
>Reporter: Uwe Geercken
>Priority: Minor
>  Labels: features, processor, ui
> Fix For: 1.4.0
>
>
> A user frequently has to open the "config" dialog when designing nifi flows. 
> Each time the user has to right-click the processor and select "config" from 
> the menu.
> It would be quicker when it would be possible to double click a processor - 
> or maybe the title are - to display the config dialog.
> This could also be designed as a confuguration of the UI that the user can 
> define (if double-clicking open the config dialog, does something else or 
> simply nothing)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (NIFI-1580) Allow double-click to display config of processor

2017-08-03 Thread Pierre Villard (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard reopened NIFI-1580:
--

Re-opening.

With current master code, double click on bending points is opening connection 
configuration instead of deleting the bending point. Didn't find a way to 
delete bending point with the current master code. The double click to open the 
configuration should only apply on the connection box.

> Allow double-click to display config of processor
> -
>
> Key: NIFI-1580
> URL: https://issues.apache.org/jira/browse/NIFI-1580
> Project: Apache NiFi
>  Issue Type: Wish
>  Components: Core UI
>Affects Versions: 0.4.1
> Environment: all
>Reporter: Uwe Geercken
>Priority: Minor
>  Labels: features, processor, ui
> Fix For: 1.4.0
>
>
> A user frequently has to open the "config" dialog when designing nifi flows. 
> Each time the user has to right-click the processor and select "config" from 
> the menu.
> It would be quicker when it would be possible to double click a processor - 
> or maybe the title are - to display the config dialog.
> This could also be designed as a confuguration of the UI that the user can 
> define (if double-clicking open the config dialog, does something else or 
> simply nothing)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-3232) Extend UI menus to allow cascading (menu item > subitem)

2017-08-03 Thread Matt Gilman (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman reassigned NIFI-3232:
-

Assignee: Matt Gilman

> Extend UI menus to allow cascading (menu item > subitem)
> 
>
> Key: NIFI-3232
> URL: https://issues.apache.org/jira/browse/NIFI-3232
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Rob Moran
>Assignee: Matt Gilman
>
> With expanding functionality UI menu options are growing, making the size of 
> some quite large. Cascading menus will allow better information hierarchy to 
> improve the presentation of available user actions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3232) Extend UI menus to allow cascading (menu item > subitem)

2017-08-03 Thread Matt Gilman (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112794#comment-16112794
 ] 

Matt Gilman commented on NIFI-3232:
---

Resolved NIFI-4044 as duplicate and will use this JIRA. Didn't realize there 
were multiple.

> Extend UI menus to allow cascading (menu item > subitem)
> 
>
> Key: NIFI-3232
> URL: https://issues.apache.org/jira/browse/NIFI-3232
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Rob Moran
>
> With expanding functionality UI menu options are growing, making the size of 
> some quite large. Cascading menus will allow better information hierarchy to 
> improve the presentation of available user actions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFIREG-6) Add nifi-registry-bootstrap module

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFIREG-6?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112783#comment-16112783
 ] 

ASF GitHub Bot commented on NIFIREG-6:
--

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/3#discussion_r131141483
  
--- Diff: 
nifi-registry-runtime/src/main/java/org/apache/nifi/registry/NiFiRegistry.java 
---
@@ -18,22 +18,69 @@
 
 import org.apache.nifi.registry.jetty.JettyServer;
 import org.apache.nifi.registry.properties.NiFiRegistryProperties;
+import org.apache.nifi.registry.util.FileUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
+import org.slf4j.bridge.SLF4JBridgeHandler;
 
+import java.io.File;
 import java.io.FileReader;
+import java.io.FileWriter;
 import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
 
 /**
  * Main entry point for NiFiRegistry.
  */
 public class NiFiRegistry {
 
-private static final Logger logger = 
LoggerFactory.getLogger(JettyServer.class);
+private static final Logger LOGGER = 
LoggerFactory.getLogger(NiFiRegistry.class);
+private static final String KEY_FILE_FLAG = "-K";
 
-private static JettyServer server;
+public static final String BOOTSTRAP_PORT_PROPERTY = 
"nifi.registry.bootstrap.listen.port";
+
+private final JettyServer server;
+private final BootstrapListener bootstrapListener;
+private volatile boolean shutdown = false;
+
+public NiFiRegistry(final NiFiRegistryProperties properties)
+throws ClassNotFoundException, IOException, 
NoSuchMethodException, InstantiationException, IllegalAccessException, 
IllegalArgumentException, InvocationTargetException {
+
+// There can only be one krb5.conf for the overall Java process so 
set this globally during
+// start up so that processors and our Kerberos authentication 
code don't have to set this
+
+// TODO enable if we support Kerberos
--- End diff --

Should we remove this for now until we need to explicitly add support for 
it?


> Add nifi-registry-bootstrap module
> --
>
> Key: NIFIREG-6
> URL: https://issues.apache.org/jira/browse/NIFIREG-6
> Project: NiFi Registry
>  Issue Type: Improvement
>Affects Versions: 0.0.1
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
>
> Currently NiFiRegistry is the entry point of the application, and in its 
> current state the main method starts a new Jetty server every time.  
> We should take a similar approach to NiFi where we have a bootstrap process 
> that launches and communicates with the application process.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFIREG-6) Add nifi-registry-bootstrap module

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFIREG-6?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112784#comment-16112784
 ] 

ASF GitHub Bot commented on NIFIREG-6:
--

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/3#discussion_r131150677
  
--- Diff: 
nifi-registry-runtime/src/main/java/org/apache/nifi/registry/NiFiRegistry.java 
---
@@ -43,28 +90,242 @@ public void run() {
 }
 }));
 
-// load the properties
+final String bootstrapPort = 
System.getProperty(BOOTSTRAP_PORT_PROPERTY);
+if (bootstrapPort != null) {
+try {
+final int port = Integer.parseInt(bootstrapPort);
+
+if (port < 1 || port > 65535) {
+throw new RuntimeException("Failed to start NiFi 
Registry because system property '" + BOOTSTRAP_PORT_PROPERTY + "' is not a 
valid integer in the range 1 - 65535");
+}
+
+bootstrapListener = new BootstrapListener(this, port);
+bootstrapListener.start();
+} catch (final NumberFormatException nfe) {
+throw new RuntimeException("Failed to start NiFi Registry 
because system property '" + BOOTSTRAP_PORT_PROPERTY + "' is not a valid 
integer in the range 1 - 65535");
+}
+} else {
+LOGGER.info("NiFi Registry started without Bootstrap Port 
information provided; will not listen for requests from Bootstrap");
+bootstrapListener = null;
+}
+
+// delete the web working dir - if the application does not start 
successfully
+// the web app directories might be in an invalid state. when this 
happens
+// jetty will not attempt to re-extract the war into the 
directory. by removing
+// the working directory, we can be assured that it will attempt 
to extract the
+// war every time the application starts.
+File webWorkingDir = properties.getWebWorkingDirectory();
+FileUtils.deleteFilesInDirectory(webWorkingDir, null, LOGGER, 
true, true);
+FileUtils.deleteFile(webWorkingDir, LOGGER, 3);
+
+detectTimingIssues();
+
+// redirect JUL log events
+SLF4JBridgeHandler.removeHandlersForRootLogger();
+SLF4JBridgeHandler.install();
+
+final long startTime = System.nanoTime();
+server = new JettyServer(properties);
+
+if (shutdown) {
+LOGGER.info("NiFi Registry has been shutdown via NiFi Registry 
Bootstrap. Will not start Controller");
+} else {
+server.start();
+
+if (bootstrapListener != null) {
+bootstrapListener.sendStartedStatus(true);
+}
+
+final long duration = System.nanoTime() - startTime;
+LOGGER.info("Controller initialization took " + duration + " 
nanoseconds "
++ "(" + (int) TimeUnit.SECONDS.convert(duration, 
TimeUnit.NANOSECONDS) + " seconds).");
+}
+}
+
+protected void shutdownHook() {
+try {
+this.shutdown = true;
+
+LOGGER.info("Initiating shutdown of Jetty web server...");
+if (server != null) {
+server.stop();
+}
+if (bootstrapListener != null) {
+bootstrapListener.stop();
+}
+LOGGER.info("Jetty web server shutdown completed (nicely or 
otherwise).");
+} catch (final Throwable t) {
+LOGGER.warn("Problem occurred ensuring Jetty web server was 
properly terminated due to " + t);
+}
+}
+
+/**
+ * Determine if the machine we're running on has timing issues.
+ */
+private void detectTimingIssues() {
--- End diff --

I'm not sure we need to detect timing issues for the registry. Can probably 
remove this check.


> Add nifi-registry-bootstrap module
> --
>
> Key: NIFIREG-6
> URL: https://issues.apache.org/jira/browse/NIFIREG-6
> Project: NiFi Registry
>  Issue Type: Improvement
>Affects Versions: 0.0.1
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
>
> Currently NiFiRegistry is the entry point of the application, and in its 
> current state the main method starts a new Jetty server every time.  
> We should take a similar approach to NiFi where we have a bootstrap process 
> that launches and communicates with the application process.



--
This message was sent by Atlassian

[GitHub] nifi-registry pull request #3: NIFIREG-6 Adding nifi-registry-bootstrap modu...

2017-08-03 Thread mcgilman

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/3#discussion_r131141483
  
--- Diff: 
nifi-registry-runtime/src/main/java/org/apache/nifi/registry/NiFiRegistry.java 
---
@@ -18,22 +18,69 @@
 
 import org.apache.nifi.registry.jetty.JettyServer;
 import org.apache.nifi.registry.properties.NiFiRegistryProperties;
+import org.apache.nifi.registry.util.FileUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
+import org.slf4j.bridge.SLF4JBridgeHandler;
 
+import java.io.File;
 import java.io.FileReader;
+import java.io.FileWriter;
 import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
 
 /**
  * Main entry point for NiFiRegistry.
  */
 public class NiFiRegistry {
 
-private static final Logger logger = 
LoggerFactory.getLogger(JettyServer.class);
+private static final Logger LOGGER = 
LoggerFactory.getLogger(NiFiRegistry.class);
+private static final String KEY_FILE_FLAG = "-K";
 
-private static JettyServer server;
+public static final String BOOTSTRAP_PORT_PROPERTY = 
"nifi.registry.bootstrap.listen.port";
+
+private final JettyServer server;
+private final BootstrapListener bootstrapListener;
+private volatile boolean shutdown = false;
+
+public NiFiRegistry(final NiFiRegistryProperties properties)
+throws ClassNotFoundException, IOException, 
NoSuchMethodException, InstantiationException, IllegalAccessException, 
IllegalArgumentException, InvocationTargetException {
+
+// There can only be one krb5.conf for the overall Java process so 
set this globally during
+// start up so that processors and our Kerberos authentication 
code don't have to set this
+
+// TODO enable if we support Kerberos
--- End diff --

Should we remove this for now until we need to explicitly add support for 
it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-registry pull request #3: NIFIREG-6 Adding nifi-registry-bootstrap modu...

2017-08-03 Thread mcgilman

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/3#discussion_r131150677
  
--- Diff: 
nifi-registry-runtime/src/main/java/org/apache/nifi/registry/NiFiRegistry.java 
---
@@ -43,28 +90,242 @@ public void run() {
 }
 }));
 
-// load the properties
+final String bootstrapPort = 
System.getProperty(BOOTSTRAP_PORT_PROPERTY);
+if (bootstrapPort != null) {
+try {
+final int port = Integer.parseInt(bootstrapPort);
+
+if (port < 1 || port > 65535) {
+throw new RuntimeException("Failed to start NiFi 
Registry because system property '" + BOOTSTRAP_PORT_PROPERTY + "' is not a 
valid integer in the range 1 - 65535");
+}
+
+bootstrapListener = new BootstrapListener(this, port);
+bootstrapListener.start();
+} catch (final NumberFormatException nfe) {
+throw new RuntimeException("Failed to start NiFi Registry 
because system property '" + BOOTSTRAP_PORT_PROPERTY + "' is not a valid 
integer in the range 1 - 65535");
+}
+} else {
+LOGGER.info("NiFi Registry started without Bootstrap Port 
information provided; will not listen for requests from Bootstrap");
+bootstrapListener = null;
+}
+
+// delete the web working dir - if the application does not start 
successfully
+// the web app directories might be in an invalid state. when this 
happens
+// jetty will not attempt to re-extract the war into the 
directory. by removing
+// the working directory, we can be assured that it will attempt 
to extract the
+// war every time the application starts.
+File webWorkingDir = properties.getWebWorkingDirectory();
+FileUtils.deleteFilesInDirectory(webWorkingDir, null, LOGGER, 
true, true);
+FileUtils.deleteFile(webWorkingDir, LOGGER, 3);
+
+detectTimingIssues();
+
+// redirect JUL log events
+SLF4JBridgeHandler.removeHandlersForRootLogger();
+SLF4JBridgeHandler.install();
+
+final long startTime = System.nanoTime();
+server = new JettyServer(properties);
+
+if (shutdown) {
+LOGGER.info("NiFi Registry has been shutdown via NiFi Registry 
Bootstrap. Will not start Controller");
+} else {
+server.start();
+
+if (bootstrapListener != null) {
+bootstrapListener.sendStartedStatus(true);
+}
+
+final long duration = System.nanoTime() - startTime;
+LOGGER.info("Controller initialization took " + duration + " 
nanoseconds "
++ "(" + (int) TimeUnit.SECONDS.convert(duration, 
TimeUnit.NANOSECONDS) + " seconds).");
+}
+}
+
+protected void shutdownHook() {
+try {
+this.shutdown = true;
+
+LOGGER.info("Initiating shutdown of Jetty web server...");
+if (server != null) {
+server.stop();
+}
+if (bootstrapListener != null) {
+bootstrapListener.stop();
+}
+LOGGER.info("Jetty web server shutdown completed (nicely or 
otherwise).");
+} catch (final Throwable t) {
+LOGGER.warn("Problem occurred ensuring Jetty web server was 
properly terminated due to " + t);
+}
+}
+
+/**
+ * Determine if the machine we're running on has timing issues.
+ */
+private void detectTimingIssues() {
--- End diff --

I'm not sure we need to detect timing issues for the registry. Can probably 
remove this check.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFIREG-6) Add nifi-registry-bootstrap module

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFIREG-6?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112690#comment-16112690
 ] 

ASF GitHub Bot commented on NIFIREG-6:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi-registry/pull/3
  
Will review...


> Add nifi-registry-bootstrap module
> --
>
> Key: NIFIREG-6
> URL: https://issues.apache.org/jira/browse/NIFIREG-6
> Project: NiFi Registry
>  Issue Type: Improvement
>Affects Versions: 0.0.1
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
>
> Currently NiFiRegistry is the entry point of the application, and in its 
> current state the main method starts a new Jetty server every time.  
> We should take a similar approach to NiFi where we have a bootstrap process 
> that launches and communicates with the application process.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-registry issue #3: NIFIREG-6 Adding nifi-registry-bootstrap module

2017-08-03 Thread mcgilman

Github user mcgilman commented on the issue:

https://github.com/apache/nifi-registry/pull/3
  
Will review...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #2047: NIFI-4210: Add support for OpenId Connect

2017-08-03 Thread mcgilman

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2047#discussion_r131135419
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/oidc/StandardOidcIdentityProvider.java
 ---
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.web.security.oidc;
+
+import com.nimbusds.jose.JOSEException;
+import com.nimbusds.jose.JWSAlgorithm;
+import com.nimbusds.jose.proc.BadJOSEException;
+import com.nimbusds.jose.util.DefaultResourceRetriever;
+import com.nimbusds.jose.util.ResourceRetriever;
+import com.nimbusds.jwt.JWT;
+import com.nimbusds.jwt.JWTClaimsSet;
+import com.nimbusds.oauth2.sdk.AuthorizationGrant;
+import com.nimbusds.oauth2.sdk.ParseException;
+import com.nimbusds.oauth2.sdk.Scope;
+import com.nimbusds.oauth2.sdk.TokenErrorResponse;
+import com.nimbusds.oauth2.sdk.TokenRequest;
+import com.nimbusds.oauth2.sdk.TokenResponse;
+import com.nimbusds.oauth2.sdk.auth.ClientAuthentication;
+import com.nimbusds.oauth2.sdk.auth.ClientAuthenticationMethod;
+import com.nimbusds.oauth2.sdk.auth.ClientSecretBasic;
+import com.nimbusds.oauth2.sdk.auth.ClientSecretPost;
+import com.nimbusds.oauth2.sdk.auth.Secret;
+import com.nimbusds.oauth2.sdk.http.HTTPRequest;
+import com.nimbusds.oauth2.sdk.http.HTTPResponse;
+import com.nimbusds.oauth2.sdk.id.ClientID;
+import com.nimbusds.oauth2.sdk.token.BearerAccessToken;
+import com.nimbusds.openid.connect.sdk.OIDCScopeValue;
+import com.nimbusds.openid.connect.sdk.OIDCTokenResponse;
+import com.nimbusds.openid.connect.sdk.OIDCTokenResponseParser;
+import com.nimbusds.openid.connect.sdk.UserInfoErrorResponse;
+import com.nimbusds.openid.connect.sdk.UserInfoRequest;
+import com.nimbusds.openid.connect.sdk.UserInfoResponse;
+import com.nimbusds.openid.connect.sdk.UserInfoSuccessResponse;
+import com.nimbusds.openid.connect.sdk.claims.IDTokenClaimsSet;
+import com.nimbusds.openid.connect.sdk.op.OIDCProviderMetadata;
+import com.nimbusds.openid.connect.sdk.token.OIDCTokens;
+import com.nimbusds.openid.connect.sdk.validators.IDTokenValidator;
+import net.minidev.json.JSONObject;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.util.FormatUtils;
+import org.apache.nifi.util.NiFiProperties;
+import org.apache.nifi.web.security.jwt.JwtService;
+import org.apache.nifi.web.security.token.LoginAuthenticationToken;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URL;
+import java.util.Calendar;
+import java.util.Date;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import static 
com.nimbusds.openid.connect.sdk.claims.UserInfo.EMAIL_CLAIM_NAME;
+
+/**
+ * OidcProvider for managing the OpenId Connect Authorization flow.
+ */
+public class StandardOidcIdentityProvider implements OidcIdentityProvider {
+
+private static final Logger logger = 
LoggerFactory.getLogger(StandardOidcIdentityProvider.class);
+
+private NiFiProperties properties;
+private JwtService jwtService;
+private OIDCProviderMetadata oidcProviderMetadata;
+private int oidcConnectTimeout;
+private int oidcReadTimeout;
+private IDTokenValidator tokenValidator;
+private ClientID clientId;
+private Secret clientSecret;
+
+/**
+ * Creates a new StandardOidcIdentityProvider.
+ *
+ * @param jwtService jwt service
+ * @param properties properties
+ */
+public StandardOidcIdentityProvider(final JwtService jwtService, final 
NiFiProperties properties) {
+this.properties = properties;
+this.jwtService = jwtService;
+
+// attempt to process the oidc configuration if configured
+if

[jira] [Commented] (NIFI-4210) Add OpenId Connect support for authenticating users

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112682#comment-16112682
 ] 

ASF GitHub Bot commented on NIFI-4210:
--

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2047#discussion_r131135419
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/src/main/java/org/apache/nifi/web/security/oidc/StandardOidcIdentityProvider.java
 ---
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.web.security.oidc;
+
+import com.nimbusds.jose.JOSEException;
+import com.nimbusds.jose.JWSAlgorithm;
+import com.nimbusds.jose.proc.BadJOSEException;
+import com.nimbusds.jose.util.DefaultResourceRetriever;
+import com.nimbusds.jose.util.ResourceRetriever;
+import com.nimbusds.jwt.JWT;
+import com.nimbusds.jwt.JWTClaimsSet;
+import com.nimbusds.oauth2.sdk.AuthorizationGrant;
+import com.nimbusds.oauth2.sdk.ParseException;
+import com.nimbusds.oauth2.sdk.Scope;
+import com.nimbusds.oauth2.sdk.TokenErrorResponse;
+import com.nimbusds.oauth2.sdk.TokenRequest;
+import com.nimbusds.oauth2.sdk.TokenResponse;
+import com.nimbusds.oauth2.sdk.auth.ClientAuthentication;
+import com.nimbusds.oauth2.sdk.auth.ClientAuthenticationMethod;
+import com.nimbusds.oauth2.sdk.auth.ClientSecretBasic;
+import com.nimbusds.oauth2.sdk.auth.ClientSecretPost;
+import com.nimbusds.oauth2.sdk.auth.Secret;
+import com.nimbusds.oauth2.sdk.http.HTTPRequest;
+import com.nimbusds.oauth2.sdk.http.HTTPResponse;
+import com.nimbusds.oauth2.sdk.id.ClientID;
+import com.nimbusds.oauth2.sdk.token.BearerAccessToken;
+import com.nimbusds.openid.connect.sdk.OIDCScopeValue;
+import com.nimbusds.openid.connect.sdk.OIDCTokenResponse;
+import com.nimbusds.openid.connect.sdk.OIDCTokenResponseParser;
+import com.nimbusds.openid.connect.sdk.UserInfoErrorResponse;
+import com.nimbusds.openid.connect.sdk.UserInfoRequest;
+import com.nimbusds.openid.connect.sdk.UserInfoResponse;
+import com.nimbusds.openid.connect.sdk.UserInfoSuccessResponse;
+import com.nimbusds.openid.connect.sdk.claims.IDTokenClaimsSet;
+import com.nimbusds.openid.connect.sdk.op.OIDCProviderMetadata;
+import com.nimbusds.openid.connect.sdk.token.OIDCTokens;
+import com.nimbusds.openid.connect.sdk.validators.IDTokenValidator;
+import net.minidev.json.JSONObject;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.util.FormatUtils;
+import org.apache.nifi.util.NiFiProperties;
+import org.apache.nifi.web.security.jwt.JwtService;
+import org.apache.nifi.web.security.token.LoginAuthenticationToken;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URL;
+import java.util.Calendar;
+import java.util.Date;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import static 
com.nimbusds.openid.connect.sdk.claims.UserInfo.EMAIL_CLAIM_NAME;
+
+/**
+ * OidcProvider for managing the OpenId Connect Authorization flow.
+ */
+public class StandardOidcIdentityProvider implements OidcIdentityProvider {
+
+private static final Logger logger = 
LoggerFactory.getLogger(StandardOidcIdentityProvider.class);
+
+private NiFiProperties properties;
+private JwtService jwtService;
+private OIDCProviderMetadata oidcProviderMetadata;
+private int oidcConnectTimeout;
+private int oidcReadTimeout;
+private IDTokenValidator tokenValidator;
+private ClientID clientId;
+private Secret clientSecret;
+
+/**
+ * Creates a new StandardOidcIdentityProvider.
+ *
+ * @param jwtService jwt service
+ * @param properties properties
+ */
+public StandardOidcIdentityProvider(final JwtService jwtService,

[jira] [Created] (NIFI-4260) ExecuteFlumeSink not working with Sink Type "avro".

2017-08-03 Thread Sebastian Schlitte (JIRA)

Sebastian Schlitte created NIFI-4260:


 Summary: ExecuteFlumeSink not working with Sink Type "avro".
 Key: NIFI-4260
 URL: https://issues.apache.org/jira/browse/NIFI-4260
 Project: Apache NiFi
  Issue Type: Bug
  Components: Extensions
Affects Versions: 1.3.0, 1.2.0
Reporter: Sebastian Schlitte
Priority: Minor


After upgrading NiFi from 1.1.2 to 1.2.0, ExecuteFlumeSink with sink type 
"avro" stops transmitting data and throws the error quoted below. The problem 
persists after upgrading to 1.3.0.
Replacing avro-ipc-1.7.4.jar with avro-ipc-1.8.1.jar in 
nifi-flume-nar-1.3.0.nar solves the issue.


{{2017-08-02 10:28:51,142 ERROR [Timer-Driven Process Thread-6] 
o.a.n.processors.flume.ExecuteFlumeSink 
ExecuteFlumeSink[id=98da368f-015d-1000--9
6768472] ExecuteFlumeSink[id=98da368f-015d-1000--96768472] failed to 
process due to org.apache.nifi.processor.exception.ProcessException: Flume
 event delivery failed; rolling back session: {}
org.apache.nifi.processor.exception.ProcessException: Flume event delivery 
failed
at 
org.apache.nifi.processors.flume.ExecuteFlumeSink.onTrigger(ExecuteFlumeSink.java:145)
at 
org.apache.nifi.processors.flume.AbstractFlumeProcessor.onTrigger(AbstractFlumeProcessor.java:148)
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:144)
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flume.EventDeliveryException: Failed to send events
at 
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
at 
org.apache.nifi.processors.flume.ExecuteFlumeSink.onTrigger(ExecuteFlumeSink.java:143)
... 12 common frames omitted
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: 
target.linux.dummy, port: 14100 }: Failed to send batch
at 
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:315)
at 
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:376)
... 13 common frames omitted
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: 
target.linux.dummy, port: 14100 }: RPC request exception
at 
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:365)
at 
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:303)
... 14 common frames omitted
Caused by: java.util.concurrent.ExecutionException: 
java.lang.AbstractMethodError: 
org.apache.avro.specific.SpecificFixed.getSchema()Lorg/apache/avro/S
chema;
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at 
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:357)
... 15 common frames omitted
Caused by: java.lang.AbstractMethodError: 
org.apache.avro.specific.SpecificFixed.getSchema()Lorg/apache/avro/Schema;
at org.apache.avro.specific.SpecificFixed.(SpecificFixed.java:36)
at org.apache.avro.ipc.MD5.(MD5.java:16)
at org.apache.avro.ipc.Requestor.writeHandshake(Requestor.java:200)
at org.apache.avro.ipc.Requestor.access$300(Requestor.java:52)
at org.apache.avro.ipc.Requestor$Request.getBytes(Requestor.java:478)
at org.apache.avro.ipc.Requestor.request(Requestor.java:147)
at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
at 
org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:84)
at com.sun.proxy.$Proxy124.appendBatch(Unknown Source)
at 
org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:348)
at 
org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:344)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted}}



--
This message was

[jira] [Commented] (NIFI-3484) GenerateTableFetch Should Allow for Right Boundary

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112452#comment-16112452
 ] 

ASF GitHub Bot commented on NIFI-3484:
--

Github user patricker commented on the issue:

https://github.com/apache/nifi/pull/1513
  
@pvillard31 Thanks. There have been some more changes recently to 
GenerateTableFetch. Let me work through the rebase.


> GenerateTableFetch Should Allow for Right Boundary
> --
>
> Key: NIFI-3484
> URL: https://issues.apache.org/jira/browse/NIFI-3484
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Peter Wicks
>Assignee: Peter Wicks
>Priority: Minor
>
> When using GenerateTableFetch it places no right hand boundary on pages of 
> data.  This can lead to issues when the statement says to get the next 1000 
> records greater then a specific key, but records were added to the table 
> between the time the processor executed and when the SQL is being executed. 
> As a result it pulls in records that did not exist when the processor was 
> run.  On the next execution of the processor these records will be pulled in 
> a second time.
> Example:
> Partition Size = 1000
> First run (no state): Count(*)=4700 and MAX(ID)=4700.
> 5 FlowFiles are generated, the last one will say to fetch 1000, not 700. (But 
> I don't think this is really a bug, just an observation).
> 5 Flow Files are now in queue to be executed by ExecuteSQL.  Before the 5th 
> file can execute 400 new rows are added to the table.  When the final SQL 
> statement is executed 300 extra records, with higher ID values, will also be 
> pulled into NiFi.
> Second run (state: ID=4700).  Count(*) ID>4700 = 400 and MAX(ID)=5100.
> 1 Flow File is generated, but includes 300 records already pulled into NiFI.
> The solution is to have an optional property that will let users use the new 
> MAX(ID) as a right boundary when generating queries.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #1513: NIFI-3484 GenerateTableFetch Should Allow for Right Bounda...

2017-08-03 Thread patricker

Github user patricker commented on the issue:

https://github.com/apache/nifi/pull/1513
  
@pvillard31 Thanks. There have been some more changes recently to 
GenerateTableFetch. Let me work through the rebase.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-4259) Run-init.bat show permission error as org.apache.nifi.bootstrap.Command Failed to set permissions so that only the owner can read pid file

2017-08-03 Thread Pierre Villard (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-4259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4259:
-
Affects Version/s: (was: 2.0.0)
   1.4.0

> Run-init.bat show permission error as org.apache.nifi.bootstrap.Command 
> Failed to set permissions so that only the owner can read pid file
> --
>
> Key: NIFI-4259
> URL: https://issues.apache.org/jira/browse/NIFI-4259
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Configuration
>Affects Versions: 1.4.0
>Reporter: chatsiri
>
> Hello Contributor,
> I  build Apache-nifi on Windows OS enviorment by use maven simple command 
> references from official build steps. After built core jar file success in  
> nifi-assembly folder. I run command line as below. That warning did I cannot 
> loaded web URL : http://localhost:8080/nifi.
>  
> {panel:title=Console log after cal call command line ./run-nifi.bat}
> PS C:\workspacejava\nifi\nifi-assembly\target\nifi-1.4.0-SNAPSHOT\bin> 
> .\run-nifi.bat
> 2017-08-03 10:18:03,086 INFO [main] org.apache.nifi.bootstrap.Command 
> Starting Apache NiFi...
> 2017-08-03 10:18:03,088 INFO [main] org.apache.nifi.bootstrap.Command Working 
> Directory: C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-S
> 2017-08-03 10:18:03,089 INFO [main] org.apache.nifi.bootstrap.Command 
> Command: C:\Program Files\Java\jdk1.8.0_131\bin\java.exe -classpath 
> C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-S\.\conf;C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-
> S\.\lib\javax.servlet-api-3.1.0.jar;C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-S\.\lib\jcl-over-slf4j-1.7.25.jar;C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-S\.\lib\jetty-schemas-3.1.jar;C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\tar
> get\NIFI-1~2.0-S\.\lib\jul-to-slf4j-1.7.25.jar;C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-S\.\lib\log4j-over-slf4j-1.7.25.jar;C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-S\.\lib\logback-classic-1.2.3.jar;C:\Users\R2C06~1.CHA\WOA236~1\
> nifi\NIFI-A~1\target\NIFI-1~2.0-S\.\lib\logback-core-1.2.3.jar;C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-S\.\lib\nifi-api-1.4.0-SNAPSHOT.jar;C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-S\.\lib\nifi-framework-api-1.4.0-SNAPSHOT.jar;C:
> \Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-S\.\lib\nifi-nar-utils-1.4.0-SNAPSHOT.jar;C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-S\.\lib\nifi-properties-1.4.0-SNAPSHOT.jar;C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-S\
> .\lib\nifi-runtime-1.4.0-SNAPSHOT.jar;C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-S\.\lib\slf4j-api-1.7.25.jar
>  -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m 
> -Djava.security.egd=file:/dev/urandom -Dsun.net.http.allowRestrictedHeaders=
> true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -XX:+UseG1GC 
> -Djava.protocol.handler.pkgs=sun.net.www.protocol 
> -Dnifi.properties.file.path=C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-S\.\conf\nifi.properties
>  -Dnifi.bootstrap.listen.port=53
> 626 -Dapp=NiFi 
> -Dorg.apache.nifi.bootstrap.config.log.dir=C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-S\bin\..\\logs
>  org.apache.nifi.NiFi
> 2017-08-03 10:18:03,975 WARN [main] org.apache.nifi.bootstrap.Command Failed 
> to set permissions so that only the owner can read pid file 
> C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-S\bin\..\run\nifi.pid;
>  this may allows others to have access to the key
> needed to communicate with NiFi. Permissions should be changed so that only 
> the owner can read this file
> 2017-08-03 10:18:04,000 WARN [main] org.apache.nifi.bootstrap.Command Failed 
> to set permissions so that only the owner can read status file 
> C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-S\bin\..\run\nifi.status;
>  this may allows others to have access to th
> e key needed to communicate with NiFi. Permissions should be changed so that 
> only the owner can read this file
> 2017-08-03 10:18:04,051 INFO [main] org.apache.nifi.bootstrap.Command 
> Launched Apache NiFi with Process ID 18036
> {panel} 
> It shows warning from console log is  
> {panel:title=Console log}
> 2017-08-03 10:18:04,000 WARN [main] org.apache.nifi.bootstrap.Command Failed 
> to set permissions so that only the owner can read status file 
> C:\Users\R2C06~1.CHA\WOA236~1\nifi\NIFI-A~1\target\NIFI-1~2.0-S\bin\..\run\nifi.status;
>  this may allows others to have access to th
> e key needed to communicate with NiFi. Permissions should be changed so that

[jira] [Commented] (NIFI-4253) Add support for Oracle 12+

2017-08-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112369#comment-16112369
 ] 

ASF GitHub Bot commented on NIFI-4253:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2048


> Add support for Oracle 12+
> --
>
> Key: NIFI-4253
> URL: https://issues.apache.org/jira/browse/NIFI-4253
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.4.0
>
>
> The current Oracle database adapter in NiFi is compatible with versions of 
> Oracle up to 11, but does not make use of constructs available in Oracle 12 
> (such as OFFSET/FETCH). Adding an adapter that can make use of these 
> improvements will provide better support for such database versions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4253) Add support for Oracle 12+

2017-08-03 Thread Pierre Villard (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4253:
-
   Resolution: Fixed
Fix Version/s: 1.4.0
   Status: Resolved  (was: Patch Available)

> Add support for Oracle 12+
> --
>
> Key: NIFI-4253
> URL: https://issues.apache.org/jira/browse/NIFI-4253
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.4.0
>
>
> The current Oracle database adapter in NiFi is compatible with versions of 
> Oracle up to 11, but does not make use of constructs available in Oracle 12 
> (such as OFFSET/FETCH). Adding an adapter that can make use of these 
> improvements will provide better support for such database versions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4253) Add support for Oracle 12+

2017-08-03 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112367#comment-16112367
 ] 

ASF subversion and git services commented on NIFI-4253:
---

Commit 2d335821471ec10f48e3e586f66f3f14c2e0dd96 in nifi's branch 
refs/heads/master from [~mattyb149]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=2d33582 ]

NIFI-4253: Add Oracle 12+ database adapter

Signed-off-by: Pierre Villard 

This closes #2048.


> Add support for Oracle 12+
> --
>
> Key: NIFI-4253
> URL: https://issues.apache.org/jira/browse/NIFI-4253
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.4.0
>
>
> The current Oracle database adapter in NiFi is compatible with versions of 
> Oracle up to 11, but does not make use of constructs available in Oracle 12 
> (such as OFFSET/FETCH). Adding an adapter that can make use of these 
> improvements will provide better support for such database versions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2048: NIFI-4253: Add Oracle 12+ database adapter

2017-08-03 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2048


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

72 matches

Mail list logo