[jira] [Updated] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-4023:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:36,420 ERROR [Index Provenance Events-1] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:37,425 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> 

[jira] [Updated] (NIFI-4077) o.a.n.p.index.lucene.LuceneEventIndex Failed to retrieve Provenance Events from store

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-4077:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> o.a.n.p.index.lucene.LuceneEventIndex Failed to retrieve Provenance Events 
> from store
> -
>
> Key: NIFI-4077
> URL: https://issues.apache.org/jira/browse/NIFI-4077
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Joseph Gresock
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> I get this error when trying to display Data Provenance on a 4-node cluster:
> 2017-06-15 18:28:23,945 ERROR [Provenance Query-2] 
> o.a.n.p.index.lucene.LuceneEventIndex Failed to retrieve Provenance Events 
> from store
> java.io.FileNotFoundException: Unable to locate file 
> /data/nifi/provenance_repository/85309321.prov
> at 
> org.apache.nifi.provenance.serialization.RecordReaders.newRecordReader(RecordReaders.java:119)
> at 
> org.apache.nifi.provenance.WriteAheadProvenanceRepository.lambda$initialize$46(WriteAheadProvenanceRepository.java:125)
> at 
> org.apache.nifi.provenance.store.iterator.SelectiveRecordReaderEventIterator.nextEvent(SelectiveRecordReaderEventIterator.java:137)
> at 
> org.apache.nifi.provenance.store.iterator.AuthorizingEventIterator.nextEvent(AuthorizingEventIterator.java:47)
> at 
> org.apache.nifi.provenance.store.PartitionedEventStore.getEvents(PartitionedEventStore.java:192)
> at 
> org.apache.nifi.provenance.store.PartitionedEventStore.getEvents(PartitionedEventStore.java:167)
> at 
> org.apache.nifi.provenance.index.lucene.LuceneEventIndex.lambda$submitQuery$58(LuceneEventIndex.java:442)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> I don't know if this is related, but I also see this error in the same logs:
> 2017-06-15 18:28:42,923 ERROR [Compress Provenance Logs-1-thread-1] 
> o.a.n.p.s.EventFileCompressor Failed to read TOC File 
> /data/nifi/provenance_repository/toc/87247412.toc
> java.io.EOFException: null
> at 
> org.apache.nifi.provenance.toc.StandardTocReader.(StandardTocReader.java:48)
> at 
> org.apache.nifi.provenance.serialization.EventFileCompressor.run(EventFileCompressor.java:93)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4142) Implement a ValidateRecord Processor

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-4142:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Implement a ValidateRecord Processor
> 
>
> Key: NIFI-4142
> URL: https://issues.apache.org/jira/browse/NIFI-4142
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> We need a processor that is capable of validating that all Records in a 
> FlowFile adhere to the proper schema.
> The Processor should be configured with a Record Reader and should route each 
> record to either 'valid' or 'invalid' based on whether or not the record 
> adheres to the reader's schema. A record would be invalid in any of the 
> following cases:
> - Missing field that is required according to the schema
> - Extra field that is not present in schema (it should be configurable 
> whether or not this is a failure)
> - Field requires coercion and strict type checking enabled (this should also 
> be configurable)
> - Field is invalid, such as the value "hello" when it should be an integer



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124463#comment-16124463
 ] 

ASF subversion and git services commented on NIFI-4023:
---

Commit 84935d4f7840fe9b22aa929afb403718c073e627 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=84935d4 ]

NIFI-4023, NIFI-4077 This closes #2075. Addressed issue where repository was 
aging off the wrong index. When it should age off Index 1, it was removing 
Index 2. As a result, the earliest index is never aged off, and the newest 
index could potentially be aged off before it is ready to be. Also addressed 
issue where a query that attempts to read an event that has aged off throws 
FileNotFoundException (NIFI-4077) instead of skipping over the event. The 
JavaDocs indicate that the EventIterator should skip records that it cannot 
find, but SelectiveRecordReaderEventIterator throw FileNotFoundException instead

Signed-off-by: joewitt 


> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:36,420 ERROR [Index Provenance Events-1] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:37,425 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> 

[jira] [Commented] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124468#comment-16124468
 ] 

ASF GitHub Bot commented on NIFI-4023:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2075


> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:36,420 ERROR [Index Provenance Events-1] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:37,425 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at 

[jira] [Commented] (NIFI-4142) Implement a ValidateRecord Processor

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124466#comment-16124466
 ] 

ASF subversion and git services commented on NIFI-4142:
---

Commit 451f9cf12407c4f7abeb8e538e3f55ecbf40abab in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=451f9cf ]

NIFI-4142: This closes #2015. Refactored Record Reader/Writer to allow for 
reading/writing "raw records". Implemented ValidateRecord. Updated Record 
Reader to take two parameters for nextRecord: (boolean coerceTypes) and 
(boolean dropUnknownFields)

Signed-off-by: joewitt 


> Implement a ValidateRecord Processor
> 
>
> Key: NIFI-4142
> URL: https://issues.apache.org/jira/browse/NIFI-4142
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> We need a processor that is capable of validating that all Records in a 
> FlowFile adhere to the proper schema.
> The Processor should be configured with a Record Reader and should route each 
> record to either 'valid' or 'invalid' based on whether or not the record 
> adheres to the reader's schema. A record would be invalid in any of the 
> following cases:
> - Missing field that is required according to the schema
> - Extra field that is not present in schema (it should be configurable 
> whether or not this is a failure)
> - Field requires coercion and strict type checking enabled (this should also 
> be configurable)
> - Field is invalid, such as the value "hello" when it should be an integer



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4142) Implement a ValidateRecord Processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124467#comment-16124467
 ] 

ASF GitHub Bot commented on NIFI-4142:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2015


> Implement a ValidateRecord Processor
> 
>
> Key: NIFI-4142
> URL: https://issues.apache.org/jira/browse/NIFI-4142
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> We need a processor that is capable of validating that all Records in a 
> FlowFile adhere to the proper schema.
> The Processor should be configured with a Record Reader and should route each 
> record to either 'valid' or 'invalid' based on whether or not the record 
> adheres to the reader's schema. A record would be invalid in any of the 
> following cases:
> - Missing field that is required according to the schema
> - Extra field that is not present in schema (it should be configurable 
> whether or not this is a failure)
> - Field requires coercion and strict type checking enabled (this should also 
> be configurable)
> - Field is invalid, such as the value "hello" when it should be an integer



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4077) o.a.n.p.index.lucene.LuceneEventIndex Failed to retrieve Provenance Events from store

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124465#comment-16124465
 ] 

ASF subversion and git services commented on NIFI-4077:
---

Commit 84935d4f7840fe9b22aa929afb403718c073e627 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=84935d4 ]

NIFI-4023, NIFI-4077 This closes #2075. Addressed issue where repository was 
aging off the wrong index. When it should age off Index 1, it was removing 
Index 2. As a result, the earliest index is never aged off, and the newest 
index could potentially be aged off before it is ready to be. Also addressed 
issue where a query that attempts to read an event that has aged off throws 
FileNotFoundException (NIFI-4077) instead of skipping over the event. The 
JavaDocs indicate that the EventIterator should skip records that it cannot 
find, but SelectiveRecordReaderEventIterator throw FileNotFoundException instead

Signed-off-by: joewitt 


> o.a.n.p.index.lucene.LuceneEventIndex Failed to retrieve Provenance Events 
> from store
> -
>
> Key: NIFI-4077
> URL: https://issues.apache.org/jira/browse/NIFI-4077
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Joseph Gresock
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> I get this error when trying to display Data Provenance on a 4-node cluster:
> 2017-06-15 18:28:23,945 ERROR [Provenance Query-2] 
> o.a.n.p.index.lucene.LuceneEventIndex Failed to retrieve Provenance Events 
> from store
> java.io.FileNotFoundException: Unable to locate file 
> /data/nifi/provenance_repository/85309321.prov
> at 
> org.apache.nifi.provenance.serialization.RecordReaders.newRecordReader(RecordReaders.java:119)
> at 
> org.apache.nifi.provenance.WriteAheadProvenanceRepository.lambda$initialize$46(WriteAheadProvenanceRepository.java:125)
> at 
> org.apache.nifi.provenance.store.iterator.SelectiveRecordReaderEventIterator.nextEvent(SelectiveRecordReaderEventIterator.java:137)
> at 
> org.apache.nifi.provenance.store.iterator.AuthorizingEventIterator.nextEvent(AuthorizingEventIterator.java:47)
> at 
> org.apache.nifi.provenance.store.PartitionedEventStore.getEvents(PartitionedEventStore.java:192)
> at 
> org.apache.nifi.provenance.store.PartitionedEventStore.getEvents(PartitionedEventStore.java:167)
> at 
> org.apache.nifi.provenance.index.lucene.LuceneEventIndex.lambda$submitQuery$58(LuceneEventIndex.java:442)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> I don't know if this is related, but I also see this error in the same logs:
> 2017-06-15 18:28:42,923 ERROR [Compress Provenance Logs-1-thread-1] 
> o.a.n.p.s.EventFileCompressor Failed to read TOC File 
> /data/nifi/provenance_repository/toc/87247412.toc
> java.io.EOFException: null
> at 
> org.apache.nifi.provenance.toc.StandardTocReader.(StandardTocReader.java:48)
> at 
> org.apache.nifi.provenance.serialization.EventFileCompressor.run(EventFileCompressor.java:93)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4077) o.a.n.p.index.lucene.LuceneEventIndex Failed to retrieve Provenance Events from store

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124464#comment-16124464
 ] 

ASF subversion and git services commented on NIFI-4077:
---

Commit 84935d4f7840fe9b22aa929afb403718c073e627 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=84935d4 ]

NIFI-4023, NIFI-4077 This closes #2075. Addressed issue where repository was 
aging off the wrong index. When it should age off Index 1, it was removing 
Index 2. As a result, the earliest index is never aged off, and the newest 
index could potentially be aged off before it is ready to be. Also addressed 
issue where a query that attempts to read an event that has aged off throws 
FileNotFoundException (NIFI-4077) instead of skipping over the event. The 
JavaDocs indicate that the EventIterator should skip records that it cannot 
find, but SelectiveRecordReaderEventIterator throw FileNotFoundException instead

Signed-off-by: joewitt 


> o.a.n.p.index.lucene.LuceneEventIndex Failed to retrieve Provenance Events 
> from store
> -
>
> Key: NIFI-4077
> URL: https://issues.apache.org/jira/browse/NIFI-4077
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Joseph Gresock
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> I get this error when trying to display Data Provenance on a 4-node cluster:
> 2017-06-15 18:28:23,945 ERROR [Provenance Query-2] 
> o.a.n.p.index.lucene.LuceneEventIndex Failed to retrieve Provenance Events 
> from store
> java.io.FileNotFoundException: Unable to locate file 
> /data/nifi/provenance_repository/85309321.prov
> at 
> org.apache.nifi.provenance.serialization.RecordReaders.newRecordReader(RecordReaders.java:119)
> at 
> org.apache.nifi.provenance.WriteAheadProvenanceRepository.lambda$initialize$46(WriteAheadProvenanceRepository.java:125)
> at 
> org.apache.nifi.provenance.store.iterator.SelectiveRecordReaderEventIterator.nextEvent(SelectiveRecordReaderEventIterator.java:137)
> at 
> org.apache.nifi.provenance.store.iterator.AuthorizingEventIterator.nextEvent(AuthorizingEventIterator.java:47)
> at 
> org.apache.nifi.provenance.store.PartitionedEventStore.getEvents(PartitionedEventStore.java:192)
> at 
> org.apache.nifi.provenance.store.PartitionedEventStore.getEvents(PartitionedEventStore.java:167)
> at 
> org.apache.nifi.provenance.index.lucene.LuceneEventIndex.lambda$submitQuery$58(LuceneEventIndex.java:442)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> I don't know if this is related, but I also see this error in the same logs:
> 2017-06-15 18:28:42,923 ERROR [Compress Provenance Logs-1-thread-1] 
> o.a.n.p.s.EventFileCompressor Failed to read TOC File 
> /data/nifi/provenance_repository/toc/87247412.toc
> java.io.EOFException: null
> at 
> org.apache.nifi.provenance.toc.StandardTocReader.(StandardTocReader.java:48)
> at 
> org.apache.nifi.provenance.serialization.EventFileCompressor.run(EventFileCompressor.java:93)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2075: NIFI-4023, NIFI-4077: Addressed issue where reposit...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2075


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #2015: NIFI-4142: Refactored Record Reader/Writer to allow...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2015


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4142) Implement a ValidateRecord Processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124461#comment-16124461
 ] 

ASF GitHub Bot commented on NIFI-4142:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2015
  
went ahead and fixed the typo and squashed locally.

Ran a test flow that created csv of various conformity to a simple spec.  
Ran it through various configratuions of the validator and it behaved as 
desired and expected in each case.  When data was marked invalid was able to 
see precisely why through the provenance events which was great.  Was able to 
see coercion/non-coercion behavior and all checks out.

+1 merging to master


> Implement a ValidateRecord Processor
> 
>
> Key: NIFI-4142
> URL: https://issues.apache.org/jira/browse/NIFI-4142
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> We need a processor that is capable of validating that all Records in a 
> FlowFile adhere to the proper schema.
> The Processor should be configured with a Record Reader and should route each 
> record to either 'valid' or 'invalid' based on whether or not the record 
> adheres to the reader's schema. A record would be invalid in any of the 
> following cases:
> - Missing field that is required according to the schema
> - Extra field that is not present in schema (it should be configurable 
> whether or not this is a failure)
> - Field requires coercion and strict type checking enabled (this should also 
> be configurable)
> - Field is invalid, such as the value "hello" when it should be an integer



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2015: NIFI-4142: Refactored Record Reader/Writer to allow for re...

[joewitt]

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2015
  
went ahead and fixed the typo and squashed locally.

Ran a test flow that created csv of various conformity to a simple spec.  
Ran it through various configratuions of the validator and it behaved as 
desired and expected in each case.  When data was marked invalid was able to 
see precisely why through the provenance events which was great.  Was able to 
see coercion/non-coercion behavior and all checks out.

+1 merging to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124459#comment-16124459
 ] 

ASF GitHub Bot commented on NIFI-4023:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2075
  
+1 will merge to master


> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:36,420 ERROR [Index Provenance Events-1] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:37,425 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at 

[GitHub] nifi issue #2075: NIFI-4023, NIFI-4077: Addressed issue where repository was...

[joewitt]

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2075
  
+1 will merge to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4288) Intermittent test failures in TestPutUDP

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124456#comment-16124456
 ] 

Joseph Witt commented on NIFI-4288:
---

thanks andy.  I'll mark it as closed.  I did also verify it locally too.

> Intermittent test failures in TestPutUDP
> 
>
> Key: NIFI-4288
> URL: https://issues.apache.org/jira/browse/NIFI-4288
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.3.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
> Fix For: 1.4.0
>
>
> In some environments TestPutUDP will produce test failures for the following:
> TestPutUDP.testInvalidIPAddress:153->checkRelationships:227 expected:<0> but 
> was:<5>
> We can @Ignore this test for an immediate fix, or could refactor the test to 
> mock the ChannelSender so that we don't actually communicate with a live 
> server and can instead mock exceptions being thrown.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-4288) Intermittent test failures in TestPutUDP

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt resolved NIFI-4288.
---
Resolution: Fixed

> Intermittent test failures in TestPutUDP
> 
>
> Key: NIFI-4288
> URL: https://issues.apache.org/jira/browse/NIFI-4288
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.3.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
>
> In some environments TestPutUDP will produce test failures for the following:
> TestPutUDP.testInvalidIPAddress:153->checkRelationships:227 expected:<0> but 
> was:<5>
> We can @Ignore this test for an immediate fix, or could refactor the test to 
> mock the ChannelSender so that we don't actually communicate with a live 
> server and can instead mock exceptions being thrown.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4288) Intermittent test failures in TestPutUDP

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-4288:
--
Fix Version/s: 1.4.0

> Intermittent test failures in TestPutUDP
> 
>
> Key: NIFI-4288
> URL: https://issues.apache.org/jira/browse/NIFI-4288
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.3.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
> Fix For: 1.4.0
>
>
> In some environments TestPutUDP will produce test failures for the following:
> TestPutUDP.testInvalidIPAddress:153->checkRelationships:227 expected:<0> but 
> was:<5>
> We can @Ignore this test for an immediate fix, or could refactor the test to 
> mock the ChannelSender so that we don't actually communicate with a live 
> server and can instead mock exceptions being thrown.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2056: NIFI-4262 - MergeContent - option to add merged uuid in or...

[mattyb149]

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2056
  
Mind rebasing this against the latest master? I made some changes in 
MergeContent and I think that is causing it not to merge cleanly. Thanks in 
advance!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4262) MergeContent - option to add merged uuid in original flow files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124446#comment-16124446
 ] 

ASF GitHub Bot commented on NIFI-4262:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2056
  
Mind rebasing this against the latest master? I made some changes in 
MergeContent and I think that is causing it not to merge cleanly. Thanks in 
advance!


> MergeContent - option to add merged uuid in original flow files
> ---
>
> Key: NIFI-4262
> URL: https://issues.apache.org/jira/browse/NIFI-4262
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> With the apparition of Wait/Notify processors it is now possible tackle new 
> challenges when it comes to synchronize the execution of different parts of 
> the workflow.
> The objective here is the following:
> Flow files are sent to a MergeContent processor. Merged flow files are then 
> sent to a processor A while original flow files are sent to a processor B. I 
> want to trigger processor B when and only when processor A has completed.
> To use the Wait/Notify approach, a common attribute must be available to be 
> used as a signal in the distributed cache. This JIRA is about adding a 
> processor property allowing a user to add the UUID of the merged flow file as 
> a new attribute of all the original flow files that are constituting the 
> merged flow file.
> The template attached to NIFI-4028 can be used for this use case. Note that 
> the fix for NIFI-4028 is needed to solve the use case described in this JIRA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4237) EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest underlying cause

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-4237:
---
Fix Version/s: 1.4.0

> EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest 
> underlying cause
> 
>
> Key: NIFI-4237
> URL: https://issues.apache.org/jira/browse/NIFI-4237
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Russell Bateman
>Assignee: Andy LoPresto
>Priority: Minor
>  Labels: encryption, logging, security
> Fix For: 1.4.0
>
>
> Our Ansible instructions upgraded NiFi and created a new 
> {{nifi.sensitive.props.key}}. In _nifi.properties_ this property, if extant, 
> is used to encrypt sensitive properties in _flow.xml.gz_. Thus, upon 
> relaunching NiFi, the wrong key was used to decrypt resulting in the reported 
> failure to start, _flow.xml.gz_ is no longer useful.
> We found the problem and fixed it after Mark Payne suggested a possible 
> cause, but if this state of things can be determined, it might save on 
> community support for this situation if the logged message were to suggest 
> what's at the bottom of this problem. The top of the stack trace appears in 
> _logs/nifi-app.log_ as below:
> 2017-07-25 23:23:31,148 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.apache.nifi.encrypt.EncryptionException:
> org.jasypt.exceptions.EncryptionOperationNotPossibleException
> at
> org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at
> org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:474)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4237) EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest underlying cause

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124442#comment-16124442
 ] 

ASF subversion and git services commented on NIFI-4237:
---

Commit ae940d862420e00023eca2e996ad03646f3ed5a4 in nifi's branch 
refs/heads/master from [~alopresto]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ae940d8 ]

NIFI-4237 Added working test for StringEncryptor decryption of sensitive flow 
values in FlowFromDOMFactory.

NIFI-4237 Cleaned up unused alternate approaches.

NIFI-4237 Added failing unit test for better error message.

NIFI-4237 Added logic to capture unhelpful encryption exception and provide 
context in message. All tests pass.

This closes #2077


> EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest 
> underlying cause
> 
>
> Key: NIFI-4237
> URL: https://issues.apache.org/jira/browse/NIFI-4237
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Russell Bateman
>Assignee: Andy LoPresto
>Priority: Minor
>  Labels: encryption, logging, security
>
> Our Ansible instructions upgraded NiFi and created a new 
> {{nifi.sensitive.props.key}}. In _nifi.properties_ this property, if extant, 
> is used to encrypt sensitive properties in _flow.xml.gz_. Thus, upon 
> relaunching NiFi, the wrong key was used to decrypt resulting in the reported 
> failure to start, _flow.xml.gz_ is no longer useful.
> We found the problem and fixed it after Mark Payne suggested a possible 
> cause, but if this state of things can be determined, it might save on 
> community support for this situation if the logged message were to suggest 
> what's at the bottom of this problem. The top of the stack trace appears in 
> _logs/nifi-app.log_ as below:
> 2017-07-25 23:23:31,148 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.apache.nifi.encrypt.EncryptionException:
> org.jasypt.exceptions.EncryptionOperationNotPossibleException
> at
> org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at
> org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:474)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4237) EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest underlying cause

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-4237:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest 
> underlying cause
> 
>
> Key: NIFI-4237
> URL: https://issues.apache.org/jira/browse/NIFI-4237
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Russell Bateman
>Assignee: Andy LoPresto
>Priority: Minor
>  Labels: encryption, logging, security
> Fix For: 1.4.0
>
>
> Our Ansible instructions upgraded NiFi and created a new 
> {{nifi.sensitive.props.key}}. In _nifi.properties_ this property, if extant, 
> is used to encrypt sensitive properties in _flow.xml.gz_. Thus, upon 
> relaunching NiFi, the wrong key was used to decrypt resulting in the reported 
> failure to start, _flow.xml.gz_ is no longer useful.
> We found the problem and fixed it after Mark Payne suggested a possible 
> cause, but if this state of things can be determined, it might save on 
> community support for this situation if the logged message were to suggest 
> what's at the bottom of this problem. The top of the stack trace appears in 
> _logs/nifi-app.log_ as below:
> 2017-07-25 23:23:31,148 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.apache.nifi.encrypt.EncryptionException:
> org.jasypt.exceptions.EncryptionOperationNotPossibleException
> at
> org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at
> org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:474)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4237) EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest underlying cause

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124441#comment-16124441
 ] 

ASF subversion and git services commented on NIFI-4237:
---

Commit ae940d862420e00023eca2e996ad03646f3ed5a4 in nifi's branch 
refs/heads/master from [~alopresto]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ae940d8 ]

NIFI-4237 Added working test for StringEncryptor decryption of sensitive flow 
values in FlowFromDOMFactory.

NIFI-4237 Cleaned up unused alternate approaches.

NIFI-4237 Added failing unit test for better error message.

NIFI-4237 Added logic to capture unhelpful encryption exception and provide 
context in message. All tests pass.

This closes #2077


> EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest 
> underlying cause
> 
>
> Key: NIFI-4237
> URL: https://issues.apache.org/jira/browse/NIFI-4237
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Russell Bateman
>Assignee: Andy LoPresto
>Priority: Minor
>  Labels: encryption, logging, security
>
> Our Ansible instructions upgraded NiFi and created a new 
> {{nifi.sensitive.props.key}}. In _nifi.properties_ this property, if extant, 
> is used to encrypt sensitive properties in _flow.xml.gz_. Thus, upon 
> relaunching NiFi, the wrong key was used to decrypt resulting in the reported 
> failure to start, _flow.xml.gz_ is no longer useful.
> We found the problem and fixed it after Mark Payne suggested a possible 
> cause, but if this state of things can be determined, it might save on 
> community support for this situation if the logged message were to suggest 
> what's at the bottom of this problem. The top of the stack trace appears in 
> _logs/nifi-app.log_ as below:
> 2017-07-25 23:23:31,148 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.apache.nifi.encrypt.EncryptionException:
> org.jasypt.exceptions.EncryptionOperationNotPossibleException
> at
> org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at
> org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:474)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4237) EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest underlying cause

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124440#comment-16124440
 ] 

ASF subversion and git services commented on NIFI-4237:
---

Commit ae940d862420e00023eca2e996ad03646f3ed5a4 in nifi's branch 
refs/heads/master from [~alopresto]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ae940d8 ]

NIFI-4237 Added working test for StringEncryptor decryption of sensitive flow 
values in FlowFromDOMFactory.

NIFI-4237 Cleaned up unused alternate approaches.

NIFI-4237 Added failing unit test for better error message.

NIFI-4237 Added logic to capture unhelpful encryption exception and provide 
context in message. All tests pass.

This closes #2077


> EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest 
> underlying cause
> 
>
> Key: NIFI-4237
> URL: https://issues.apache.org/jira/browse/NIFI-4237
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Russell Bateman
>Assignee: Andy LoPresto
>Priority: Minor
>  Labels: encryption, logging, security
>
> Our Ansible instructions upgraded NiFi and created a new 
> {{nifi.sensitive.props.key}}. In _nifi.properties_ this property, if extant, 
> is used to encrypt sensitive properties in _flow.xml.gz_. Thus, upon 
> relaunching NiFi, the wrong key was used to decrypt resulting in the reported 
> failure to start, _flow.xml.gz_ is no longer useful.
> We found the problem and fixed it after Mark Payne suggested a possible 
> cause, but if this state of things can be determined, it might save on 
> community support for this situation if the logged message were to suggest 
> what's at the bottom of this problem. The top of the stack trace appears in 
> _logs/nifi-app.log_ as below:
> 2017-07-25 23:23:31,148 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.apache.nifi.encrypt.EncryptionException:
> org.jasypt.exceptions.EncryptionOperationNotPossibleException
> at
> org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at
> org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:474)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4237) EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest underlying cause

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1612#comment-1612
 ] 

ASF GitHub Bot commented on NIFI-4237:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2077


> EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest 
> underlying cause
> 
>
> Key: NIFI-4237
> URL: https://issues.apache.org/jira/browse/NIFI-4237
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Russell Bateman
>Assignee: Andy LoPresto
>Priority: Minor
>  Labels: encryption, logging, security
>
> Our Ansible instructions upgraded NiFi and created a new 
> {{nifi.sensitive.props.key}}. In _nifi.properties_ this property, if extant, 
> is used to encrypt sensitive properties in _flow.xml.gz_. Thus, upon 
> relaunching NiFi, the wrong key was used to decrypt resulting in the reported 
> failure to start, _flow.xml.gz_ is no longer useful.
> We found the problem and fixed it after Mark Payne suggested a possible 
> cause, but if this state of things can be determined, it might save on 
> community support for this situation if the logged message were to suggest 
> what's at the bottom of this problem. The top of the stack trace appears in 
> _logs/nifi-app.log_ as below:
> 2017-07-25 23:23:31,148 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.apache.nifi.encrypt.EncryptionException:
> org.jasypt.exceptions.EncryptionOperationNotPossibleException
> at
> org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at
> org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:474)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4237) EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest underlying cause

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124439#comment-16124439
 ] 

ASF subversion and git services commented on NIFI-4237:
---

Commit ae940d862420e00023eca2e996ad03646f3ed5a4 in nifi's branch 
refs/heads/master from [~alopresto]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ae940d8 ]

NIFI-4237 Added working test for StringEncryptor decryption of sensitive flow 
values in FlowFromDOMFactory.

NIFI-4237 Cleaned up unused alternate approaches.

NIFI-4237 Added failing unit test for better error message.

NIFI-4237 Added logic to capture unhelpful encryption exception and provide 
context in message. All tests pass.

This closes #2077


> EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest 
> underlying cause
> 
>
> Key: NIFI-4237
> URL: https://issues.apache.org/jira/browse/NIFI-4237
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Russell Bateman
>Assignee: Andy LoPresto
>Priority: Minor
>  Labels: encryption, logging, security
>
> Our Ansible instructions upgraded NiFi and created a new 
> {{nifi.sensitive.props.key}}. In _nifi.properties_ this property, if extant, 
> is used to encrypt sensitive properties in _flow.xml.gz_. Thus, upon 
> relaunching NiFi, the wrong key was used to decrypt resulting in the reported 
> failure to start, _flow.xml.gz_ is no longer useful.
> We found the problem and fixed it after Mark Payne suggested a possible 
> cause, but if this state of things can be determined, it might save on 
> community support for this situation if the logged message were to suggest 
> what's at the bottom of this problem. The top of the stack trace appears in 
> _logs/nifi-app.log_ as below:
> 2017-07-25 23:23:31,148 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.apache.nifi.encrypt.EncryptionException:
> org.jasypt.exceptions.EncryptionOperationNotPossibleException
> at
> org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at
> org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:474)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4237) EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest underlying cause

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124438#comment-16124438
 ] 

ASF GitHub Bot commented on NIFI-4237:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2077
  
I would like to learn why EncryptionException is a RuntimeException and not 
a checked exception, but since that's not related per se, +1 LGTM, merging to 
master, thank you!


> EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest 
> underlying cause
> 
>
> Key: NIFI-4237
> URL: https://issues.apache.org/jira/browse/NIFI-4237
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Russell Bateman
>Assignee: Andy LoPresto
>Priority: Minor
>  Labels: encryption, logging, security
>
> Our Ansible instructions upgraded NiFi and created a new 
> {{nifi.sensitive.props.key}}. In _nifi.properties_ this property, if extant, 
> is used to encrypt sensitive properties in _flow.xml.gz_. Thus, upon 
> relaunching NiFi, the wrong key was used to decrypt resulting in the reported 
> failure to start, _flow.xml.gz_ is no longer useful.
> We found the problem and fixed it after Mark Payne suggested a possible 
> cause, but if this state of things can be determined, it might save on 
> community support for this situation if the logged message were to suggest 
> what's at the bottom of this problem. The top of the stack trace appears in 
> _logs/nifi-app.log_ as below:
> 2017-07-25 23:23:31,148 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.apache.nifi.encrypt.EncryptionException:
> org.jasypt.exceptions.EncryptionOperationNotPossibleException
> at
> org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at
> org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:474)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2077: NIFI-4237 Improve error messaging on encryption fai...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2077


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #2077: NIFI-4237 Improve error messaging on encryption failures

[mattyb149]

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2077
  
I would like to learn why EncryptionException is a RuntimeException and not 
a checked exception, but since that's not related per se, +1 LGTM, merging to 
master, thank you!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4290) PublishKafkaRecord_0_10: failed to process due to java.lang.NullPointerException

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124437#comment-16124437
 ] 

Joseph Witt commented on NIFI-4290:
---

Removing fix version until there is contribution and review traction.  
Hopefully it gets fixed soon.

> PublishKafkaRecord_0_10: failed to process due to 
> java.lang.NullPointerException
> 
>
> Key: NIFI-4290
> URL: https://issues.apache.org/jira/browse/NIFI-4290
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.4.0
> Environment: NiFi 1.4 Master
> Confluent Kafka 3.2
>Reporter: Mayank Rathi
> Attachments: nifi-app.log
>
>
> Hello All,
> I am moving data to Kafka using NiFi's PublishKafkaRecord processor. I am 
> using ConfluentSchemaRegistry Controller service and getting below error:
> 2017-08-11 20:54:25,937 ERROR [Timer-Driven Process Thread-4] 
> o.a.n.p.k.pubsub.PublishKafkaRecord_0_10 
> PublishKafkaRecord_0_10[id=b3c03961-015d-1000-0946-79ccbe2ffbbd] 
> PublishKafkaRecord_0_10[id=b3c03961-015d-1000-0946-79ccbe2ffbbd] failed to 
> process due to java.lang.NullPointerException; rolling back session: {}
> java.lang.NullPointerException: null
> I do not see any error on Kafka side. 
> Attached are the logs after setting processors in Debug mode. Here is the 
> flow:
> ExecuteSQL --> SplitAvro --> PublishKafkaRecord_0_10
> Thanks!!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4290) PublishKafkaRecord_0_10: failed to process due to java.lang.NullPointerException

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-4290:
--
Fix Version/s: (was: 1.4.0)

> PublishKafkaRecord_0_10: failed to process due to 
> java.lang.NullPointerException
> 
>
> Key: NIFI-4290
> URL: https://issues.apache.org/jira/browse/NIFI-4290
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.4.0
> Environment: NiFi 1.4 Master
> Confluent Kafka 3.2
>Reporter: Mayank Rathi
> Attachments: nifi-app.log
>
>
> Hello All,
> I am moving data to Kafka using NiFi's PublishKafkaRecord processor. I am 
> using ConfluentSchemaRegistry Controller service and getting below error:
> 2017-08-11 20:54:25,937 ERROR [Timer-Driven Process Thread-4] 
> o.a.n.p.k.pubsub.PublishKafkaRecord_0_10 
> PublishKafkaRecord_0_10[id=b3c03961-015d-1000-0946-79ccbe2ffbbd] 
> PublishKafkaRecord_0_10[id=b3c03961-015d-1000-0946-79ccbe2ffbbd] failed to 
> process due to java.lang.NullPointerException; rolling back session: {}
> java.lang.NullPointerException: null
> I do not see any error on Kafka side. 
> Attached are the logs after setting processors in Debug mode. Here is the 
> flow:
> ExecuteSQL --> SplitAvro --> PublishKafkaRecord_0_10
> Thanks!!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4290) PublishKafkaRecord_0_10: failed to process due to java.lang.NullPointerException

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-4290:
--
Priority: Critical  (was: Minor)

> PublishKafkaRecord_0_10: failed to process due to 
> java.lang.NullPointerException
> 
>
> Key: NIFI-4290
> URL: https://issues.apache.org/jira/browse/NIFI-4290
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.4.0
> Environment: NiFi 1.4 Master
> Confluent Kafka 3.2
>Reporter: Mayank Rathi
>Priority: Critical
> Fix For: 1.4.0
>
> Attachments: nifi-app.log
>
>
> Hello All,
> I am moving data to Kafka using NiFi's PublishKafkaRecord processor. I am 
> using ConfluentSchemaRegistry Controller service and getting below error:
> 2017-08-11 20:54:25,937 ERROR [Timer-Driven Process Thread-4] 
> o.a.n.p.k.pubsub.PublishKafkaRecord_0_10 
> PublishKafkaRecord_0_10[id=b3c03961-015d-1000-0946-79ccbe2ffbbd] 
> PublishKafkaRecord_0_10[id=b3c03961-015d-1000-0946-79ccbe2ffbbd] failed to 
> process due to java.lang.NullPointerException; rolling back session: {}
> java.lang.NullPointerException: null
> I do not see any error on Kafka side. 
> Attached are the logs after setting processors in Debug mode. Here is the 
> flow:
> ExecuteSQL --> SplitAvro --> PublishKafkaRecord_0_10
> Thanks!!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4290) PublishKafkaRecord_0_10: failed to process due to java.lang.NullPointerException

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-4290:
--
Priority: Major  (was: Critical)

> PublishKafkaRecord_0_10: failed to process due to 
> java.lang.NullPointerException
> 
>
> Key: NIFI-4290
> URL: https://issues.apache.org/jira/browse/NIFI-4290
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.4.0
> Environment: NiFi 1.4 Master
> Confluent Kafka 3.2
>Reporter: Mayank Rathi
> Fix For: 1.4.0
>
> Attachments: nifi-app.log
>
>
> Hello All,
> I am moving data to Kafka using NiFi's PublishKafkaRecord processor. I am 
> using ConfluentSchemaRegistry Controller service and getting below error:
> 2017-08-11 20:54:25,937 ERROR [Timer-Driven Process Thread-4] 
> o.a.n.p.k.pubsub.PublishKafkaRecord_0_10 
> PublishKafkaRecord_0_10[id=b3c03961-015d-1000-0946-79ccbe2ffbbd] 
> PublishKafkaRecord_0_10[id=b3c03961-015d-1000-0946-79ccbe2ffbbd] failed to 
> process due to java.lang.NullPointerException; rolling back session: {}
> java.lang.NullPointerException: null
> I do not see any error on Kafka side. 
> Attached are the logs after setting processors in Debug mode. Here is the 
> flow:
> ExecuteSQL --> SplitAvro --> PublishKafkaRecord_0_10
> Thanks!!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4290) PublishKafkaRecord_0_10: failed to process due to java.lang.NullPointerException

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124436#comment-16124436
 ] 

Joseph Witt commented on NIFI-4290:
---

[~markap14] looks like this case can happen when the publisher/lease has been 
initialized but has not yet published any messages.  It is during the first 
publish that the inflightmessagetracker gets initialized for the first time yet 
if this never gets called, which can happen if the schema is not found, then 
the attempt to register the failure will fail due to this NPE.  ..it appears.  
Any reason we cannot initialize the tracker when we create the lease itself?

> PublishKafkaRecord_0_10: failed to process due to 
> java.lang.NullPointerException
> 
>
> Key: NIFI-4290
> URL: https://issues.apache.org/jira/browse/NIFI-4290
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.4.0
> Environment: NiFi 1.4 Master
> Confluent Kafka 3.2
>Reporter: Mayank Rathi
>Priority: Minor
> Fix For: 1.4.0
>
> Attachments: nifi-app.log
>
>
> Hello All,
> I am moving data to Kafka using NiFi's PublishKafkaRecord processor. I am 
> using ConfluentSchemaRegistry Controller service and getting below error:
> 2017-08-11 20:54:25,937 ERROR [Timer-Driven Process Thread-4] 
> o.a.n.p.k.pubsub.PublishKafkaRecord_0_10 
> PublishKafkaRecord_0_10[id=b3c03961-015d-1000-0946-79ccbe2ffbbd] 
> PublishKafkaRecord_0_10[id=b3c03961-015d-1000-0946-79ccbe2ffbbd] failed to 
> process due to java.lang.NullPointerException; rolling back session: {}
> java.lang.NullPointerException: null
> I do not see any error on Kafka side. 
> Attached are the logs after setting processors in Debug mode. Here is the 
> flow:
> ExecuteSQL --> SplitAvro --> PublishKafkaRecord_0_10
> Thanks!!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4286) GenerateTableFetch with Partition Size of 0 produces NPE

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-4286:
---
Status: Patch Available  (was: In Progress)

> GenerateTableFetch with Partition Size of 0 produces NPE
> 
>
> Key: NIFI-4286
> URL: https://issues.apache.org/jira/browse/NIFI-4286
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Bryan Bende
>Assignee: Matt Burgess
>Priority: Minor
>
> The description of partition size says "A value of zero indicates that a 
> single FlowFile is to be generated whose SQL statement will fetch all rows in 
> table".
> If you set the partition size to 0 the processor will get an NPE at these two 
> lines:
> long limit = partitionSize == 0 ? null : partitionSize;
> long offset = partitionSize == 0 ? null : i * partitionSize;



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4286) GenerateTableFetch with Partition Size of 0 produces NPE

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124432#comment-16124432
 ] 

ASF GitHub Bot commented on NIFI-4286:
--

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/2078

NIFI-4286: Fix NPE in GenerateTableFetch when Partition Size = 0

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-4286

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2078.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2078


commit 6910de784f1254a2bab5960963716d8758f82b75
Author: Matt Burgess 
Date:   2017-08-12T03:38:17Z

NIFI-4286: Fix NPE in GenerateTableFetch when Partition Size = 0




> GenerateTableFetch with Partition Size of 0 produces NPE
> 
>
> Key: NIFI-4286
> URL: https://issues.apache.org/jira/browse/NIFI-4286
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Bryan Bende
>Assignee: Matt Burgess
>Priority: Minor
>
> The description of partition size says "A value of zero indicates that a 
> single FlowFile is to be generated whose SQL statement will fetch all rows in 
> table".
> If you set the partition size to 0 the processor will get an NPE at these two 
> lines:
> long limit = partitionSize == 0 ? null : partitionSize;
> long offset = partitionSize == 0 ? null : i * partitionSize;



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2078: NIFI-4286: Fix NPE in GenerateTableFetch when Parti...

[mattyb149]

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/2078

NIFI-4286: Fix NPE in GenerateTableFetch when Partition Size = 0

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-4286

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2078.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2078


commit 6910de784f1254a2bab5960963716d8758f82b75
Author: Matt Burgess 
Date:   2017-08-12T03:38:17Z

NIFI-4286: Fix NPE in GenerateTableFetch when Partition Size = 0




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (NIFI-4286) GenerateTableFetch with Partition Size of 0 produces NPE

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess reassigned NIFI-4286:
--

Assignee: Matt Burgess

> GenerateTableFetch with Partition Size of 0 produces NPE
> 
>
> Key: NIFI-4286
> URL: https://issues.apache.org/jira/browse/NIFI-4286
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Bryan Bende
>Assignee: Matt Burgess
>Priority: Minor
>
> The description of partition size says "A value of zero indicates that a 
> single FlowFile is to be generated whose SQL statement will fetch all rows in 
> table".
> If you set the partition size to 0 the processor will get an NPE at these two 
> lines:
> long limit = partitionSize == 0 ? null : partitionSize;
> long offset = partitionSize == 0 ? null : i * partitionSize;



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4290) PublishKafkaRecord_0_10: failed to process due to java.lang.NullPointerException

[Mayank Rathi (JIRA)]

Mayank Rathi created NIFI-4290:
--

 Summary: PublishKafkaRecord_0_10: failed to process due to 
java.lang.NullPointerException
 Key: NIFI-4290
 URL: https://issues.apache.org/jira/browse/NIFI-4290
 Project: Apache NiFi
  Issue Type: Bug
  Components: Extensions
Affects Versions: 1.4.0
 Environment: NiFi 1.4 Master
Confluent Kafka 3.2
Reporter: Mayank Rathi
Priority: Minor
 Fix For: 1.4.0
 Attachments: nifi-app.log

Hello All,

I am moving data to Kafka using NiFi's PublishKafkaRecord processor. I am using 
ConfluentSchemaRegistry Controller service and getting below error:

2017-08-11 20:54:25,937 ERROR [Timer-Driven Process Thread-4] 
o.a.n.p.k.pubsub.PublishKafkaRecord_0_10 
PublishKafkaRecord_0_10[id=b3c03961-015d-1000-0946-79ccbe2ffbbd] 
PublishKafkaRecord_0_10[id=b3c03961-015d-1000-0946-79ccbe2ffbbd] failed to 
process due to java.lang.NullPointerException; rolling back session: {}
java.lang.NullPointerException: null

I do not see any error on Kafka side. 

Attached are the logs after setting processors in Debug mode. Here is the flow:

ExecuteSQL --> SplitAvro --> PublishKafkaRecord_0_10

Thanks!!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4289) Implement put processor for InfluxDB

[Mans Singh (JIRA)]

Mans Singh created NIFI-4289:


 Summary: Implement put processor for InfluxDB
 Key: NIFI-4289
 URL: https://issues.apache.org/jira/browse/NIFI-4289
 Project: Apache NiFi
  Issue Type: New Feature
  Components: Extensions
Affects Versions: 1.3.0
 Environment: All
Reporter: Mans Singh
Assignee: Mans Singh
Priority: Minor
 Fix For: 1.4.0


Support inserting time series measurements into InfluxDB.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4288) Intermittent test failures in TestPutUDP

[Andy LoPresto (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124346#comment-16124346
 ] 

Andy LoPresto commented on NIFI-4288:
-

[~joewitt] I am not sure what your preference was on this ticket -- mark it as 
resolved, set the fix version to 1.4.0 and close it, or leave the ticket open 
and do not assign a fix version? I have merged the PR. 

> Intermittent test failures in TestPutUDP
> 
>
> Key: NIFI-4288
> URL: https://issues.apache.org/jira/browse/NIFI-4288
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.3.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
>
> In some environments TestPutUDP will produce test failures for the following:
> TestPutUDP.testInvalidIPAddress:153->checkRelationships:227 expected:<0> but 
> was:<5>
> We can @Ignore this test for an immediate fix, or could refactor the test to 
> mock the ChannelSender so that we don't actually communicate with a live 
> server and can instead mock exceptions being thrown.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4288) Intermittent test failures in TestPutUDP

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124343#comment-16124343
 ] 

ASF GitHub Bot commented on NIFI-4288:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2076


> Intermittent test failures in TestPutUDP
> 
>
> Key: NIFI-4288
> URL: https://issues.apache.org/jira/browse/NIFI-4288
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.3.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
>
> In some environments TestPutUDP will produce test failures for the following:
> TestPutUDP.testInvalidIPAddress:153->checkRelationships:227 expected:<0> but 
> was:<5>
> We can @Ignore this test for an immediate fix, or could refactor the test to 
> mock the ChannelSender so that we don't actually communicate with a live 
> server and can instead mock exceptions being thrown.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2076: NIFI-4288 Ignoring tests that intermittently fail i...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2076


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4288) Intermittent test failures in TestPutUDP

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124341#comment-16124341
 ] 

ASF subversion and git services commented on NIFI-4288:
---

Commit 28d5a70ec0bf1c5b2126ae4bbb5af470012ed6b1 in nifi's branch 
refs/heads/master from [~bbende]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=28d5a70 ]

NIFI-4288 Ignoring tests that intermittently fail in TestPutUDP and TestPutTCP

This closes #2076.

Signed-off-by: Andy LoPresto 


> Intermittent test failures in TestPutUDP
> 
>
> Key: NIFI-4288
> URL: https://issues.apache.org/jira/browse/NIFI-4288
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.3.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
>
> In some environments TestPutUDP will produce test failures for the following:
> TestPutUDP.testInvalidIPAddress:153->checkRelationships:227 expected:<0> but 
> was:<5>
> We can @Ignore this test for an immediate fix, or could refactor the test to 
> mock the ChannelSender so that we don't actually communicate with a live 
> server and can instead mock exceptions being thrown.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4288) Intermittent test failures in TestPutUDP

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124342#comment-16124342
 ] 

ASF GitHub Bot commented on NIFI-4288:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2076
  
Ran `contrib-check` and all tests pass. +1, merging. 


> Intermittent test failures in TestPutUDP
> 
>
> Key: NIFI-4288
> URL: https://issues.apache.org/jira/browse/NIFI-4288
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.3.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
>
> In some environments TestPutUDP will produce test failures for the following:
> TestPutUDP.testInvalidIPAddress:153->checkRelationships:227 expected:<0> but 
> was:<5>
> We can @Ignore this test for an immediate fix, or could refactor the test to 
> mock the ChannelSender so that we don't actually communicate with a live 
> server and can instead mock exceptions being thrown.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2076: NIFI-4288 Ignoring tests that intermittently fail in TestP...

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2076
  
Ran `contrib-check` and all tests pass. +1, merging. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124137#comment-16124137
 ] 

ASF GitHub Bot commented on NIFI-4023:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2075
  
seeing far better behavior on the flow i could cause these issues on.  Such 
as the one i posted in NIFI-3329.  Will monitor a bit more and if still good 
then merge


> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:36,420 ERROR [Index Provenance Events-1] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:37,425 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> 

[GitHub] nifi issue #2075: NIFI-4023, NIFI-4077: Addressed issue where repository was...

[joewitt]

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2075
  
seeing far better behavior on the flow i could cause these issues on.  Such 
as the one i posted in NIFI-3329.  Will monitor a bit more and if still good 
then merge


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4142) Implement a ValidateRecord Processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124100#comment-16124100
 ] 

ASF GitHub Bot commented on NIFI-4142:
--

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2015#discussion_r132786485
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateRecord.java
 ---
@@ -0,0 +1,457 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.avro.Schema;
+import org.apache.avro.Schema.Parser;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.avro.AvroSchemaValidator;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaField;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schema.validation.SchemaValidationContext;
+import org.apache.nifi.schema.validation.StandardSchemaValidator;
+import org.apache.nifi.schemaregistry.services.SchemaRegistry;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.RawRecordWriter;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+import 
org.apache.nifi.serialization.record.validation.RecordSchemaValidator;
+import 
org.apache.nifi.serialization.record.validation.SchemaValidationResult;
+import org.apache.nifi.serialization.record.validation.ValidationError;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"record", "schema", "validate"})
+@CapabilityDescription("Validates the Records of an incoming FlowFile 
against a given schema. All records that adhere to the schema are routed to the 
\"valid\" relationship while "
++ "records that do not adhere to hte schema are routed to the 
\"invalid\" relationship. It is therefore possible for a single incoming 
FlowFile to be split into two individual "
++ "FlowFiles if some records are valid according to the schema and 
others are not. Any FlowFile that is 

[GitHub] nifi pull request #2015: NIFI-4142: Refactored Record Reader/Writer to allow...

[joewitt]

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2015#discussion_r132786485
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateRecord.java
 ---
@@ -0,0 +1,457 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import org.apache.avro.Schema;
+import org.apache.avro.Schema.Parser;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.avro.AvroSchemaValidator;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaField;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schema.validation.SchemaValidationContext;
+import org.apache.nifi.schema.validation.StandardSchemaValidator;
+import org.apache.nifi.schemaregistry.services.SchemaRegistry;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.RawRecordWriter;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+import 
org.apache.nifi.serialization.record.validation.RecordSchemaValidator;
+import 
org.apache.nifi.serialization.record.validation.SchemaValidationResult;
+import org.apache.nifi.serialization.record.validation.ValidationError;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"record", "schema", "validate"})
+@CapabilityDescription("Validates the Records of an incoming FlowFile 
against a given schema. All records that adhere to the schema are routed to the 
\"valid\" relationship while "
++ "records that do not adhere to hte schema are routed to the 
\"invalid\" relationship. It is therefore possible for a single incoming 
FlowFile to be split into two individual "
++ "FlowFiles if some records are valid according to the schema and 
others are not. Any FlowFile that is routed to the \"invalid\" relationship 
will emit a ROUTE Provenance Event "
++ "with the Details field populated to explain why records were 
invalid. In addition, to gain further explanation of why records were invalid, 

[jira] [Commented] (NIFI-4142) Implement a ValidateRecord Processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124094#comment-16124094
 ] 

ASF GitHub Bot commented on NIFI-4142:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2015
  
@markap14  looks like the option says "Used Reader's Schema".  I'm assuming 
that is a typo and it should say "Use Reader's Schema".  But that is not in 
this PR it appears.  Did that sneak in elsewhere?


> Implement a ValidateRecord Processor
> 
>
> Key: NIFI-4142
> URL: https://issues.apache.org/jira/browse/NIFI-4142
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> We need a processor that is capable of validating that all Records in a 
> FlowFile adhere to the proper schema.
> The Processor should be configured with a Record Reader and should route each 
> record to either 'valid' or 'invalid' based on whether or not the record 
> adheres to the reader's schema. A record would be invalid in any of the 
> following cases:
> - Missing field that is required according to the schema
> - Extra field that is not present in schema (it should be configurable 
> whether or not this is a failure)
> - Field requires coercion and strict type checking enabled (this should also 
> be configurable)
> - Field is invalid, such as the value "hello" when it should be an integer



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2015: NIFI-4142: Refactored Record Reader/Writer to allow for re...

[joewitt]

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2015
  
@markap14  looks like the option says "Used Reader's Schema".  I'm assuming 
that is a typo and it should say "Use Reader's Schema".  But that is not in 
this PR it appears.  Did that sneak in elsewhere?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4288) Intermittent test failures in TestPutUDP

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124072#comment-16124072
 ] 

ASF GitHub Bot commented on NIFI-4288:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2076
  
Reviewing...


> Intermittent test failures in TestPutUDP
> 
>
> Key: NIFI-4288
> URL: https://issues.apache.org/jira/browse/NIFI-4288
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.3.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
>
> In some environments TestPutUDP will produce test failures for the following:
> TestPutUDP.testInvalidIPAddress:153->checkRelationships:227 expected:<0> but 
> was:<5>
> We can @Ignore this test for an immediate fix, or could refactor the test to 
> mock the ChannelSender so that we don't actually communicate with a live 
> server and can instead mock exceptions being thrown.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2076: NIFI-4288 Ignoring tests that intermittently fail in TestP...

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2076
  
Reviewing...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-4237) EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest underlying cause

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto updated NIFI-4237:

Description: 
Our Ansible instructions upgraded NiFi and created a new 
{{nifi.sensitive.props.key}}. In _nifi.properties_ this property, if extant, is 
used to encrypt sensitive properties in _flow.xml.gz_. Thus, upon relaunching 
NiFi, the wrong key was used to decrypt resulting in the reported failure to 
start, _flow.xml.gz_ is no longer useful.

We found the problem and fixed it after Mark Payne suggested a possible cause, 
but if this state of things can be determined, it might save on community 
support for this situation if the logged message were to suggest what's at the 
bottom of this problem. The top of the stack trace appears in 
_logs/nifi-app.log_ as below:

2017-07-25 23:23:31,148 WARN [main] org.apache.nifi.web.server.JettyServer
Failed to start web server... shutting down.
org.apache.nifi.encrypt.EncryptionException:
org.jasypt.exceptions.EncryptionOperationNotPossibleException
at
org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
~[nifi-framework-core-1.1.2.jar:1.1.2]
at
org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:474)
~[nifi-framework-core-1.1.2.jar:1.1.2]
at...

  was:
Our Ansible instructions upgraded NiFi and created a new 
{{nifi.sensitive.props.key}}. In _nifi.properties_ this property, if extant, is 
used to encrypt sensitive properties in _flow.xml.gz_. Thus, upon relaunching 
NiFi, the wrong key was used to decrypt resulting in the reported failure to 
start, _flow.xml.gz_ is no longer useful.

We found the problem and fixed it after Mark Payne suggested a possible cause, 
but if this state of things can be determined, it might save on community 
support for this situation if the logged message were to suggest what's at the 
bottom of this problem. The top of the stack trace appears in 
_logs/nifi-bootstrap.log_ as below:

2017-07-25 23:23:31,148 WARN [main] org.apache.nifi.web.server.JettyServer
Failed to start web server... shutting down.
org.apache.nifi.encrypt.EncryptionException:
org.jasypt.exceptions.EncryptionOperationNotPossibleException
at
org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
~[nifi-framework-core-1.1.2.jar:1.1.2]
at
org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:474)
~[nifi-framework-core-1.1.2.jar:1.1.2]
at...


> EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest 
> underlying cause
> 
>
> Key: NIFI-4237
> URL: https://issues.apache.org/jira/browse/NIFI-4237
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Russell Bateman
>Assignee: Andy LoPresto
>Priority: Minor
>  Labels: encryption, logging, security
>
> Our Ansible instructions upgraded NiFi and created a new 
> {{nifi.sensitive.props.key}}. In _nifi.properties_ this property, if extant, 
> is used to encrypt sensitive properties in _flow.xml.gz_. Thus, upon 
> relaunching NiFi, the wrong key was used to decrypt resulting in the reported 
> failure to start, _flow.xml.gz_ is no longer useful.
> We found the problem and fixed it after Mark Payne suggested a possible 
> cause, but if this state of things can be determined, it might save on 
> community support for this situation if the logged message were to suggest 
> what's at the bottom of this problem. The top of the stack trace appears in 
> _logs/nifi-app.log_ as below:
> 2017-07-25 23:23:31,148 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.apache.nifi.encrypt.EncryptionException:
> org.jasypt.exceptions.EncryptionOperationNotPossibleException
> at
> org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at
> org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:474)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4237) EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest underlying cause

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto updated NIFI-4237:

   Labels: encryption logging security  (was: )
Affects Version/s: 1.3.0
   Status: Patch Available  (was: Open)

> EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest 
> underlying cause
> 
>
> Key: NIFI-4237
> URL: https://issues.apache.org/jira/browse/NIFI-4237
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Russell Bateman
>Assignee: Andy LoPresto
>Priority: Minor
>  Labels: security, encryption, logging
>
> Our Ansible instructions upgraded NiFi and created a new 
> {{nifi.sensitive.props.key}}. In _nifi.properties_ this property, if extant, 
> is used to encrypt sensitive properties in _flow.xml.gz_. Thus, upon 
> relaunching NiFi, the wrong key was used to decrypt resulting in the reported 
> failure to start, _flow.xml.gz_ is no longer useful.
> We found the problem and fixed it after Mark Payne suggested a possible 
> cause, but if this state of things can be determined, it might save on 
> community support for this situation if the logged message were to suggest 
> what's at the bottom of this problem. The top of the stack trace appears in 
> _logs/nifi-bootstrap.log_ as below:
> 2017-07-25 23:23:31,148 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.apache.nifi.encrypt.EncryptionException:
> org.jasypt.exceptions.EncryptionOperationNotPossibleException
> at
> org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at
> org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:474)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4237) EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest underlying cause

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124065#comment-16124065
 ] 

ASF GitHub Bot commented on NIFI-4237:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2077
  
I added a unit test which ensures the log error output is more helpful for 
users when the `flow.xml.gz` decryption fails. 

Verify with deterministic cipher text vectors:

```
1966 [main] INFO  
org.apache.nifi.controller.serialization.FlowFromDOMFactoryTest - Cipher text: 
enc{8ae49f94922876d07602e737f9d0095f397e8bdf73d3151ecde439e24af95715}
2068 [main] ERROR 
org.apache.nifi.controller.serialization.FlowFromDOMFactory - There was a 
problem decrypting a sensitive flow configuration value. Check that the 
nifi.sensitive.props.key value in nifi.properties matches the value used to 
encrypt the flow.xml.gz file
org.apache.nifi.encrypt.EncryptionException: 
org.jasypt.exceptions.EncryptionOperationNotPossibleException
at 
org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
at 
org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:501)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at 
org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrapNoCoerce.invoke(StaticMetaMethodSite.java:151)
at 
org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:91)
at 
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
at 
org.apache.nifi.controller.serialization.FlowFromDOMFactoryTest$_testShouldProvideBetterErrorMessageOnDecryptionFailure_closure2.doCall(FlowFromDOMFactoryTest.groovy:129)
at 
org.apache.nifi.controller.serialization.FlowFromDOMFactoryTest$_testShouldProvideBetterErrorMessageOnDecryptionFailure_closure2.doCall(FlowFromDOMFactoryTest.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at 
org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
at groovy.lang.Closure.call(Closure.java:426)
at groovy.lang.Closure.call(Closure.java:420)
at groovy.test.GroovyAssert.shouldFail(GroovyAssert.java:119)
at groovy.test.GroovyAssert$shouldFail.callStatic(Unknown Source)
at 
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:56)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:194)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:214)
at 
org.apache.nifi.controller.serialization.FlowFromDOMFactoryTest.testShouldProvideBetterErrorMessageOnDecryptionFailure(FlowFromDOMFactoryTest.groovy:128)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 

[GitHub] nifi issue #2077: NIFI-4237 Improve error messaging on encryption failures

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2077
  
I added a unit test which ensures the log error output is more helpful for 
users when the `flow.xml.gz` decryption fails. 

Verify with deterministic cipher text vectors:

```
1966 [main] INFO  
org.apache.nifi.controller.serialization.FlowFromDOMFactoryTest - Cipher text: 
enc{8ae49f94922876d07602e737f9d0095f397e8bdf73d3151ecde439e24af95715}
2068 [main] ERROR 
org.apache.nifi.controller.serialization.FlowFromDOMFactory - There was a 
problem decrypting a sensitive flow configuration value. Check that the 
nifi.sensitive.props.key value in nifi.properties matches the value used to 
encrypt the flow.xml.gz file
org.apache.nifi.encrypt.EncryptionException: 
org.jasypt.exceptions.EncryptionOperationNotPossibleException
at 
org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
at 
org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:501)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at 
org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrapNoCoerce.invoke(StaticMetaMethodSite.java:151)
at 
org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:91)
at 
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
at 
org.apache.nifi.controller.serialization.FlowFromDOMFactoryTest$_testShouldProvideBetterErrorMessageOnDecryptionFailure_closure2.doCall(FlowFromDOMFactoryTest.groovy:129)
at 
org.apache.nifi.controller.serialization.FlowFromDOMFactoryTest$_testShouldProvideBetterErrorMessageOnDecryptionFailure_closure2.doCall(FlowFromDOMFactoryTest.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at 
org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
at groovy.lang.Closure.call(Closure.java:426)
at groovy.lang.Closure.call(Closure.java:420)
at groovy.test.GroovyAssert.shouldFail(GroovyAssert.java:119)
at groovy.test.GroovyAssert$shouldFail.callStatic(Unknown Source)
at 
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:56)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:194)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:214)
at 
org.apache.nifi.controller.serialization.FlowFromDOMFactoryTest.testShouldProvideBetterErrorMessageOnDecryptionFailure(FlowFromDOMFactoryTest.groovy:128)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at 

[jira] [Commented] (NIFI-4237) EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest underlying cause

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124059#comment-16124059
 ] 

ASF GitHub Bot commented on NIFI-4237:
--

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/2077

NIFI-4237 Improve error messaging on encryption failures

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-4237

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2077.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2077


commit 5752925ac51035db1a5e9f7e9797e308fc5770f2
Author: Andy LoPresto 
Date:   2017-08-11T20:15:16Z

NIFI-4237 Added working test for StringEncryptor decryption of sensitive 
flow values in FlowFromDOMFactory.

commit 7c35cfccb2868814184aa4768ee398d0038eaab1
Author: Andy LoPresto 
Date:   2017-08-11T20:17:09Z

NIFI-4237 Cleaned up unused alternate approaches.

commit a45a259a99a176aea848190d23253a54e88aed95
Author: Andy LoPresto 
Date:   2017-08-11T20:20:11Z

NIFI-4237 Added failing unit test for better error message.

commit 1a2a4ddebdd506b058b359747f0a1016a6ff3346
Author: Andy LoPresto 
Date:   2017-08-11T20:30:44Z

NIFI-4237 Added logic to capture unhelpful encryption exception and provide 
context in message.
All tests pass.




> EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest 
> underlying cause
> 
>
> Key: NIFI-4237
> URL: https://issues.apache.org/jira/browse/NIFI-4237
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Russell Bateman
>Assignee: Andy LoPresto
>Priority: Minor
>
> Our Ansible instructions upgraded NiFi and created a new 
> {{nifi.sensitive.props.key}}. In _nifi.properties_ this property, if extant, 
> is used to encrypt sensitive properties in _flow.xml.gz_. Thus, upon 
> relaunching NiFi, the wrong key was used to decrypt resulting in the reported 
> failure to start, _flow.xml.gz_ is no longer useful.
> We found the problem and fixed it after Mark Payne suggested a possible 
> cause, but if this state of things can be determined, it might save on 
> community support for this situation if the logged message were to suggest 
> what's at the bottom of this problem. The top of the stack trace appears in 
> _logs/nifi-bootstrap.log_ as below:
> 2017-07-25 23:23:31,148 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.apache.nifi.encrypt.EncryptionException:
> org.jasypt.exceptions.EncryptionOperationNotPossibleException
> at
> org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
> 

[GitHub] nifi pull request #2077: NIFI-4237 Improve error messaging on encryption fai...

[alopresto]

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/2077

NIFI-4237 Improve error messaging on encryption failures

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-4237

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2077.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2077


commit 5752925ac51035db1a5e9f7e9797e308fc5770f2
Author: Andy LoPresto 
Date:   2017-08-11T20:15:16Z

NIFI-4237 Added working test for StringEncryptor decryption of sensitive 
flow values in FlowFromDOMFactory.

commit 7c35cfccb2868814184aa4768ee398d0038eaab1
Author: Andy LoPresto 
Date:   2017-08-11T20:17:09Z

NIFI-4237 Cleaned up unused alternate approaches.

commit a45a259a99a176aea848190d23253a54e88aed95
Author: Andy LoPresto 
Date:   2017-08-11T20:20:11Z

NIFI-4237 Added failing unit test for better error message.

commit 1a2a4ddebdd506b058b359747f0a1016a6ff3346
Author: Andy LoPresto 
Date:   2017-08-11T20:30:44Z

NIFI-4237 Added logic to capture unhelpful encryption exception and provide 
context in message.
All tests pass.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4288) Intermittent test failures in TestPutUDP

[Bryan Bende (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124044#comment-16124044
 ] 

Bryan Bende commented on NIFI-4288:
---

Put up a PR that ignores the test and included an ignore for the same type of 
tests in TestPutTCP.

> Intermittent test failures in TestPutUDP
> 
>
> Key: NIFI-4288
> URL: https://issues.apache.org/jira/browse/NIFI-4288
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.3.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
>
> In some environments TestPutUDP will produce test failures for the following:
> TestPutUDP.testInvalidIPAddress:153->checkRelationships:227 expected:<0> but 
> was:<5>
> We can @Ignore this test for an immediate fix, or could refactor the test to 
> mock the ChannelSender so that we don't actually communicate with a live 
> server and can instead mock exceptions being thrown.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-4288) Intermittent test failures in TestPutUDP

[Bryan Bende (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Bende reassigned NIFI-4288:
-

Assignee: Bryan Bende

> Intermittent test failures in TestPutUDP
> 
>
> Key: NIFI-4288
> URL: https://issues.apache.org/jira/browse/NIFI-4288
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.3.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
>
> In some environments TestPutUDP will produce test failures for the following:
> TestPutUDP.testInvalidIPAddress:153->checkRelationships:227 expected:<0> but 
> was:<5>
> We can @Ignore this test for an immediate fix, or could refactor the test to 
> mock the ChannelSender so that we don't actually communicate with a live 
> server and can instead mock exceptions being thrown.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2076: NIFI-4288 Ignoring tests that intermittently fail i...

[bbende]

GitHub user bbende opened a pull request:

https://github.com/apache/nifi/pull/2076

NIFI-4288 Ignoring tests that intermittently fail in TestPutUDP and T…

…estPutTCP

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bbende/nifi NIFI-4288

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2076.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2076


commit 0b7450fc6bf975a4896cdc326478dd90a02c89e6
Author: Bryan Bende 
Date:   2017-08-11T20:55:52Z

NIFI-4288 Ignoring tests that intermittently fail in TestPutUDP and 
TestPutTCP




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4288) Intermittent test failures in TestPutUDP

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124023#comment-16124023
 ] 

Joseph Witt commented on NIFI-4288:
---

meh probably should ignore/merge/close with 1.4.0 fix version.  We have a JIRA 
already to deal with ignored tests

> Intermittent test failures in TestPutUDP
> 
>
> Key: NIFI-4288
> URL: https://issues.apache.org/jira/browse/NIFI-4288
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.3.0
>Reporter: Bryan Bende
>Priority: Minor
>
> In some environments TestPutUDP will produce test failures for the following:
> TestPutUDP.testInvalidIPAddress:153->checkRelationships:227 expected:<0> but 
> was:<5>
> We can @Ignore this test for an immediate fix, or could refactor the test to 
> mock the ChannelSender so that we don't actually communicate with a live 
> server and can instead mock exceptions being thrown.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4288) Intermittent test failures in TestPutUDP

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124020#comment-16124020
 ] 

Joseph Witt commented on NIFI-4288:
---

am favorable to ignoring for now and merging, leaving ticket open, and no 
assigning a fix version.

> Intermittent test failures in TestPutUDP
> 
>
> Key: NIFI-4288
> URL: https://issues.apache.org/jira/browse/NIFI-4288
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.3.0
>Reporter: Bryan Bende
>Priority: Minor
>
> In some environments TestPutUDP will produce test failures for the following:
> TestPutUDP.testInvalidIPAddress:153->checkRelationships:227 expected:<0> but 
> was:<5>
> We can @Ignore this test for an immediate fix, or could refactor the test to 
> mock the ChannelSender so that we don't actually communicate with a live 
> server and can instead mock exceptions being thrown.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4288) Intermittent test failures in TestPutUDP

[Bryan Bende (JIRA)]

Bryan Bende created NIFI-4288:
-

 Summary: Intermittent test failures in TestPutUDP
 Key: NIFI-4288
 URL: https://issues.apache.org/jira/browse/NIFI-4288
 Project: Apache NiFi
  Issue Type: Improvement
Affects Versions: 1.3.0
Reporter: Bryan Bende
Priority: Minor


In some environments TestPutUDP will produce test failures for the following:

TestPutUDP.testInvalidIPAddress:153->checkRelationships:227 expected:<0> but 
was:<5>

We can @Ignore this test for an immediate fix, or could refactor the test to 
mock the ChannelSender so that we don't actually communicate with a live server 
and can instead mock exceptions being thrown.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3736) NiFi not honoring the "nifi.content.claim.max.appendable.size" property

[Mark Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123963#comment-16123963
 ] 

Mark Payne commented on NIFI-3736:
--

[~joewitt] since you are the one that re-opened the JIRA - are you okay with 
closing this now, after having created NIFI-4287?

> NiFi not honoring the "nifi.content.claim.max.appendable.size" property
> ---
>
> Key: NIFI-3736
> URL: https://issues.apache.org/jira/browse/NIFI-3736
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Michael Hogue
> Fix For: 1.4.0
>
>
> The nifi.properties file has two properties for controlling how many 
> FlowFiles to jam into one Content Claim. Unfortunately, it looks like this is 
> no longer honored in FileSystemRepository.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3736) NiFi not honoring the "nifi.content.claim.max.appendable.size" and "nifi.content.claim.max.flow.files" properties

[Mark Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123960#comment-16123960
 ] 

Mark Payne commented on NIFI-3736:
--

I spawned a new JIRA to address the last issue here, that we are not honoring 
the "nifi.content.claim.max.flow.files". Since the PR here does address the 
first part of "nifi.content.claim.max.appendable.size" I think it makes sense 
to just rename this ticket and spawn a new thread for the other (related but 
separate) issue.

> NiFi not honoring the "nifi.content.claim.max.appendable.size" and 
> "nifi.content.claim.max.flow.files" properties
> -
>
> Key: NIFI-3736
> URL: https://issues.apache.org/jira/browse/NIFI-3736
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Michael Hogue
> Fix For: 1.4.0
>
>
> The nifi.properties file has two properties for controlling how many 
> FlowFiles to jam into one Content Claim. Unfortunately, it looks like this is 
> no longer honored in FileSystemRepository.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4287) NiFi not honoring the "nifi.content.claim.max.flow.files" property

[Mark Payne (JIRA)]

Mark Payne created NIFI-4287:


 Summary: NiFi not honoring the "nifi.content.claim.max.flow.files" 
property
 Key: NIFI-4287
 URL: https://issues.apache.org/jira/browse/NIFI-4287
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Reporter: Mark Payne






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3736) NiFi not honoring the "nifi.content.claim.max.appendable.size" and "nifi.content.claim.max.flow.files" properties

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123937#comment-16123937
 ] 

ASF subversion and git services commented on NIFI-3736:
---

Commit d6ebc53076f597c038e81600dccddc5d16b27663 in nifi's branch 
refs/heads/master from [~boardm26]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d6ebc53 ]

NIFI-3736 modify default nifi.content.claim.max.appendable.size in 
nifi.properties to 1 MB


> NiFi not honoring the "nifi.content.claim.max.appendable.size" and 
> "nifi.content.claim.max.flow.files" properties
> -
>
> Key: NIFI-3736
> URL: https://issues.apache.org/jira/browse/NIFI-3736
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Michael Hogue
> Fix For: 1.4.0
>
>
> The nifi.properties file has two properties for controlling how many 
> FlowFiles to jam into one Content Claim. Unfortunately, it looks like this is 
> no longer honored in FileSystemRepository.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3736) NiFi not honoring the "nifi.content.claim.max.appendable.size" and "nifi.content.claim.max.flow.files" properties

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123935#comment-16123935
 ] 

ASF GitHub Bot commented on NIFI-3736:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2041
  
Sorry - missed this comment. Nope, no strong intent. Perhaps even just a 
typo. Will merge.


> NiFi not honoring the "nifi.content.claim.max.appendable.size" and 
> "nifi.content.claim.max.flow.files" properties
> -
>
> Key: NIFI-3736
> URL: https://issues.apache.org/jira/browse/NIFI-3736
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Michael Hogue
> Fix For: 1.4.0
>
>
> The nifi.properties file has two properties for controlling how many 
> FlowFiles to jam into one Content Claim. Unfortunately, it looks like this is 
> no longer honored in FileSystemRepository.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2041: NIFI-3736 modify default nifi.content.claim.max.appendable...

[markap14]

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2041
  
Sorry - missed this comment. Nope, no strong intent. Perhaps even just a 
typo. Will merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123926#comment-16123926
 ] 

ASF GitHub Bot commented on NIFI-4023:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2075
  
on it.  Will let it run for a while.


> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:36,420 ERROR [Index Provenance Events-1] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:37,425 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 

[jira] [Commented] (NIFI-4142) Implement a ValidateRecord Processor

[Mark Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123885#comment-16123885
 ] 

Mark Payne commented on NIFI-4142:
--

[~joewitt] rebased and squashed. Thanks!

> Implement a ValidateRecord Processor
> 
>
> Key: NIFI-4142
> URL: https://issues.apache.org/jira/browse/NIFI-4142
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> We need a processor that is capable of validating that all Records in a 
> FlowFile adhere to the proper schema.
> The Processor should be configured with a Record Reader and should route each 
> record to either 'valid' or 'invalid' based on whether or not the record 
> adheres to the reader's schema. A record would be invalid in any of the 
> following cases:
> - Missing field that is required according to the schema
> - Extra field that is not present in schema (it should be configurable 
> whether or not this is a failure)
> - Field requires coercion and strict type checking enabled (this should also 
> be configurable)
> - Field is invalid, such as the value "hello" when it should be an integer



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-4023:
-
Fix Version/s: 1.4.0
   Status: Patch Available  (was: Open)

> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:36,420 ERROR [Index Provenance Events-1] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:37,425 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> 

[jira] [Updated] (NIFI-4077) o.a.n.p.index.lucene.LuceneEventIndex Failed to retrieve Provenance Events from store

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-4077:
-
 Assignee: Mark Payne
Fix Version/s: 1.4.0
   Status: Patch Available  (was: Open)

> o.a.n.p.index.lucene.LuceneEventIndex Failed to retrieve Provenance Events 
> from store
> -
>
> Key: NIFI-4077
> URL: https://issues.apache.org/jira/browse/NIFI-4077
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Joseph Gresock
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> I get this error when trying to display Data Provenance on a 4-node cluster:
> 2017-06-15 18:28:23,945 ERROR [Provenance Query-2] 
> o.a.n.p.index.lucene.LuceneEventIndex Failed to retrieve Provenance Events 
> from store
> java.io.FileNotFoundException: Unable to locate file 
> /data/nifi/provenance_repository/85309321.prov
> at 
> org.apache.nifi.provenance.serialization.RecordReaders.newRecordReader(RecordReaders.java:119)
> at 
> org.apache.nifi.provenance.WriteAheadProvenanceRepository.lambda$initialize$46(WriteAheadProvenanceRepository.java:125)
> at 
> org.apache.nifi.provenance.store.iterator.SelectiveRecordReaderEventIterator.nextEvent(SelectiveRecordReaderEventIterator.java:137)
> at 
> org.apache.nifi.provenance.store.iterator.AuthorizingEventIterator.nextEvent(AuthorizingEventIterator.java:47)
> at 
> org.apache.nifi.provenance.store.PartitionedEventStore.getEvents(PartitionedEventStore.java:192)
> at 
> org.apache.nifi.provenance.store.PartitionedEventStore.getEvents(PartitionedEventStore.java:167)
> at 
> org.apache.nifi.provenance.index.lucene.LuceneEventIndex.lambda$submitQuery$58(LuceneEventIndex.java:442)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> I don't know if this is related, but I also see this error in the same logs:
> 2017-06-15 18:28:42,923 ERROR [Compress Provenance Logs-1-thread-1] 
> o.a.n.p.s.EventFileCompressor Failed to read TOC File 
> /data/nifi/provenance_repository/toc/87247412.toc
> java.io.EOFException: null
> at 
> org.apache.nifi.provenance.toc.StandardTocReader.(StandardTocReader.java:48)
> at 
> org.apache.nifi.provenance.serialization.EventFileCompressor.run(EventFileCompressor.java:93)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne reassigned NIFI-4023:


Assignee: Mark Payne

> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:36,420 ERROR [Index Provenance Events-1] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:37,425 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> 

[jira] [Commented] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123848#comment-16123848
 ] 

ASF GitHub Bot commented on NIFI-4023:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2075

NIFI-4023, NIFI-4077: Addressed issue where repository was aging off …

…the wrong index. When it should age off Index 1, it was removing Index 2. 
As a result, the earliest index is never aged off, and the newest index could 
potentially be aged off before it is ready to be. Also addressed issue where a 
query that attempts to read an event that has aged off throws 
FileNotFoundException (NIFI-4077) instead of skipping over the event. The 
JavaDocs indicate that the EventIterator should skip records that it cannot 
find, but SelectiveRecordReaderEventIterator throw FileNotFoundException instead

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4023

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2075.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2075


commit 7e60e9e26481237a0f1d7c9d9b056a4604177b72
Author: Mark Payne 
Date:   2017-08-11T17:08:29Z

NIFI-4023, NIFI-4077: Addressed issue where repository was aging off the 
wrong index. When it should age off Index 1, it was removing Index 2. As a 
result, the earliest index is never aged off, and the newest index could 
potentially be aged off before it is ready to be. Also addressed issue where a 
query that attempts to read an event that has aged off throws 
FileNotFoundException (NIFI-4077) instead of skipping over the event. The 
JavaDocs indicate that the EventIterator should skip records that it cannot 
find, but SelectiveRecordReaderEventIterator throw FileNotFoundException instead




> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> 

[jira] [Commented] (NIFI-4276) Add Provenance Migration section to User Guide

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123818#comment-16123818
 ] 

ASF GitHub Bot commented on NIFI-4276:
--

Github user scottyaslan commented on the issue:

https://github.com/apache/nifi/pull/2074
  
Reviewing...


> Add Provenance Migration section to User Guide
> --
>
> Key: NIFI-4276
> URL: https://issues.apache.org/jira/browse/NIFI-4276
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> WriteAheadProvenanceRepository configuration was introduced in version 1.2.0 
> (NIFI-3356).  Should add a section to the documentation that covers how to 
> migrate from default configuration of PersistentProvenantRepository and 
> recommend any system property value changes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #2074: NIFI-4276 Add Write Ahead Provenance section to User Guide

[scottyaslan]

Github user scottyaslan commented on the issue:

https://github.com/apache/nifi/pull/2074
  
Reviewing...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #2074: NIFI-4276 Add Write Ahead Provenance section to Use...

[andrewmlim]

GitHub user andrewmlim opened a pull request:

https://github.com/apache/nifi/pull/2074

NIFI-4276 Add Write Ahead Provenance section to User Guide

Included factors and recommendations to consider when migrating from 
Persistent Provenance configuration.

Also corrected default value for nifi.provenance.repository.index.threads 
in Admin Guide.  Default is 2 not 1.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrewmlim/nifi NIFI-4276

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2074.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2074


commit af353ac7872f18f932e10dd1757a9d2146eb011a
Author: Andrew Lim 
Date:   2017-08-11T18:33:18Z

NIFI-4276 Add Write Ahead Provenance section to User Guide




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-4276) Add Provenance Migration section to User Guide

[Andrew Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123804#comment-16123804
 ] 

Andrew Lim commented on NIFI-4276:
--

Will correct error in Admin guide as part of this ticket.  
{{nifi.provenance.repository.index.threads}} property defaults to 2 not 1.

> Add Provenance Migration section to User Guide
> --
>
> Key: NIFI-4276
> URL: https://issues.apache.org/jira/browse/NIFI-4276
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> WriteAheadProvenanceRepository configuration was introduced in version 1.2.0 
> (NIFI-3356).  Should add a section to the documentation that covers how to 
> migrate from default configuration of PersistentProvenantRepository and 
> recommend any system property value changes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-1589) replaceText can't match [

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-1589.
--
Resolution: Cannot Reproduce

Closing as cannot reproduce, attached a template used for testing.

> replaceText can't match [
> -
>
> Key: NIFI-1589
> URL: https://issues.apache.org/jira/browse/NIFI-1589
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Djee
>Priority: Minor
> Attachments: NIFI-1589.xml
>
>
> When trying to match "[", the opening bracket causes an error if not escaped 
> (Not a Java Regular Expression). But, "[" will not match "[" but 
> "[".
> I am not sure if this a bug or working as intended, but there should be a way 
> to regex the opening bracket character.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-1589) replaceText can't match [

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-1589:
-
Attachment: NIFI-1589.xml

> replaceText can't match [
> -
>
> Key: NIFI-1589
> URL: https://issues.apache.org/jira/browse/NIFI-1589
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Djee
>Priority: Minor
> Attachments: NIFI-1589.xml
>
>
> When trying to match "[", the opening bracket causes an error if not escaped 
> (Not a Java Regular Expression). But, "[" will not match "[" but 
> "[".
> I am not sure if this a bug or working as intended, but there should be a way 
> to regex the opening bracket character.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-1567) EvaluateJsonPath processor should work against attributes

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-1567.
--
Resolution: Duplicate

> EvaluateJsonPath processor should work against attributes
> -
>
> Key: NIFI-1567
> URL: https://issues.apache.org/jira/browse/NIFI-1567
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.4.1
>Reporter: Adis Cesir
>Priority: Minor
>
> Consider that certain flowfile contents are not always clean JSON but rather 
> embedded JSON in other sets of data. Currently there is not a god way to 
> extract JSON out of this set of data other than parsing with regex and 
> keeping the existing flow file content.
> Having the ability to parse JSON on already extracted attributes should make 
> processing complex and embedded sets of JSON data much easier.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-1535) usage button under controller service section does not work

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-1535.
--
Resolution: Fixed

Solved with the new UI.

> usage button under controller service section does not work
> ---
>
> Key: NIFI-1535
> URL: https://issues.apache.org/jira/browse/NIFI-1535
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.5.0
>Reporter: Juan Sequeiros
>Priority: Minor
>
> The button that suggests to show me the usage of the controller service as it 
> does through the help docs does not work.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-1530) ExecuteStreamCommand bulletin level selection not working

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-1530.
--
Resolution: Cannot Reproduce

Not sure when it has been solved but I think it was with the log framework 
refactoring for 1.x. Anyway, I cannot reproduce the issue using the provided 
template. Closing.

> ExecuteStreamCommand bulletin level selection not working
> -
>
> Key: NIFI-1530
> URL: https://issues.apache.org/jira/browse/NIFI-1530
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework, Extensions
>Affects Versions: 0.5.0
>Reporter: Aldrin Piri
>Priority: Minor
> Attachments: ExecuteStreamCommand_bulletins.xml
>
>
> In evaluating NIFI-1514, it seems as though the bulletin level does not seem 
> to be working correctly for that processor.  Despite the debug level 
> statements, these particular items never seem to appear in the UI.  Not sure 
> if this is an isolated instance for this particular processor or possibly 
> something to be addressed at the framework level.
> The attached template exhibits the described bug.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-1446) Empty DBConnectionPool select box

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-1446.
--
Resolution: Not A Bug

Closing per comments.

> Empty DBConnectionPool select box
> -
>
> Key: NIFI-1446
> URL: https://issues.apache.org/jira/browse/NIFI-1446
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 0.4.1
>Reporter: Fabien
>
> Hi all
> I'm currently developing a custom Processor based on the ExecuteSQL processor 
> (with a ControllerService of type DBConnectionPool).
> After successfully deploying, I can create a new instance of my processor 
> with the GUI, but in the properties tab, I can not select any of my 
> DBConnectionPool (the select box is empty).
>  - My processor is exactly the same as ExecuteSQL.java
>  - My project pom is :
> {code:xml}
> 
> http://maven.apache.org/POM/4.0.0; 
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
>   4.0.0
>   
>   myparentgroupid
>   mypoarentartifact
>   0.0.1-SNAPSHOT
>   
>   mynifibundle
>   nar
>   
>   0.4.1
>   
>   
>   
>   
>   org.apache.nifi
>   nifi-nar-maven-plugin
>   1.1.0
>   true
>   
>   
>   
>   
>   
>   org.apache.nifi
>   nifi-api
>   ${nifi.version}
>   
>   
>   org.apache.nifi
>   nifi-processor-utils
>   ${nifi.version}
>   
>   
>   org.apache.nifi
>   nifi-dbcp-service-api
>   ${nifi.version}
>   
>   
>   org.apache.nifi
>   nifi-mock
>   ${nifi.version}
>   test
>   
>   
>   com.h2database
>   h2
>   test
>   
>   
> 
> {code}
>  - I used the maven "clean install" goal to create my nar file and deployed 
> this nar into "my_install_NIFI_dir/lib"
> PS : Very nice and promising project, love it



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-4062) Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery

[Arun Manivannan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Manivannan reassigned NIFI-4062:
-

Assignee: Arun Manivannan

> Provide an option to disable DTD validation for EvaluateXPath and 
> EvaluateXQuery
> 
>
> Key: NIFI-4062
> URL: https://issues.apache.org/jira/browse/NIFI-4062
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Arun Manivannan
>  Labels: beginner
>
> Currently EvaluateXPath and EvaluateXQuery can fail if the source document 
> has DOCTYPE definition in it, especially using external DTD. It would be 
> useful if these processor provides a property to disable DTD validation when 
> evaluate XPath or XQuery.
> References:
> https://community.hortonworks.com/questions/107121/does-evaluatexpath-support-doctype.html
> https://stackoverflow.com/questions/243728/how-to-disable-dtd-at-runtime-in-javas-xpath



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-1439) Consistency in pop-up Stats page

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-1439.
--
   Resolution: Fixed
Fix Version/s: 1.2.0

Contained by NIFI-3356.

> Consistency in pop-up Stats page 
> -
>
> Key: NIFI-1439
> URL: https://issues.apache.org/jira/browse/NIFI-1439
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Wynner
>Priority: Minor
>  Labels: beginner, newbiee
> Fix For: 1.2.0
>
>
> In the stats pop-up for a processor, all of the stats have the (5 mins) 
> listed as part of the stats except for Average Task Duration, please make 
> this consistent. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-811) Average Task Duration metric should be more granular

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-811.
-
   Resolution: Fixed
Fix Version/s: 1.2.0

Contained by NIFI-3356.

> Average Task Duration metric should be more granular
> 
>
> Key: NIFI-811
> URL: https://issues.apache.org/jira/browse/NIFI-811
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
> Fix For: 1.2.0
>
>
> Consider rendering this value in nanoseconds or microseconds. When showing 
> the values in milliseconds, a lot of components show 0 milliseconds



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-1399) filter controller services list

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-1399.
--
Resolution: Fixed

Solved with the new UI.

> filter controller services list
> ---
>
> Key: NIFI-1399
> URL: https://issues.apache.org/jira/browse/NIFI-1399
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Brandon DeVries
>Priority: Minor
>
> In the controller services tab of NiFi Flow settings, if the list of 
> controller services is sufficiently long to require a scroll bar, "find" 
> won't work for any services that have scrolled off the page (at least on 
> firefox + windows).  Adding a "filter" capability like is available when 
> adding a service or looking at processor documentation would help in locating 
> the service you're looking for.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-1372) Create ConvertAvroToCSV

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-1372.
--
   Resolution: Fixed
Fix Version/s: 1.2.0

Closing considering the new record-based processors.

> Create ConvertAvroToCSV
> ---
>
> Key: NIFI-1372
> URL: https://issues.apache.org/jira/browse/NIFI-1372
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 0.4.1
>Reporter: Jeremy Dyer
>Priority: Minor
> Fix For: 1.2.0
>
>
> Currently there is no way to get from Avro to CSV. Certain processors like 
> ExecuteSQL emit their results in the Avro format. While you can currently 
> convert from Avro to JSON certain older systems will only accept CSV. Instead 
> of adding the functionality to ExecuteSQL to emit CSV it makes sense to add a 
> new processor that allows creating CSV from Avro in other places as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-1351) Newlines in Processor Name can overrun the box

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-1351.
--
Resolution: Cannot Reproduce

No longer an issue in 1.x line and the new UI.

> Newlines in Processor Name can overrun the box
> --
>
> Key: NIFI-1351
> URL: https://issues.apache.org/jira/browse/NIFI-1351
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Brandon DeVries
>Priority: Minor
>
> If you include newlines in a processor name (e.g. via a copy / paste), you 
> can end up with the text overrunning the boundary of the processor box on the 
> UI.  The processor can also no longer be configured... it has to be deleted 
> from the graph.  An example processor name would be:
> {panel}
> this
> is
> a
> really
> long
> processor
> name
> that
> overruns
> the
> field!
> {panel}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-1341) Enhance documentation surrounding the use of mime.type for POST/InvokeHTTP

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-1341.
--
   Resolution: Fixed
Fix Version/s: 0.5.0

> Enhance documentation surrounding the use of mime.type for POST/InvokeHTTP
> --
>
> Key: NIFI-1341
> URL: https://issues.apache.org/jira/browse/NIFI-1341
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website, Extensions
>Affects Versions: 0.4.1
>Reporter: Aldrin Piri
>Priority: Minor
> Fix For: 0.5.0
>
>
> It is a common case where a user would like to send data to a given endpoint 
> that may have been created outside of NiFi and, accordingly, has some known 
> format or type associated with it that the operator knows given its sourcing. 
>  In these cases, NiFi may not know that a file may be something like XML or 
> JSON and defaults to more generic types like plain text or octet-stream.
> Both InvokeHTTP and PostHTTP make use of the CoreAttribute mime.type for 
> establishing Content-Type on requests where a body is posted to a specified 
> endpoint.  We should make this process more clear from the documentation and 
> consider ways to make this more intuitive for users to accomplish.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-4237) EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest underlying cause

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto reassigned NIFI-4237:
---

Assignee: Andy LoPresto

> EncryptionOperationNotPossibleException in nifi-bootstrap.log might suggest 
> underlying cause
> 
>
> Key: NIFI-4237
> URL: https://issues.apache.org/jira/browse/NIFI-4237
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Russell Bateman
>Assignee: Andy LoPresto
>Priority: Minor
>
> Our Ansible instructions upgraded NiFi and created a new 
> {{nifi.sensitive.props.key}}. In _nifi.properties_ this property, if extant, 
> is used to encrypt sensitive properties in _flow.xml.gz_. Thus, upon 
> relaunching NiFi, the wrong key was used to decrypt resulting in the reported 
> failure to start, _flow.xml.gz_ is no longer useful.
> We found the problem and fixed it after Mark Payne suggested a possible 
> cause, but if this state of things can be determined, it might save on 
> community support for this situation if the logged message were to suggest 
> what's at the bottom of this problem. The top of the stack trace appears in 
> _logs/nifi-bootstrap.log_ as below:
> 2017-07-25 23:23:31,148 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.apache.nifi.encrypt.EncryptionException:
> org.jasypt.exceptions.EncryptionOperationNotPossibleException
> at
> org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at
> org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:474)
> ~[nifi-framework-core-1.1.2.jar:1.1.2]
> at...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-1231) cloned files not getting new entryDate

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-1231.
--
Resolution: Not A Bug

Closing as "not a bug" per comments in the JIRA.

> cloned files not getting new entryDate
> --
>
> Key: NIFI-1231
> URL: https://issues.apache.org/jira/browse/NIFI-1231
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.3.0
>Reporter: Matthew Clarke
>Priority: Minor
>  Labels: beginner, newbie
>
> When FlowFiles are cloned, the clones should get a new entryDate assigned to 
> them. the lineageStartDate should still carry over from the original FlowFile.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-1137) Documentation correction needed (Expression Language Guide - divide function)

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-1137.
--
   Resolution: Fixed
Fix Version/s: 1.1.0

> Documentation correction needed (Expression Language Guide - divide function)
> -
>
> Key: NIFI-1137
> URL: https://issues.apache.org/jira/browse/NIFI-1137
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Reporter: Matthew Clarke
>Priority: Minor
> Fix For: 1.1.0
>
>
> The "Description" is incorrect for the divide function. As currently written, 
> the description implies that the supplied function input (operand) is divided 
> by the subject. The actual behavior is that the subject is divided by the 
> operand.
> The "Operand" needs to have the word "add" removed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-4285) Not referenced controller service not exported in template

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-4285.
--
Resolution: Duplicate

Closing as duplicate, thanks [~mcgilman].

> Not referenced controller service not exported in template
> --
>
> Key: NIFI-4285
> URL: https://issues.apache.org/jira/browse/NIFI-4285
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Pierre Villard
>Priority: Minor
>
> In case there is a controller service that is not directly referenced by a 
> processor, it won't be exported in a template.
> That's the case for the Distributed Map Cache Server for example.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)