[jira] [Commented] (NIFI-4731) BigQuery processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373838#comment-16373838
 ] 

ASF GitHub Bot commented on NIFI-4731:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2420
  
please address the author tags.

On Feb 22, 2018 8:57 PM, "nologic"  wrote:

> cool, all checks have passed. Is there anything else you need need before
> this thing gets scheduled for a merge?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> , or mute
> the thread
> 

> .
>



> BigQuery processors
> ---
>
> Key: NIFI-4731
> URL: https://issues.apache.org/jira/browse/NIFI-4731
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Mikhail Sosonkin
>Priority: Major
>
> NIFI should have processors for putting data into BigQuery (Streaming and 
> Batch).
> Initial working processors can be found this repository: 
> https://github.com/nologic/nifi/tree/NIFI-4731/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery
> I'd like to get them into Nifi proper.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2420: NIFI-4731

[joewitt]

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2420
  
please address the author tags.

On Feb 22, 2018 8:57 PM, "nologic"  wrote:

> cool, all checks have passed. Is there anything else you need need before
> this thing gets scheduled for a merge?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> , or mute
> the thread
> 

> .
>



---

[jira] [Commented] (NIFI-4731) BigQuery processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373810#comment-16373810
 ] 

ASF GitHub Bot commented on NIFI-4731:
--

Github user nologic commented on the issue:

https://github.com/apache/nifi/pull/2420
  
cool, all checks have passed. Is there anything else you need need before 
this thing gets scheduled for a merge?


> BigQuery processors
> ---
>
> Key: NIFI-4731
> URL: https://issues.apache.org/jira/browse/NIFI-4731
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Mikhail Sosonkin
>Priority: Major
>
> NIFI should have processors for putting data into BigQuery (Streaming and 
> Batch).
> Initial working processors can be found this repository: 
> https://github.com/nologic/nifi/tree/NIFI-4731/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery
> I'd like to get them into Nifi proper.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2420: NIFI-4731

[nologic]

Github user nologic commented on the issue:

https://github.com/apache/nifi/pull/2420
  
cool, all checks have passed. Is there anything else you need need before 
this thing gets scheduled for a merge?


---

[jira] [Created] (NIFI-4904) PutElasticSearch5 should support higher than elasticsearch 5.0.0

[Dye357 (JIRA)]

Dye357 created NIFI-4904:


 Summary: PutElasticSearch5 should support higher than 
elasticsearch 5.0.0
 Key: NIFI-4904
 URL: https://issues.apache.org/jira/browse/NIFI-4904
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Affects Versions: 1.5.0
 Environment: Ubuntu
Reporter: Dye357


Currently the PutElasticSearch5 component is using the following transport 
artifact


 org.elasticsearch.client
 transport
 ${es.version}
 

Where es.version is 5.0.1. Upgrading to the highest 5.x dependency would enable 
this component to be compatible with later 5.x versions of elastic search as 
well as early versions of elastic search 6.x.

Here is Nifi 1.5.0 connecting to ES 6.2.1 on port 9300:

[2018-02-23T01:41:04,162][WARN ][o.e.t.n.Netty4Transport ] [uQSW8O8] exception 
caught on transport layer [NettyTcpChannel\{localAddress=/127.0.0.1:9300, 
remoteAddress=/127.0.0.1:57457}], closing connection
java.lang.IllegalStateException: Received message from unsupported version: 
[5.0.0] minimal compatible version is: [5.6.0]
 at 
org.elasticsearch.transport.TcpTransport.ensureVersionCompatibility(TcpTransport.java:1430)
 ~[elasticsearch-6.2.1.jar:6.2.1]
 at 
org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1377)
 ~[elasticsearch-6.2.1.jar:6.2.1]
 at 
org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:64)
 ~[transport-netty4-6.2.1.jar:6.2.1]
 at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 [netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 [netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
 [netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
 [netty-codec-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
 [netty-codec-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
 [netty-codec-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
 [netty-codec-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 [netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 [netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
 [netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) 
[netty-handler-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 [netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 [netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
 [netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
 [netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 [netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 [netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
 [netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
 [netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) 
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545)
 [netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) 
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) 
[netty-transport-4.1.16.Final.jar:4.1.16.Final]
 at 

[jira] [Commented] (NIFI-4616) ConsumeKafka and ConsumeKafka_0_10 can block indefinitely if unable to communicate with Kafka broker that is SSL enabled

[Stephen Barry (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373577#comment-16373577
 ] 

Stephen Barry commented on NIFI-4616:
-

I am facing this issue with SSL secured Kafka cluster. Reproduced the behaviour 
on NiFi 1.5.0 with ConsumeKafka_1_0 as well

> ConsumeKafka and ConsumeKafka_0_10 can block indefinitely if unable to 
> communicate with Kafka broker that is SSL enabled
> 
>
> Key: NIFI-4616
> URL: https://issues.apache.org/jira/browse/NIFI-4616
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.4.0
>Reporter: Aldrin Piri
>Priority: Major
> Fix For: 1.1.0
>
>
> If I use ConsumeKafka and point to a broker that is in a bad state, I see 
> ConsumeKafka block indefinitely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373446#comment-16373446
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2101
  
The L LGTM :) the MIT license reference goes in the assembly LICENSE and 
the NAR license (which it is), and nothing in the NOTICE unless its transitive 
dependencies have NOTICEs, I verified only Apache Commons Lang does, and it is 
included. 


> Implement put processor for InfluxDB
> 
>
> Key: NIFI-4289
> URL: https://issues.apache.org/jira/browse/NIFI-4289
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.3.0
> Environment: All
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
>  Labels: insert, measurements,, put, timeseries
> Fix For: 1.6.0
>
>
> Support inserting time series measurements into InfluxDB.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2101: NIFI-4289 - InfluxDB put processor

[mattyb149]

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2101
  
The L LGTM :) the MIT license reference goes in the assembly LICENSE and 
the NAR license (which it is), and nothing in the NOTICE unless its transitive 
dependencies have NOTICEs, I verified only Apache Commons Lang does, and it is 
included. 


---

[jira] [Commented] (NIFI-4774) FlowFile Repository should write updates to the same FlowFile to the same partition

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373414#comment-16373414
 ] 

ASF GitHub Bot commented on NIFI-4774:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2487#discussion_r170084710
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java
 ---
@@ -129,17 +138,22 @@ public WriteAheadFlowFileRepository() {
 checkpointDelayMillis = 0l;
 numPartitions = 0;
 checkpointExecutor = null;
-flowFileRepositoryPath = null;
+walImplementation = null;
--- End diff --

Should this be DEFAULT_WAL_IMPLEMENTATION?


> FlowFile Repository should write updates to the same FlowFile to the same 
> partition
> ---
>
> Key: NIFI-4774
> URL: https://issues.apache.org/jira/browse/NIFI-4774
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.6.0
>
>
> As-is, in the case of power loss or Operating System crash, we could have an 
> update that is lost, and then an update for the same FlowFile that is not 
> lost, because the updates for a given FlowFile can span partitions. If an 
> update were written to Partition 1 and then to Partition 2 and Partition 2 is 
> flushed to disk by the Operating System and then the Operating System crashes 
> or power is lost before Partition 1 is flushed to disk, we could lose the 
> update to Partition 1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4774) FlowFile Repository should write updates to the same FlowFile to the same partition

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373415#comment-16373415
 ] 

ASF GitHub Bot commented on NIFI-4774:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2487#discussion_r170084644
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java
 ---
@@ -80,6 +80,15 @@
  */
 public class WriteAheadFlowFileRepository implements FlowFileRepository, 
SyncListener {
 private static final String FLOWFILE_REPOSITORY_DIRECTORY_PREFIX = 
"nifi.flowfile.repository.directory";
+private static final String WRITE_AHEAD_LOG_IMPL = 
"nifi.flowfile.repository.wal.implementation";
+
+// TODO: Update Admin Guide
--- End diff --

I think this TODO is DONE :)


> FlowFile Repository should write updates to the same FlowFile to the same 
> partition
> ---
>
> Key: NIFI-4774
> URL: https://issues.apache.org/jira/browse/NIFI-4774
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.6.0
>
>
> As-is, in the case of power loss or Operating System crash, we could have an 
> update that is lost, and then an update for the same FlowFile that is not 
> lost, because the updates for a given FlowFile can span partitions. If an 
> update were written to Partition 1 and then to Partition 2 and Partition 2 is 
> flushed to disk by the Operating System and then the Operating System crashes 
> or power is lost before Partition 1 is flushed to disk, we could lose the 
> update to Partition 1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2487: NIFI-4774: Allow user to choose which write-ahead l...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2487#discussion_r170084644
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java
 ---
@@ -80,6 +80,15 @@
  */
 public class WriteAheadFlowFileRepository implements FlowFileRepository, 
SyncListener {
 private static final String FLOWFILE_REPOSITORY_DIRECTORY_PREFIX = 
"nifi.flowfile.repository.directory";
+private static final String WRITE_AHEAD_LOG_IMPL = 
"nifi.flowfile.repository.wal.implementation";
+
+// TODO: Update Admin Guide
--- End diff --

I think this TODO is DONE :)


---

[GitHub] nifi pull request #2487: NIFI-4774: Allow user to choose which write-ahead l...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2487#discussion_r170084710
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java
 ---
@@ -129,17 +138,22 @@ public WriteAheadFlowFileRepository() {
 checkpointDelayMillis = 0l;
 numPartitions = 0;
 checkpointExecutor = null;
-flowFileRepositoryPath = null;
+walImplementation = null;
--- End diff --

Should this be DEFAULT_WAL_IMPLEMENTATION?


---

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373413#comment-16373413
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2101
  
It looks OK to me, I'd just suggest allowing EL on username/password. Will 
probably be useful with the Registry stuff. I know sensitive properties are not 
handled by the registry work yet, but that could be in the future.

Before giving a final +1, I'd appreciate another opinion on L aspects but 
it LGTM. Thanks for all your work @mans2singh 


> Implement put processor for InfluxDB
> 
>
> Key: NIFI-4289
> URL: https://issues.apache.org/jira/browse/NIFI-4289
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.3.0
> Environment: All
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
>  Labels: insert, measurements,, put, timeseries
> Fix For: 1.6.0
>
>
> Support inserting time series measurements into InfluxDB.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2101: NIFI-4289 - InfluxDB put processor

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2101
  
It looks OK to me, I'd just suggest allowing EL on username/password. Will 
probably be useful with the Registry stuff. I know sensitive properties are not 
handled by the registry work yet, but that could be in the future.

Before giving a final +1, I'd appreciate another opinion on L aspects but 
it LGTM. Thanks for all your work @mans2singh 


---

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373375#comment-16373375
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mans2singh commented on the issue:

https://github.com/apache/nifi/pull/2101
  
@pvillard31 - 

You were right - After renaming the integration tests they don't get 
executed by default mvn install.  Also removed test profile from pom.xml as 
recommended.

Please let me know if you have any additional recommendation.

Thanks again.


> Implement put processor for InfluxDB
> 
>
> Key: NIFI-4289
> URL: https://issues.apache.org/jira/browse/NIFI-4289
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.3.0
> Environment: All
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
>  Labels: insert, measurements,, put, timeseries
> Fix For: 1.6.0
>
>
> Support inserting time series measurements into InfluxDB.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2101: NIFI-4289 - InfluxDB put processor

[mans2singh]

Github user mans2singh commented on the issue:

https://github.com/apache/nifi/pull/2101
  
@pvillard31 - 

You were right - After renaming the integration tests they don't get 
executed by default mvn install.  Also removed test profile from pom.xml as 
recommended.

Please let me know if you have any additional recommendation.

Thanks again.


---

[jira] [Commented] (NIFI-4903) Database Fetch processors cannot handle timestamp fields with Oracle 12+

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373344#comment-16373344
 ] 

ASF GitHub Bot commented on NIFI-4903:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2488


> Database Fetch processors cannot handle timestamp fields with Oracle 12+
> 
>
> Key: NIFI-4903
> URL: https://issues.apache.org/jira/browse/NIFI-4903
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.6.0
>
>
> When using GenerateTableFetch or QueryDatabaseTable with an Oracle 12+ 
> adapter, if the maximum value column is set to a column of type timestamp, 
> then upon the second fetch the following error occurs:
> java.sql.SQLDataException: ORA-01843: not a valid month
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4903) Database Fetch processors cannot handle timestamp fields with Oracle 12+

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4903:
-
   Resolution: Fixed
Fix Version/s: 1.6.0
   Status: Resolved  (was: Patch Available)

> Database Fetch processors cannot handle timestamp fields with Oracle 12+
> 
>
> Key: NIFI-4903
> URL: https://issues.apache.org/jira/browse/NIFI-4903
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.6.0
>
>
> When using GenerateTableFetch or QueryDatabaseTable with an Oracle 12+ 
> adapter, if the maximum value column is set to a column of type timestamp, 
> then upon the second fetch the following error occurs:
> java.sql.SQLDataException: ORA-01843: not a valid month
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2488: NIFI-4903: Fixed timestamp bug with fetch processor...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2488


---

[jira] [Commented] (NIFI-4903) Database Fetch processors cannot handle timestamp fields with Oracle 12+

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373343#comment-16373343
 ] 

ASF subversion and git services commented on NIFI-4903:
---

Commit 62732cbb8827d0e41dde90af325775a727501f45 in nifi's branch 
refs/heads/master from [~ca9mbu]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=62732cb ]

NIFI-4903 - Fixed timestamp bug with fetch processors using Oracle 12+

Signed-off-by: Pierre Villard 

This closes #2488.


> Database Fetch processors cannot handle timestamp fields with Oracle 12+
> 
>
> Key: NIFI-4903
> URL: https://issues.apache.org/jira/browse/NIFI-4903
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> When using GenerateTableFetch or QueryDatabaseTable with an Oracle 12+ 
> adapter, if the maximum value column is set to a column of type timestamp, 
> then upon the second fetch the following error occurs:
> java.sql.SQLDataException: ORA-01843: not a valid month
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4903) Database Fetch processors cannot handle timestamp fields with Oracle 12+

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373342#comment-16373342
 ] 

ASF GitHub Bot commented on NIFI-4903:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2488
  
+1, merging to master, thanks @mattyb149 


> Database Fetch processors cannot handle timestamp fields with Oracle 12+
> 
>
> Key: NIFI-4903
> URL: https://issues.apache.org/jira/browse/NIFI-4903
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> When using GenerateTableFetch or QueryDatabaseTable with an Oracle 12+ 
> adapter, if the maximum value column is set to a column of type timestamp, 
> then upon the second fetch the following error occurs:
> java.sql.SQLDataException: ORA-01843: not a valid month
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2488: NIFI-4903: Fixed timestamp bug with fetch processors using...

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2488
  
+1, merging to master, thanks @mattyb149 


---

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373321#comment-16373321
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170070317
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -226,6 +242,8 @@ private ObjectWriter getObjectWriter(ObjectMapper 
mapper, String ppSetting) {
 
 @Override
 public void onTrigger(final ProcessContext context, final 
ProcessSession session) throws ProcessException {
+FlowFile input = session.get();
--- End diff --

It's not checking for input == null here, that would be one of the 
requirements for the "flowfile-driven behavior" rather than the "timer-driven 
behavior". It might cause an NPE at line 266, and otherwise if null you may be 
treating it as flowfile-driven when really it is more like timer-driven. 
ExecuteSQL has a good example of how to handle this.


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373320#comment-16373320
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170070552
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -236,12 +254,33 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 
context.getProperty(QUERY).evaluateAttributeExpressions().getValue());
 }
 
-final Document query = context.getProperty(QUERY).isSet()
-? 
Document.parse(context.getProperty(QUERY).evaluateAttributeExpressions().getValue())
 : null;
+final Document query;
+if (!context.hasIncomingConnection() && 
!context.getProperty(QUERY).isSet()) {
+query = Document.parse("{}");
--- End diff --

I thought this was going to be a validation error? It can be done in 
OnScheduled, see ExecuteSQL for an example. Otherwise how would the user know 
that his/her configuration won't actually perform any work?


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373322#comment-16373322
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170070960
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoTest.java
 ---
@@ -69,6 +70,7 @@
 @Before
 public void setup() {
 runner = TestRunners.newTestRunner(GetMongo.class);
+//runner.setProperty(GetMongo.QUERY, "{}");
--- End diff --

This can now be removed


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170070317
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -226,6 +242,8 @@ private ObjectWriter getObjectWriter(ObjectMapper 
mapper, String ppSetting) {
 
 @Override
 public void onTrigger(final ProcessContext context, final 
ProcessSession session) throws ProcessException {
+FlowFile input = session.get();
--- End diff --

It's not checking for input == null here, that would be one of the 
requirements for the "flowfile-driven behavior" rather than the "timer-driven 
behavior". It might cause an NPE at line 266, and otherwise if null you may be 
treating it as flowfile-driven when really it is more like timer-driven. 
ExecuteSQL has a good example of how to handle this.


---

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170070552
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -236,12 +254,33 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 
context.getProperty(QUERY).evaluateAttributeExpressions().getValue());
 }
 
-final Document query = context.getProperty(QUERY).isSet()
-? 
Document.parse(context.getProperty(QUERY).evaluateAttributeExpressions().getValue())
 : null;
+final Document query;
+if (!context.hasIncomingConnection() && 
!context.getProperty(QUERY).isSet()) {
+query = Document.parse("{}");
--- End diff --

I thought this was going to be a validation error? It can be done in 
OnScheduled, see ExecuteSQL for an example. Otherwise how would the user know 
that his/her configuration won't actually perform any work?


---

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170070960
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoTest.java
 ---
@@ -69,6 +70,7 @@
 @Before
 public void setup() {
 runner = TestRunners.newTestRunner(GetMongo.class);
+//runner.setProperty(GetMongo.QUERY, "{}");
--- End diff --

This can now be removed


---

[jira] [Commented] (MINIFICPP-412) Publish MiNiFi Docs to website

[Aldrin Piri (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373290#comment-16373290
 ] 

Aldrin Piri commented on MINIFICPP-412:
---

Should also capture updating these, if published, in our release process.

> Publish MiNiFi Docs to website
> --
>
> Key: MINIFICPP-412
> URL: https://issues.apache.org/jira/browse/MINIFICPP-412
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Aldrin Piri
>Priority: Major
>
> Items such as Processors, Extensions, and Expression language docs are 
> currently only available with the source.  We should also get these at least 
> linked if not published to the MiNiFi site for easier consumption.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (MINIFICPP-412) Publish MiNiFi Docs to website

[Aldrin Piri (JIRA)]

Aldrin Piri created MINIFICPP-412:
-

 Summary: Publish MiNiFi Docs to website
 Key: MINIFICPP-412
 URL: https://issues.apache.org/jira/browse/MINIFICPP-412
 Project: NiFi MiNiFi C++
  Issue Type: Improvement
Reporter: Aldrin Piri


Items such as Processors, Extensions, and Expression language docs are 
currently only available with the source.  We should also get these at least 
linked if not published to the MiNiFi site for easier consumption.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373265#comment-16373265
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2443
  
@mattyb149 Ready for review.


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2443: NIFI-4827 Added support for reading queries from the flowf...

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2443
  
@mattyb149 Ready for review.


---

[jira] [Updated] (NIFI-4903) Database Fetch processors cannot handle timestamp fields with Oracle 12+

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-4903:
---
Status: Patch Available  (was: In Progress)

> Database Fetch processors cannot handle timestamp fields with Oracle 12+
> 
>
> Key: NIFI-4903
> URL: https://issues.apache.org/jira/browse/NIFI-4903
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> When using GenerateTableFetch or QueryDatabaseTable with an Oracle 12+ 
> adapter, if the maximum value column is set to a column of type timestamp, 
> then upon the second fetch the following error occurs:
> java.sql.SQLDataException: ORA-01843: not a valid month
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4903) Database Fetch processors cannot handle timestamp fields with Oracle 12+

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373249#comment-16373249
 ] 

ASF GitHub Bot commented on NIFI-4903:
--

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/2488

NIFI-4903: Fixed timestamp bug with fetch processors using Oracle 12+

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-4903

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2488.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2488


commit ad6130da0eb4caa81d5fbb67c6a5dc09264d0392
Author: Matthew Burgess 
Date:   2018-02-22T18:50:08Z

NIFI-4903: Fixed timestamp bug with fetch processors using Oracle 12+




> Database Fetch processors cannot handle timestamp fields with Oracle 12+
> 
>
> Key: NIFI-4903
> URL: https://issues.apache.org/jira/browse/NIFI-4903
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> When using GenerateTableFetch or QueryDatabaseTable with an Oracle 12+ 
> adapter, if the maximum value column is set to a column of type timestamp, 
> then upon the second fetch the following error occurs:
> java.sql.SQLDataException: ORA-01843: not a valid month
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2488: NIFI-4903: Fixed timestamp bug with fetch processor...

[mattyb149]

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/2488

NIFI-4903: Fixed timestamp bug with fetch processors using Oracle 12+

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-4903

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2488.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2488


commit ad6130da0eb4caa81d5fbb67c6a5dc09264d0392
Author: Matthew Burgess 
Date:   2018-02-22T18:50:08Z

NIFI-4903: Fixed timestamp bug with fetch processors using Oracle 12+




---

[jira] [Assigned] (NIFI-4903) Database Fetch processors cannot handle timestamp fields with Oracle 12+

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess reassigned NIFI-4903:
--

Assignee: Matt Burgess

> Database Fetch processors cannot handle timestamp fields with Oracle 12+
> 
>
> Key: NIFI-4903
> URL: https://issues.apache.org/jira/browse/NIFI-4903
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> When using GenerateTableFetch or QueryDatabaseTable with an Oracle 12+ 
> adapter, if the maximum value column is set to a column of type timestamp, 
> then upon the second fetch the following error occurs:
> java.sql.SQLDataException: ORA-01843: not a valid month
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4903) Database Fetch processors cannot handle timestamp fields with Oracle 12+

[Matt Burgess (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373238#comment-16373238
 ] 

Matt Burgess commented on NIFI-4903:


This is a logic error in AbstractDatabaseFetchProcessor which checks to see if 
the database adapter name is equal to "Oracle". Instead it should check to see 
if the database adapter name contains the string "Oracle", which will allow it 
to work for the Oracle 12+ adapter (whose name is "Oracle 12+") and any future 
Oracle-related adapters. Note I'm calling for "contains" rather than 
"startsWith" to give more flexibility for adapter names (such as "My Special 
Oracle Adapter", e.g.).

> Database Fetch processors cannot handle timestamp fields with Oracle 12+
> 
>
> Key: NIFI-4903
> URL: https://issues.apache.org/jira/browse/NIFI-4903
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Priority: Major
>
> When using GenerateTableFetch or QueryDatabaseTable with an Oracle 12+ 
> adapter, if the maximum value column is set to a column of type timestamp, 
> then upon the second fetch the following error occurs:
> java.sql.SQLDataException: ORA-01843: not a valid month
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-4903) Database Fetch processors cannot handle timestamp fields with Oracle 12+

[Matt Burgess (JIRA)]

Matt Burgess created NIFI-4903:
--

 Summary: Database Fetch processors cannot handle timestamp fields 
with Oracle 12+
 Key: NIFI-4903
 URL: https://issues.apache.org/jira/browse/NIFI-4903
 Project: Apache NiFi
  Issue Type: Bug
  Components: Extensions
Reporter: Matt Burgess


When using GenerateTableFetch or QueryDatabaseTable with an Oracle 12+ adapter, 
if the maximum value column is set to a column of type timestamp, then upon the 
second fetch the following error occurs:

java.sql.SQLDataException: ORA-01843: not a valid month

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-2416) Add OPTION Clause to database fetch processors (QueryDatabaseTable, GenerateTableFetch)

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-2416:
---
Summary: Add OPTION Clause to database fetch processors 
(QueryDatabaseTable, GenerateTableFetch)  (was: Add OPTION Clause to 
QueryDatabaseTable (and soon, the GenerateTableFetch) Nifi Processor)

> Add OPTION Clause to database fetch processors (QueryDatabaseTable, 
> GenerateTableFetch)
> ---
>
> Key: NIFI-2416
> URL: https://issues.apache.org/jira/browse/NIFI-2416
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: younes kafi
>Priority: Major
>
> Enable the possibility to add an Option Clause like 'OPTION(hash join)' while 
> performing an incremental import using QueryDatabaseTable processor.
> An option clause can improve highly a mssql query and it would be great to be 
> supported by Nifi. 
> I think this is a mssql specific option, so maybe taking into consideration 
> whether the various drivers (Oracle, MySQL, Postgres, etc.) support this 
> clause or its equivalent would be a real plus.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4898) Remote Process Group in a SSL setup generates Java Exception

[Pierre Villard (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373216#comment-16373216
 ] 

Pierre Villard commented on NIFI-4898:
--

We upgraded some libraries between 1.4.0 and 1.5.0 and if there are common 
libraries used in your JARs (but with the previous version we used) this could 
cause conflicts leading to the error you reported. In your case, this is 
probably linked to:

org.apache.httpcomponents / httpclient from 4.4.1 to 4.5.3

org.apache.httpcomponents / httpcore 4.4.4

If you can, it's probably best to include your jars for custom processors in 
the NAR instead of using the default /lib directory. This way you can leverage 
classloading isolation if needed.

> Remote Process Group in a SSL setup generates Java Exception
> 
>
> Key: NIFI-4898
> URL: https://issues.apache.org/jira/browse/NIFI-4898
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.5.0
> Environment: NiFi Version 1.5.0
> Java 1.8.0_161-b12
> CentOS Linux release 7.4.1708
>Reporter: Josef Zahner
>Priority: Major
>
> In a SSL secured NiFi setup, doesn't matter whether it is a cluster or not, 
> NiFi creates a Java exception as soon as I create a "Remote Process Group". 
> It doesn't mater which URL (even one which doesn't exists) I insert or if I 
> choose RAW or HTTP, the error is always the same and occurs as soon as I 
> click "add" on the "Remote Process Group".
> On NiFi 1.4.0 this works without any issues.
> Error:
> {code:java}
> 2018-02-21 10:42:10,006 ERROR [Remote Process Group 
> b7bde0cc-0161-1000-2e7f-3167a78d8386 Thread-1] 
> org.apache.nifi.engine.FlowEngine A flow controller task execution stopped 
> abnormally
> java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: 
> org.apache.http.impl.client.HttpClientBuilder.setSSLContext(Ljavax/net/ssl/SSLContext;)Lorg/apache/http/impl/client/HttpClientBuilder;
> at java.util.concurrent.FutureTask.report(Unknown Source)
> at java.util.concurrent.FutureTask.get(Unknown Source)
> at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.NoSuchMethodError: 
> org.apache.http.impl.client.HttpClientBuilder.setSSLContext(Ljavax/net/ssl/SSLContext;)Lorg/apache/http/impl/client/HttpClientBuilder;
> at 
> org.apache.nifi.remote.util.SiteToSiteRestApiClient.setupClient(SiteToSiteRestApiClient.java:278)
> at 
> org.apache.nifi.remote.util.SiteToSiteRestApiClient.getHttpClient(SiteToSiteRestApiClient.java:219)
> at 
> org.apache.nifi.remote.util.SiteToSiteRestApiClient.execute(SiteToSiteRestApiClient.java:1189)
> at 
> org.apache.nifi.remote.util.SiteToSiteRestApiClient.execute(SiteToSiteRestApiClient.java:1237)
> at 
> org.apache.nifi.remote.util.SiteToSiteRestApiClient.fetchController(SiteToSiteRestApiClient.java:419)
> at 
> org.apache.nifi.remote.util.SiteToSiteRestApiClient.getController(SiteToSiteRestApiClient.java:394)
> at 
> org.apache.nifi.remote.util.SiteToSiteRestApiClient.getController(SiteToSiteRestApiClient.java:361)
> at 
> org.apache.nifi.remote.util.SiteToSiteRestApiClient.getController(SiteToSiteRestApiClient.java:346)
> at 
> org.apache.nifi.remote.StandardRemoteProcessGroup.refreshFlowContents(StandardRemoteProcessGroup.java:842)
> at 
> org.apache.nifi.remote.StandardRemoteProcessGroup.lambda$initialize$0(StandardRemoteProcessGroup.java:193)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
>  Source)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>  Source)
> ... 3 common frames omitted
> 2018-02-21 10:42:10,009 ERROR [Remote Process Group 
> b7bde0cc-0161-1000-2e7f-3167a78d8386 Thread-1] 
> org.apache.nifi.engine.FlowEngine A flow controller task execution stopped 
> abnormally
> java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: 
> org.apache.http.impl.client.HttpClientBuilder.setSSLContext(Ljavax/net/ssl/SSLContext;)Lorg/apache/http/impl/client/HttpClientBuilder;
> at java.util.concurrent.FutureTask.report(Unknown Source)
> at java.util.concurrent.FutureTask.get(Unknown Source)
> at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.NoSuchMethodError: 
> 

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373196#comment-16373196
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170044639
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoTest.java
 ---
@@ -296,4 +298,55 @@ public void testQueryAttribute() {
 Assert.assertEquals("Value was wrong", val, "{}");
 }
 }
+public void testQueryLocationConfig() {
+//Test EL
+Map attributes = new HashMap();
+attributes.put("field", "c");
+attributes.put("value", "4");
+String query = "{ \"${field}\": { \"$gte\": ${value}}}";
+runner.setProperty(GetMongo.QUERY, query);
+runner.setProperty(GetMongo.RESULTS_PER_FLOWFILE, "10");
+runner.setValidateExpressionUsage(true);
+runner.enqueue("test", attributes);
+runner.run(1, true, true);
+
+runner.assertTransferCount(GetMongo.REL_FAILURE, 0);
+runner.assertTransferCount(GetMongo.REL_ORIGINAL, 1);
+runner.assertTransferCount(GetMongo.REL_SUCCESS, 1);
+
+runner.clearTransferState();
+
+query = "{ \"c\": { \"$gte\": 4 }}";
+runner.setProperty(GetMongo.QUERY, query);
+runner.run(1, true, true);
+runner.assertTransferCount(GetMongo.REL_FAILURE, 0);
+runner.assertTransferCount(GetMongo.REL_ORIGINAL, 0);
+runner.assertTransferCount(GetMongo.REL_SUCCESS, 1);
+
+runner.clearTransferState();
+
+runner.removeProperty(GetMongo.QUERY);
+runner.enqueue(query);
+runner.run(1, true, true);
+
+runner.assertTransferCount(GetMongo.REL_FAILURE, 0);
+runner.assertTransferCount(GetMongo.REL_ORIGINAL, 1);
+runner.assertTransferCount(GetMongo.REL_SUCCESS, 1);
+
+runner.clearTransferState();
+
+boolean exception = false;
+try {
+runner.assertValid();
+runner.setIncomingConnection(false);
+runner.run(1, true, true);
+} catch (Throwable pe) {
+exception = true;
+}
+
+Assert.assertTrue("No exception was thrown!", exception);
--- End diff --

Done


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170044639
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoTest.java
 ---
@@ -296,4 +298,55 @@ public void testQueryAttribute() {
 Assert.assertEquals("Value was wrong", val, "{}");
 }
 }
+public void testQueryLocationConfig() {
+//Test EL
+Map attributes = new HashMap();
+attributes.put("field", "c");
+attributes.put("value", "4");
+String query = "{ \"${field}\": { \"$gte\": ${value}}}";
+runner.setProperty(GetMongo.QUERY, query);
+runner.setProperty(GetMongo.RESULTS_PER_FLOWFILE, "10");
+runner.setValidateExpressionUsage(true);
+runner.enqueue("test", attributes);
+runner.run(1, true, true);
+
+runner.assertTransferCount(GetMongo.REL_FAILURE, 0);
+runner.assertTransferCount(GetMongo.REL_ORIGINAL, 1);
+runner.assertTransferCount(GetMongo.REL_SUCCESS, 1);
+
+runner.clearTransferState();
+
+query = "{ \"c\": { \"$gte\": 4 }}";
+runner.setProperty(GetMongo.QUERY, query);
+runner.run(1, true, true);
+runner.assertTransferCount(GetMongo.REL_FAILURE, 0);
+runner.assertTransferCount(GetMongo.REL_ORIGINAL, 0);
+runner.assertTransferCount(GetMongo.REL_SUCCESS, 1);
+
+runner.clearTransferState();
+
+runner.removeProperty(GetMongo.QUERY);
+runner.enqueue(query);
+runner.run(1, true, true);
+
+runner.assertTransferCount(GetMongo.REL_FAILURE, 0);
+runner.assertTransferCount(GetMongo.REL_ORIGINAL, 1);
+runner.assertTransferCount(GetMongo.REL_SUCCESS, 1);
+
+runner.clearTransferState();
+
+boolean exception = false;
+try {
+runner.assertValid();
+runner.setIncomingConnection(false);
+runner.run(1, true, true);
+} catch (Throwable pe) {
+exception = true;
+}
+
+Assert.assertTrue("No exception was thrown!", exception);
--- End diff --

Done


---

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373190#comment-16373190
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170043742
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoTest.java
 ---
@@ -69,6 +70,7 @@
 @Before
 public void setup() {
 runner = TestRunners.newTestRunner(GetMongo.class);
+runner.setProperty(GetMongo.QUERY, "{}");
--- End diff --

I don't think you have access to connection information during validation 
(all you get is a ValidationContext), I was looking into adding it at one 
point, but never got around to it.


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170043742
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoTest.java
 ---
@@ -69,6 +70,7 @@
 @Before
 public void setup() {
 runner = TestRunners.newTestRunner(GetMongo.class);
+runner.setProperty(GetMongo.QUERY, "{}");
--- End diff --

I don't think you have access to connection information during validation 
(all you get is a ValidationContext), I was looking into adding it at one 
point, but never got around to it.


---

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373189#comment-16373189
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170043364
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -226,6 +242,11 @@ private ObjectWriter getObjectWriter(ObjectMapper 
mapper, String ppSetting) {
 
 @Override
 public void onTrigger(final ProcessContext context, final 
ProcessSession session) throws ProcessException {
+FlowFile input = session.get();
+if (!context.hasIncomingConnection() && 
(context.getProperty(QUERY) == null)) {
+throw new RuntimeException("Without an incoming connection, 
the Query property must be set.");
--- End diff --

I just realized that this block to check for an incoming connection and the 
Query property will get executed on each onTrigger. I believe it should be done 
during setup (with an OnScheduled method), check ExecuteSQL for an example. 
Otherwise this processor can generate an exception on each execution, and if 
the Run Schedule is zero seconds...


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170043364
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -226,6 +242,11 @@ private ObjectWriter getObjectWriter(ObjectMapper 
mapper, String ppSetting) {
 
 @Override
 public void onTrigger(final ProcessContext context, final 
ProcessSession session) throws ProcessException {
+FlowFile input = session.get();
+if (!context.hasIncomingConnection() && 
(context.getProperty(QUERY) == null)) {
+throw new RuntimeException("Without an incoming connection, 
the Query property must be set.");
--- End diff --

I just realized that this block to check for an incoming connection and the 
Query property will get executed on each onTrigger. I believe it should be done 
during setup (with an OnScheduled method), check ExecuteSQL for an example. 
Otherwise this processor can generate an exception on each execution, and if 
the Run Schedule is zero seconds...


---

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373177#comment-16373177
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170042887
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoTest.java
 ---
@@ -69,6 +70,7 @@
 @Before
 public void setup() {
 runner = TestRunners.newTestRunner(GetMongo.class);
+runner.setProperty(GetMongo.QUERY, "{}");
--- End diff --

Ok. What I'm going to do there is make it so that empty query is only valid 
when there is no incoming connection.


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170042887
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoTest.java
 ---
@@ -69,6 +70,7 @@
 @Before
 public void setup() {
 runner = TestRunners.newTestRunner(GetMongo.class);
+runner.setProperty(GetMongo.QUERY, "{}");
--- End diff --

Ok. What I'm going to do there is make it so that empty query is only valid 
when there is no incoming connection.


---

[jira] [Commented] (NIFI-4774) FlowFile Repository should write updates to the same FlowFile to the same partition

[Mark Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373163#comment-16373163
 ] 

Mark Payne commented on NIFI-4774:
--

I have submitted a new PR that allows the user to configure which 
implementation of the write-ahead log to use.

> FlowFile Repository should write updates to the same FlowFile to the same 
> partition
> ---
>
> Key: NIFI-4774
> URL: https://issues.apache.org/jira/browse/NIFI-4774
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.6.0
>
>
> As-is, in the case of power loss or Operating System crash, we could have an 
> update that is lost, and then an update for the same FlowFile that is not 
> lost, because the updates for a given FlowFile can span partitions. If an 
> update were written to Partition 1 and then to Partition 2 and Partition 2 is 
> flushed to disk by the Operating System and then the Operating System crashes 
> or power is lost before Partition 1 is flushed to disk, we could lose the 
> update to Partition 1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4774) FlowFile Repository should write updates to the same FlowFile to the same partition

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373160#comment-16373160
 ] 

ASF GitHub Bot commented on NIFI-4774:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2487

NIFI-4774: Allow user to choose which write-ahead log implementation …

…should be used by the WriteAheadFlowFileRepository

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4774-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2487.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2487


commit 07c1b1140178b83d69f31a4ec99e8a23d134d962
Author: Mark Payne 
Date:   2018-02-22T17:51:38Z

NIFI-4774: Allow user to choose which write-ahead log implementation should 
be used by the WriteAheadFlowFileRepository




> FlowFile Repository should write updates to the same FlowFile to the same 
> partition
> ---
>
> Key: NIFI-4774
> URL: https://issues.apache.org/jira/browse/NIFI-4774
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.6.0
>
>
> As-is, in the case of power loss or Operating System crash, we could have an 
> update that is lost, and then an update for the same FlowFile that is not 
> lost, because the updates for a given FlowFile can span partitions. If an 
> update were written to Partition 1 and then to Partition 2 and Partition 2 is 
> flushed to disk by the Operating System and then the Operating System crashes 
> or power is lost before Partition 1 is flushed to disk, we could lose the 
> update to Partition 1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2487: NIFI-4774: Allow user to choose which write-ahead l...

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2487

NIFI-4774: Allow user to choose which write-ahead log implementation …

…should be used by the WriteAheadFlowFileRepository

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4774-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2487.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2487


commit 07c1b1140178b83d69f31a4ec99e8a23d134d962
Author: Mark Payne 
Date:   2018-02-22T17:51:38Z

NIFI-4774: Allow user to choose which write-ahead log implementation should 
be used by the WriteAheadFlowFileRepository




---

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373148#comment-16373148
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170039382
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -226,6 +242,11 @@ private ObjectWriter getObjectWriter(ObjectMapper 
mapper, String ppSetting) {
 
 @Override
 public void onTrigger(final ProcessContext context, final 
ProcessSession session) throws ProcessException {
+FlowFile input = session.get();
+if (!context.hasIncomingConnection() && 
(context.getProperty(QUERY) == null)) {
+throw new RuntimeException("Without an incoming connection, 
the Query property must be set.");
--- End diff --

Done.


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4827) Make GetMongo able to use flowfiles for queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373120#comment-16373120
 ] 

ASF GitHub Bot commented on NIFI-4827:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170034475
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoTest.java
 ---
@@ -296,4 +298,55 @@ public void testQueryAttribute() {
 Assert.assertEquals("Value was wrong", val, "{}");
 }
 }
+public void testQueryLocationConfig() {
+//Test EL
+Map attributes = new HashMap();
+attributes.put("field", "c");
+attributes.put("value", "4");
+String query = "{ \"${field}\": { \"$gte\": ${value}}}";
+runner.setProperty(GetMongo.QUERY, query);
+runner.setProperty(GetMongo.RESULTS_PER_FLOWFILE, "10");
+runner.setValidateExpressionUsage(true);
+runner.enqueue("test", attributes);
+runner.run(1, true, true);
+
+runner.assertTransferCount(GetMongo.REL_FAILURE, 0);
+runner.assertTransferCount(GetMongo.REL_ORIGINAL, 1);
+runner.assertTransferCount(GetMongo.REL_SUCCESS, 1);
+
+runner.clearTransferState();
+
+query = "{ \"c\": { \"$gte\": 4 }}";
+runner.setProperty(GetMongo.QUERY, query);
+runner.run(1, true, true);
+runner.assertTransferCount(GetMongo.REL_FAILURE, 0);
+runner.assertTransferCount(GetMongo.REL_ORIGINAL, 0);
+runner.assertTransferCount(GetMongo.REL_SUCCESS, 1);
+
+runner.clearTransferState();
+
+runner.removeProperty(GetMongo.QUERY);
+runner.enqueue(query);
+runner.run(1, true, true);
+
+runner.assertTransferCount(GetMongo.REL_FAILURE, 0);
+runner.assertTransferCount(GetMongo.REL_ORIGINAL, 1);
+runner.assertTransferCount(GetMongo.REL_SUCCESS, 1);
+
+runner.clearTransferState();
+
+boolean exception = false;
+try {
+runner.assertValid();
+runner.setIncomingConnection(false);
+runner.run(1, true, true);
+} catch (Throwable pe) {
+exception = true;
+}
+
+Assert.assertTrue("No exception was thrown!", exception);
--- End diff --

After changing to throw a ProcessException, you could instead have an 
Exception object set to null (rather than a boolean set to false), then you can 
assertNotNull(exception) as well as assertTrue(exception instanceof 
ProcessException).


> Make GetMongo able to use flowfiles for queries
> ---
>
> Key: NIFI-4827
> URL: https://issues.apache.org/jira/browse/NIFI-4827
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> GetMongo should be able to retrieve a valid query from the flowfile content 
> or allow the incoming flowfile to provide attributes to power EL statements 
> in the Query configuration field. Allowing the body to be used would allow 
> GetMongo to be used in a much more generic way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170034475
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoTest.java
 ---
@@ -296,4 +298,55 @@ public void testQueryAttribute() {
 Assert.assertEquals("Value was wrong", val, "{}");
 }
 }
+public void testQueryLocationConfig() {
+//Test EL
+Map attributes = new HashMap();
+attributes.put("field", "c");
+attributes.put("value", "4");
+String query = "{ \"${field}\": { \"$gte\": ${value}}}";
+runner.setProperty(GetMongo.QUERY, query);
+runner.setProperty(GetMongo.RESULTS_PER_FLOWFILE, "10");
+runner.setValidateExpressionUsage(true);
+runner.enqueue("test", attributes);
+runner.run(1, true, true);
+
+runner.assertTransferCount(GetMongo.REL_FAILURE, 0);
+runner.assertTransferCount(GetMongo.REL_ORIGINAL, 1);
+runner.assertTransferCount(GetMongo.REL_SUCCESS, 1);
+
+runner.clearTransferState();
+
+query = "{ \"c\": { \"$gte\": 4 }}";
+runner.setProperty(GetMongo.QUERY, query);
+runner.run(1, true, true);
+runner.assertTransferCount(GetMongo.REL_FAILURE, 0);
+runner.assertTransferCount(GetMongo.REL_ORIGINAL, 0);
+runner.assertTransferCount(GetMongo.REL_SUCCESS, 1);
+
+runner.clearTransferState();
+
+runner.removeProperty(GetMongo.QUERY);
+runner.enqueue(query);
+runner.run(1, true, true);
+
+runner.assertTransferCount(GetMongo.REL_FAILURE, 0);
+runner.assertTransferCount(GetMongo.REL_ORIGINAL, 1);
+runner.assertTransferCount(GetMongo.REL_SUCCESS, 1);
+
+runner.clearTransferState();
+
+boolean exception = false;
+try {
+runner.assertValid();
+runner.setIncomingConnection(false);
+runner.run(1, true, true);
+} catch (Throwable pe) {
+exception = true;
+}
+
+Assert.assertTrue("No exception was thrown!", exception);
--- End diff --

After changing to throw a ProcessException, you could instead have an 
Exception object set to null (rather than a boolean set to false), then you can 
assertNotNull(exception) as well as assertTrue(exception instanceof 
ProcessException).


---

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170032821
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
 ---
@@ -226,6 +242,11 @@ private ObjectWriter getObjectWriter(ObjectMapper 
mapper, String ppSetting) {
 
 @Override
 public void onTrigger(final ProcessContext context, final 
ProcessSession session) throws ProcessException {
+FlowFile input = session.get();
+if (!context.hasIncomingConnection() && 
(context.getProperty(QUERY) == null)) {
+throw new RuntimeException("Without an incoming connection, 
the Query property must be set.");
--- End diff --

The RuntimeException here and on line 276 should be ProcessException, it's 
a NiFi-specific runtime exception that indicates we realize something went 
wrong but we don't know how to handle any input/output we might have (like 
routing to failure).  Also rather than `context.getProperty(QUERY) == null` 
since there will be a PropertyValue object returned, you should use 
`context.getProperty(QUERY).isSet()`.


---

[GitHub] nifi pull request #2443: NIFI-4827 Added support for reading queries from th...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2443#discussion_r170033540
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoTest.java
 ---
@@ -69,6 +70,7 @@
 @Before
 public void setup() {
 runner = TestRunners.newTestRunner(GetMongo.class);
+runner.setProperty(GetMongo.QUERY, "{}");
--- End diff --

Does this need to be here? Seems like it might hide the cases when Query is 
empty, which would be the existing behavior. If it's because of the validator, 
then it should accept an empty query as valid since the property is not 
required.  Also, can you add a unit test (or update the existing one) for when 
the Query is empty and there is no incoming flow file (but there is an incoming 
connection)? That would test the other half of the conditions for the exception 
block you added at the top of onTrigger.


---

[GitHub] nifi issue #2443: NIFI-4827 Added support for reading queries from the flowf...

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2443
  
@mattyb149 I have time today that if you can get to this I can switch over 
to 4838 and get that ready for close out too.


---

[jira] [Resolved] (MINIFICPP-407) Change wording

[Aldrin Piri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aldrin Piri resolved MINIFICPP-407.
---
Resolution: Fixed

> Change wording
> --
>
> Key: MINIFICPP-407
> URL: https://issues.apache.org/jira/browse/MINIFICPP-407
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: marco polo
>Assignee: marco polo
>Priority: Major
> Fix For: 0.5.0
>
>
> Change wording of Readme to tell that we are GA



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (MINIFICPP-407) Change wording

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372954#comment-16372954
 ] 

ASF GitHub Bot commented on MINIFICPP-407:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/266


> Change wording
> --
>
> Key: MINIFICPP-407
> URL: https://issues.apache.org/jira/browse/MINIFICPP-407
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: marco polo
>Assignee: marco polo
>Priority: Major
> Fix For: 0.5.0
>
>
> Change wording of Readme to tell that we are GA



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp pull request #266: MINIFICPP-407: Update readme to clarify t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/266


---

[jira] [Commented] (MINIFICPP-411) MiNIFi Controller's output needs some fixing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372924#comment-16372924
 ] 

ASF GitHub Bot commented on MINIFICPP-411:
--

GitHub user phrocker opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/269

MINIFICPP-411: Resolve issues with command reporting. Show the size b…

…efore clear, and fix docs to clarify that a response isn't immediate

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [ ] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-411

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/269.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #269


commit 09ee8e08345d1a9866a1b25c9d3a13f3185b5499
Author: Marc Parisi 
Date:   2018-02-22T15:04:38Z

MINIFICPP-411: Resolve issues with command reporting. Show the size before 
clear, and fix docs to clarify that a response isn't immediate




> MiNIFi Controller's output needs some fixing
> 
>
> Key: MINIFICPP-411
> URL: https://issues.apache.org/jira/browse/MINIFICPP-411
> Project: NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: marco polo
>Assignee: marco polo
>Priority: Major
>
> A few operations need improved verification. Read me is not correct markup



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (MINIFICPP-411) MiNIFi Controller's output needs some fixing

[marco polo (JIRA)]

marco polo created MINIFICPP-411:


 Summary: MiNIFi Controller's output needs some fixing
 Key: MINIFICPP-411
 URL: https://issues.apache.org/jira/browse/MINIFICPP-411
 Project: NiFi MiNiFi C++
  Issue Type: Bug
Reporter: marco polo
Assignee: marco polo


A few operations need improved verification. Read me is not correct markup



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372856#comment-16372856
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r169969726
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/pom.xml ---
@@ -0,0 +1,88 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+org.apache.nifi
+nifi-influxdb-bundle
+1.6.0-SNAPSHOT
+
+
+nifi-influxdb-processors
+jar
+
+
+   
+   org.influxdb
+   influxdb-java
+
+
+org.apache.commons
+commons-lang3
+
+
+org.apache.nifi
+nifi-api
+
+
+org.apache.nifi
+nifi-utils
+
+
+org.apache.nifi
+nifi-mock
+test
+
+
+org.slf4j
+slf4j-simple
+test
+
+
+junit
+junit
+test
+
+
+com.google.guava
+guava
+test
+
+
+
+
+default
+
+true
+
+
+
+
+org.apache.maven.plugins
+maven-surefire-plugin
+
+
+**/IT*.java
--- End diff --

Oh my bad, I didn't pay close attention... Usually the IT classes do not 
include the word Test. That's why I thought the root profile would be enough. 
I'd suggest to rename the IT classes by removing the Test word to be consistent 
with the other bundles.


> Implement put processor for InfluxDB
> 
>
> Key: NIFI-4289
> URL: https://issues.apache.org/jira/browse/NIFI-4289
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.3.0
> Environment: All
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
>  Labels: insert, measurements,, put, timeseries
> Fix For: 1.6.0
>
>
> Support inserting time series measurements into InfluxDB.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2101: NIFI-4289 - InfluxDB put processor

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r169969726
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/pom.xml ---
@@ -0,0 +1,88 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+org.apache.nifi
+nifi-influxdb-bundle
+1.6.0-SNAPSHOT
+
+
+nifi-influxdb-processors
+jar
+
+
+   
+   org.influxdb
+   influxdb-java
+
+
+org.apache.commons
+commons-lang3
+
+
+org.apache.nifi
+nifi-api
+
+
+org.apache.nifi
+nifi-utils
+
+
+org.apache.nifi
+nifi-mock
+test
+
+
+org.slf4j
+slf4j-simple
+test
+
+
+junit
+junit
+test
+
+
+com.google.guava
+guava
+test
+
+
+
+
+default
+
+true
+
+
+
+
+org.apache.maven.plugins
+maven-surefire-plugin
+
+
+**/IT*.java
--- End diff --

Oh my bad, I didn't pay close attention... Usually the IT classes do not 
include the word Test. That's why I thought the root profile would be enough. 
I'd suggest to rename the IT classes by removing the Test word to be consistent 
with the other bundles.


---

[jira] [Commented] (NIFI-4289) Implement put processor for InfluxDB

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372851#comment-16372851
 ] 

ASF GitHub Bot commented on NIFI-4289:
--

Github user mans2singh commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r169968256
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/pom.xml ---
@@ -0,0 +1,88 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+org.apache.nifi
+nifi-influxdb-bundle
+1.6.0-SNAPSHOT
+
+
+nifi-influxdb-processors
+jar
+
+
+   
+   org.influxdb
+   influxdb-java
+
+
+org.apache.commons
+commons-lang3
+
+
+org.apache.nifi
+nifi-api
+
+
+org.apache.nifi
+nifi-utils
+
+
+org.apache.nifi
+nifi-mock
+test
+
+
+org.slf4j
+slf4j-simple
+test
+
+
+junit
+junit
+test
+
+
+com.google.guava
+guava
+test
+
+
+
+
+default
+
+true
+
+
+
+
+org.apache.maven.plugins
+maven-surefire-plugin
+
+
+**/IT*.java
--- End diff --

@pvillard31 

If I remove this profile/configuration from the pom.xml and run `mvn clean 
install` in that directory, the integration tests get executed and fail if 
InfluxDb is not running on the local server.  

```
Tests in error: 
  ITPutInfluxDBTest.setUp:58 » InfluxDBIO java.net.ConnectException: Failed 
to c...
  ITPutInfluxDBTest.setUp:58 » InfluxDBIO java.net.ConnectException: Failed 
to c...
  ITPutInfluxDBTest.setUp:58 » InfluxDBIO java.net.ConnectException: Failed 
to c...
  ITPutInfluxDBTest.setUp:58 » InfluxDBIO java.net.ConnectException: Failed 
to c...
  ITPutInfluxDBTest.setUp:58 » InfluxDBIO java.net.ConnectException: Failed 
to c...
  ITPutInfluxDBTest.setUp:58 » InfluxDBIO java.net.ConnectException: Failed 
to c...
```

Can you please let me know how I can skip running the integration tests 
without using this profile ?

Thanks for @MikeThomsen and your feedback.  


> Implement put processor for InfluxDB
> 
>
> Key: NIFI-4289
> URL: https://issues.apache.org/jira/browse/NIFI-4289
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.3.0
> Environment: All
>Reporter: Mans Singh
>Assignee: Mans Singh
>Priority: Minor
>  Labels: insert, measurements,, put, timeseries
> Fix For: 1.6.0
>
>
> Support inserting time series measurements into InfluxDB.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2101: NIFI-4289 - InfluxDB put processor

[mans2singh]

Github user mans2singh commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2101#discussion_r169968256
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/pom.xml ---
@@ -0,0 +1,88 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+org.apache.nifi
+nifi-influxdb-bundle
+1.6.0-SNAPSHOT
+
+
+nifi-influxdb-processors
+jar
+
+
+   
+   org.influxdb
+   influxdb-java
+
+
+org.apache.commons
+commons-lang3
+
+
+org.apache.nifi
+nifi-api
+
+
+org.apache.nifi
+nifi-utils
+
+
+org.apache.nifi
+nifi-mock
+test
+
+
+org.slf4j
+slf4j-simple
+test
+
+
+junit
+junit
+test
+
+
+com.google.guava
+guava
+test
+
+
+
+
+default
+
+true
+
+
+
+
+org.apache.maven.plugins
+maven-surefire-plugin
+
+
+**/IT*.java
--- End diff --

@pvillard31 

If I remove this profile/configuration from the pom.xml and run `mvn clean 
install` in that directory, the integration tests get executed and fail if 
InfluxDb is not running on the local server.  

```
Tests in error: 
  ITPutInfluxDBTest.setUp:58 » InfluxDBIO java.net.ConnectException: 
Failed to c...
  ITPutInfluxDBTest.setUp:58 » InfluxDBIO java.net.ConnectException: 
Failed to c...
  ITPutInfluxDBTest.setUp:58 » InfluxDBIO java.net.ConnectException: 
Failed to c...
  ITPutInfluxDBTest.setUp:58 » InfluxDBIO java.net.ConnectException: 
Failed to c...
  ITPutInfluxDBTest.setUp:58 » InfluxDBIO java.net.ConnectException: 
Failed to c...
  ITPutInfluxDBTest.setUp:58 » InfluxDBIO java.net.ConnectException: 
Failed to c...
```

Can you please let me know how I can skip running the integration tests 
without using this profile ?

Thanks for @MikeThomsen and your feedback.  


---

[GitHub] nifi issue #2339: NIFI-4683: Add ability to execute Spark jobs and code via ...

[joewitt]

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2339
  
That probably needs to be a new JIRA and improvement.


---

[GitHub] nifi issue #2339: NIFI-4683: Add ability to execute Spark jobs and code via ...

[jomach]

Github user jomach commented on the issue:

https://github.com/apache/nifi/pull/2339
  
hi Guys, this is a great change ! I know that the concept from nifi does 
not want it but yeah is life. :) 

One question I have what if the Livy server has  Kerberos active where can 
I give it a keytab to the controlerservice ?


---

[jira] [Commented] (NIFI-4683) Add ability to execute Spark jobs via Livy

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372837#comment-16372837
 ] 

ASF GitHub Bot commented on NIFI-4683:
--

Github user jomach commented on the issue:

https://github.com/apache/nifi/pull/2339
  
hi Guys, this is a great change ! I know that the concept from nifi does 
not want it but yeah is life. :) 

One question I have what if the Livy server has  Kerberos active where can 
I give it a keytab to the controlerservice ?


> Add ability to execute Spark jobs via Livy
> --
>
> Key: NIFI-4683
> URL: https://issues.apache.org/jira/browse/NIFI-4683
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.5.0
>
>
> Proposal for a new feature to enable NiFi users to execute Spark jobs. A 
> natural entry point for this is to use Apache Livy, as it is a "REST service 
> for Apache Spark". This would allow NiFi to submit Spark jobs without needing 
> to bundle a Spark client itself (and maintain versions of Spark, e.g.).
> Some of the components that could be involved include:
> LivySessionController Controller Service (CS) - provides connections to 
> available sessions in Livy
> * Users could request a type of connection or to retrieve the same connection 
> back by session id if available.
> * Properties to configure Livy session such as number of executors, memory
> * Property for connection pool size
> * Will interact with Livy ensure that only connections that are 
> idle/available are added to the pool and checked back in
> * Key for pool could be based on session id or type
> * Ensure to provide any user credentials
> * Leverages SSLContext for security
> LivyProcessor
> * Obtains Spark JARs/files via properties and/or flow file attribute(s)
> * Obtains connection information from LivySessionController
> * Provides attributes to configure session, maintain session id, attach to 
> session id
> * Potential advanced UI available for testing code (probably a follow-on Jira)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4899) Unable to find valid certification path to requested target

[Josef Zahner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josef Zahner updated NIFI-4899:
---
Attachment: nifi_cert_issue.zip

> Unable to find valid certification path to requested target
> ---
>
> Key: NIFI-4899
> URL: https://issues.apache.org/jira/browse/NIFI-4899
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.5.0
> Environment: NiFi Version 1.5.0 
> Java 1.8.0_161-b12 
> CentOS Linux release 7.4.1708
>Reporter: Josef Zahner
>Priority: Minor
>  Labels: certificate, login, ssl
> Attachments: Screen Shot 2018-02-21 at 11.08.13.png, 
> nifi_cert_issue.zip
>
>
> In my clustered ssl environment, if I start the webgui the first time, enter 
> my login credentials (verified via LDAP) and go ahead (click "LOG IN") I'm 
> getting the error below:
> !Screen Shot 2018-02-21 at 11.08.13.png!
> {code:java}
> javax.ws.rs.ProcessingException: javax.net.ssl.SSLHandshakeException: 
> sun.security.validator.ValidatorException: PKIX path building failed: 
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
> valid certification path to requested target
> at 
> org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:284)
> at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:278)
> at 
> org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:753)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:229)
> at 
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:414)
> at 
> org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:752)
> at 
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:661)
> at 
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:875)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: javax.net.ssl.SSLHandshakeException: 
> sun.security.validator.ValidatorException: PKIX path building failed: 
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
> valid certification path to requested target
> at sun.security.ssl.Alerts.getSSLException(Unknown Source)
> at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)
> at sun.security.ssl.Handshaker.fatalSE(Unknown Source)
> at sun.security.ssl.Handshaker.fatalSE(Unknown Source)
> at sun.security.ssl.ClientHandshaker.serverCertificate(Unknown Source)
> at sun.security.ssl.ClientHandshaker.processMessage(Unknown Source)
> at sun.security.ssl.Handshaker.processLoop(Unknown Source)
> at sun.security.ssl.Handshaker.process_record(Unknown Source)
> at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
> at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
> at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
> at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
> at sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
> at 
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown 
> Source)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
> at java.net.HttpURLConnection.getResponseCode(Unknown Source)
> at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown 
> Source)
> at 
> org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:390)
> at 
> org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:282)
> ... 14 common frames omitted
> Caused by: sun.security.validator.ValidatorException: PKIX path building 
> failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to 
> find valid certification path to requested target
> at sun.security.validator.PKIXValidator.doBuild(Unknown Source)
> at sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
> at sun.security.validator.Validator.validate(Unknown Source)
> at sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source)
> at sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
> at 

[jira] [Commented] (NIFI-4899) Unable to find valid certification path to requested target

[Josef Zahner (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372700#comment-16372700
 ] 

Josef Zahner commented on NIFI-4899:


The problem occurs only on a clustered setup. I've deployed the same ansible 
template (just without enabling clustering) on a standalone NiFi and the 
message doesn't appear anymore.

We have our own root CA within our company and I've created the 
keystore/truststore based on that. However, I have the same issue as well with 
the toolkit and self-signed certs. 

I've added the related config files and certs for my test with nifi-toolkit in 
a two node cluster environment. It's just lab, so doesn't matter.

 

> Unable to find valid certification path to requested target
> ---
>
> Key: NIFI-4899
> URL: https://issues.apache.org/jira/browse/NIFI-4899
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.5.0
> Environment: NiFi Version 1.5.0 
> Java 1.8.0_161-b12 
> CentOS Linux release 7.4.1708
>Reporter: Josef Zahner
>Priority: Minor
>  Labels: certificate, login, ssl
> Attachments: Screen Shot 2018-02-21 at 11.08.13.png, 
> nifi_cert_issue.zip
>
>
> In my clustered ssl environment, if I start the webgui the first time, enter 
> my login credentials (verified via LDAP) and go ahead (click "LOG IN") I'm 
> getting the error below:
> !Screen Shot 2018-02-21 at 11.08.13.png!
> {code:java}
> javax.ws.rs.ProcessingException: javax.net.ssl.SSLHandshakeException: 
> sun.security.validator.ValidatorException: PKIX path building failed: 
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
> valid certification path to requested target
> at 
> org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:284)
> at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:278)
> at 
> org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:753)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:229)
> at 
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:414)
> at 
> org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:752)
> at 
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:661)
> at 
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:875)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: javax.net.ssl.SSLHandshakeException: 
> sun.security.validator.ValidatorException: PKIX path building failed: 
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
> valid certification path to requested target
> at sun.security.ssl.Alerts.getSSLException(Unknown Source)
> at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)
> at sun.security.ssl.Handshaker.fatalSE(Unknown Source)
> at sun.security.ssl.Handshaker.fatalSE(Unknown Source)
> at sun.security.ssl.ClientHandshaker.serverCertificate(Unknown Source)
> at sun.security.ssl.ClientHandshaker.processMessage(Unknown Source)
> at sun.security.ssl.Handshaker.processLoop(Unknown Source)
> at sun.security.ssl.Handshaker.process_record(Unknown Source)
> at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
> at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
> at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
> at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
> at sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
> at 
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown 
> Source)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
> at java.net.HttpURLConnection.getResponseCode(Unknown Source)
> at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown 
> Source)
> at 
> org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:390)
> at 
> org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:282)
> ... 14 common frames omitted
> Caused by: sun.security.validator.ValidatorException: PKIX path building 
> failed: