[jira] [Commented] (NIFI-3332) Bug in ListXXX causes matching timestamps to be ignored on later runs

[Koji Kawamura (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882081#comment-15882081
 ] 

Koji Kawamura commented on NIFI-3332:
-

[~jskora] I think the success marker file can be created by a processor in the 
same flow or external program that creates files need to be listed, but as a 
manual and additional step. If files are created by a 3rd party and you can't 
modify the behavior of such program, then this pattern would be useless. When 
you do, it can guarantee that all files written to a dir before _SUCCESS file 
is created are listed.

To complete NiFi's file listing capabilities, I come up with additional 
processors, those are, WatchFiles and DiffFiles:

!listfiles.png|width=100%!

h3. DiffFiles

Compared with ListFile, DiffFiles doesn't use managed state, instead it uses 
flow file.
It outputs list of filenames and probably a set of meta data such as timestamp, 
file size and hash ... etc as the content of an outgoing single FlowFile 
(probably JSON), so that it can handle large amount of those data.
When it receives the incoming FlowFile, check diffs between the incoming list 
(previous list) and current fetched list, then emits diffs to added, updated 
and removed accordingly.

This processor can be used against any filesystem, remote or local. It doesn't 
rely on timestamp as ListFile does.

h3. WatchFiles

As a project having more streaming processing in mind, we might need 
WatchDirectory processor for local file system or any other file systems that 
support watch API. It would be the same implementation pattern with ConsumeXXX.
Using watch service API available since 1.7.
Watching a Directory for Changes
https://docs.oracle.com/javase/tutorial/essential/io/notification.html

Do you think these additional processors will be helpful? I think if these are 
available, _SUCCESS file is no longer needed.

> Bug in ListXXX causes matching timestamps to be ignored on later runs
> -
>
> Key: NIFI-3332
> URL: https://issues.apache.org/jira/browse/NIFI-3332
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.7.1, 1.1.1
>Reporter: Joe Skora
>Assignee: Koji Kawamura
>Priority: Critical
> Attachments: listfiles.png, Test-showing-ListFile-timestamp-bug.log, 
> Test-showing-ListFile-timestamp-bug.patch
>
>
> The new state implementation for the ListXXX processors based on 
> AbstractListProcessor creates a race conditions when processor runs occur 
> while a batch of files is being written with the same timestamp.
> The changes to state management dropped tracking of the files processed for a 
> given timestamp.  Without the record of files processed, the remainder of the 
> batch is ignored on the next processor run since their timestamp is not 
> greater than the one timestamp stored in processor state.  With the file 
> tracking it was possible to process files that matched the timestamp exactly 
> and exclude the previously processed files.
> A basic time goes as follows.
>   T0 - system creates or receives batch of files with Tx timestamp where Tx 
> is more than the current timestamp in processor state.
>   T1 - system writes 1st half of Tx batch to the ListFile source directory.
>   T2 - ListFile runs picking up 1st half of Tx batch and stores Tx timestamp 
> in processor state.
>   T3 - system writes 2nd half of Tx batch to ListFile source directory.
>   T4 - ListFile runs ignoring any files with T <= Tx, eliminating 2nd half Tx 
> timestamp batch.
> I've attached a patch[1] for TestListFile.java that adds an instrumented unit 
> test demonstrates the problem and a log[2] of the output from one such run.  
> The test writes 3 files each in two batches with processor runs after each 
> batch.  Batch 2 writes files with timestamps older than, equal to, and newer 
> than the timestamp stored when batch 1 was processed, but only the newer file 
> is picked up.  The older file is correctly ignored but file with the matchin 
> timestamp file should have been processed.
> [1] Test-showing-ListFile-timestamp-bug.patch
> [2] Test-showing-ListFile-timestamp-bug.log



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3332) Bug in ListXXX causes matching timestamps to be ignored on later runs

[Koji Kawamura (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Kawamura updated NIFI-3332:

Attachment: listfiles.png

> Bug in ListXXX causes matching timestamps to be ignored on later runs
> -
>
> Key: NIFI-3332
> URL: https://issues.apache.org/jira/browse/NIFI-3332
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.7.1, 1.1.1
>Reporter: Joe Skora
>Assignee: Koji Kawamura
>Priority: Critical
> Attachments: listfiles.png, Test-showing-ListFile-timestamp-bug.log, 
> Test-showing-ListFile-timestamp-bug.patch
>
>
> The new state implementation for the ListXXX processors based on 
> AbstractListProcessor creates a race conditions when processor runs occur 
> while a batch of files is being written with the same timestamp.
> The changes to state management dropped tracking of the files processed for a 
> given timestamp.  Without the record of files processed, the remainder of the 
> batch is ignored on the next processor run since their timestamp is not 
> greater than the one timestamp stored in processor state.  With the file 
> tracking it was possible to process files that matched the timestamp exactly 
> and exclude the previously processed files.
> A basic time goes as follows.
>   T0 - system creates or receives batch of files with Tx timestamp where Tx 
> is more than the current timestamp in processor state.
>   T1 - system writes 1st half of Tx batch to the ListFile source directory.
>   T2 - ListFile runs picking up 1st half of Tx batch and stores Tx timestamp 
> in processor state.
>   T3 - system writes 2nd half of Tx batch to ListFile source directory.
>   T4 - ListFile runs ignoring any files with T <= Tx, eliminating 2nd half Tx 
> timestamp batch.
> I've attached a patch[1] for TestListFile.java that adds an instrumented unit 
> test demonstrates the problem and a log[2] of the output from one such run.  
> The test writes 3 files each in two batches with processor runs after each 
> batch.  Batch 2 writes files with timestamps older than, equal to, and newer 
> than the timestamp stored when batch 1 was processed, but only the newer file 
> is picked up.  The older file is correctly ignored but file with the matchin 
> timestamp file should have been processed.
> [1] Test-showing-ListFile-timestamp-bug.patch
> [2] Test-showing-ListFile-timestamp-bug.log



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi-minifi-cpp issue #37: MINIFI-171 Dynamic Properties support for process...

[jdye64]

Github user jdye64 commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/37
  
@apiri Where do you think we stand on this? I know you had some concerns 
about the implementation approach in the beginning. Is that still the case? If 
so what can we do to resolve those?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3358) UI Modularize JS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881855#comment-15881855
 ] 

ASF GitHub Bot commented on NIFI-3358:
--

GitHub user scottyaslan opened a pull request:

https://github.com/apache/nifi/pull/1540

[NIFI-3358] remove duplicate calls to nfProcessGroupConfiguration.sho…

…wConfiguration()

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/scottyaslan/nifi NIFI-3358

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1540.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1540


commit 1c77af346131398ae2a34b06acc5fe60268f41ee
Author: Scott Aslan 
Date:   2017-02-24T03:16:07Z

[NIFI-3358] remove duplicate calls to 
nfProcessGroupConfiguration.showConfiguration()




> UI Modularize JS 
> -
>
> Key: NIFI-3358
> URL: https://issues.apache.org/jira/browse/NIFI-3358
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Scott Aslan
>Assignee: Scott Aslan
> Fix For: 1.2.0
>
>
> This is an initial parent ticket to capture work and efforts to enhance the 
> UI components to promote maintainability and eventual reusability for UI 
> extensions.
> Subcomponents of this effort should be treated as subtasks.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3358) UI Modularize JS

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan updated NIFI-3358:
--
Status: Patch Available  (was: Reopened)

> UI Modularize JS 
> -
>
> Key: NIFI-3358
> URL: https://issues.apache.org/jira/browse/NIFI-3358
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Scott Aslan
>Assignee: Scott Aslan
> Fix For: 1.2.0
>
>
> This is an initial parent ticket to capture work and efforts to enhance the 
> UI components to promote maintainability and eventual reusability for UI 
> extensions.
> Subcomponents of this effort should be treated as subtasks.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1540: [NIFI-3358] remove duplicate calls to nfProcessGrou...

[scottyaslan]

GitHub user scottyaslan opened a pull request:

https://github.com/apache/nifi/pull/1540

[NIFI-3358] remove duplicate calls to nfProcessGroupConfiguration.sho…

…wConfiguration()

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/scottyaslan/nifi NIFI-3358

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1540.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1540


commit 1c77af346131398ae2a34b06acc5fe60268f41ee
Author: Scott Aslan 
Date:   2017-02-24T03:16:07Z

[NIFI-3358] remove duplicate calls to 
nfProcessGroupConfiguration.showConfiguration()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Reopened] (NIFI-3358) UI Modularize JS

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan reopened NIFI-3358:
---

A js error is thrown when attempting to repeatedly open and close the root 
process group config dialog.

> UI Modularize JS 
> -
>
> Key: NIFI-3358
> URL: https://issues.apache.org/jira/browse/NIFI-3358
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Scott Aslan
>Assignee: Scott Aslan
> Fix For: 1.2.0
>
>
> This is an initial parent ticket to capture work and efforts to enhance the 
> UI components to promote maintainability and eventual reusability for UI 
> extensions.
> Subcomponents of this effort should be treated as subtasks.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-385) Add Kerberos support in nifi-kite-nar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881804#comment-15881804
 ] 

ASF GitHub Bot commented on NIFI-385:
-

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1528
  
@WilliamNouet this looks like a duplicated PR from the other one 
introducing Morphlines? If so, mind to close it?


> Add Kerberos support in nifi-kite-nar
> -
>
> Key: NIFI-385
> URL: https://issues.apache.org/jira/browse/NIFI-385
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Ryan Blue
>
> Kite should be able to connect to a Kerberized Hadoop cluster to store data. 
> Kite's Flume connector has working code. The Kite dataset needs to be 
> instantiated in a {{doPrivileged}} block and its internal {{FileSystem}} 
> object will hold the credentials after that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1528: NIFI-385 Add Kerberos support in nifi-kite-nar

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1528
  
@WilliamNouet this looks like a duplicated PR from the other one 
introducing Morphlines? If so, mind to close it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3518) Create a Morphlines processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881707#comment-15881707
 ] 

ASF GitHub Bot commented on NIFI-3518:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1529#discussion_r102860737
  
--- Diff: 
nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-processor/src/main/java/org/apache/nifi/processors/morphlines/Morphlines.java
 ---
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.morphlines;
+
+import com.google.common.base.Preconditions;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.*;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.kitesdk.morphline.api.Command;
+import org.kitesdk.morphline.api.MorphlineContext;
+import org.kitesdk.morphline.api.Record;
+import org.kitesdk.morphline.base.Fields;
+
+import org.kitesdk.morphline.base.Notifications;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.*;
+import java.util.stream.*;
+import java.util.concurrent.atomic.*;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.processor.exception.*;
+
+@Tags({"kitesdk", "morphlines"})
+@CapabilityDescription("Provide a description")
+public class MorphlinesProcessor extends AbstractProcessor {
+
+private static Command morphline;
+private volatile Record record = new Record();
+private static volatile Collector collector = new Collector();
+
+public static final PropertyDescriptor MORPHLINES_ID = new 
PropertyDescriptor
+.Builder().name("Morphlines ID")
+.description("Identifier of the morphlines context")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+public static final PropertyDescriptor MORPHLINES_FILE = new 
PropertyDescriptor
+.Builder().name("Morphlines File")
+.description("File for the morphlines context")
+.required(true)
+.addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+public static final PropertyDescriptor MORPHLINES_OUTPUT_FIELD = new 
PropertyDescriptor
+.Builder().name("Morphlines output field")
+.description("Field name of output in Morphlines. Default is 
'_attachment_body'.")
+.required(false)
+.defaultValue("_attachment_body")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder()
+.name("success")
+.description("Relationship for success.")
+.build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder()
+.name("failure")
+.description("Relationship for failure of morphlines.")
   

[GitHub] nifi pull request #1529: NIFI-3518 Create a Morphlines processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1529#discussion_r102860737
  
--- Diff: 
nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-processor/src/main/java/org/apache/nifi/processors/morphlines/Morphlines.java
 ---
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.morphlines;
+
+import com.google.common.base.Preconditions;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.*;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.kitesdk.morphline.api.Command;
+import org.kitesdk.morphline.api.MorphlineContext;
+import org.kitesdk.morphline.api.Record;
+import org.kitesdk.morphline.base.Fields;
+
+import org.kitesdk.morphline.base.Notifications;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.*;
+import java.util.stream.*;
+import java.util.concurrent.atomic.*;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.processor.exception.*;
+
+@Tags({"kitesdk", "morphlines"})
+@CapabilityDescription("Provide a description")
+public class MorphlinesProcessor extends AbstractProcessor {
+
+private static Command morphline;
+private volatile Record record = new Record();
+private static volatile Collector collector = new Collector();
+
+public static final PropertyDescriptor MORPHLINES_ID = new 
PropertyDescriptor
+.Builder().name("Morphlines ID")
+.description("Identifier of the morphlines context")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+public static final PropertyDescriptor MORPHLINES_FILE = new 
PropertyDescriptor
+.Builder().name("Morphlines File")
+.description("File for the morphlines context")
+.required(true)
+.addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+public static final PropertyDescriptor MORPHLINES_OUTPUT_FIELD = new 
PropertyDescriptor
+.Builder().name("Morphlines output field")
+.description("Field name of output in Morphlines. Default is 
'_attachment_body'.")
+.required(false)
+.defaultValue("_attachment_body")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder()
+.name("success")
+.description("Relationship for success.")
+.build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder()
+.name("failure")
+.description("Relationship for failure of morphlines.")
+.build();
+
+public static final Relationship REL_ORIGINAL = new 
Relationship.Builder()
+.name("original")
+.description("Relationship for original flowfiles.")
+   

[jira] [Commented] (NIFI-3518) Create a Morphlines processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881703#comment-15881703
 ] 

ASF GitHub Bot commented on NIFI-3518:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1529#discussion_r102860592
  
--- Diff: 
nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-processor/src/main/java/org/apache/nifi/processors/morphlines/Morphlines.java
 ---
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.morphlines;
+
+import com.google.common.base.Preconditions;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.*;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.kitesdk.morphline.api.Command;
+import org.kitesdk.morphline.api.MorphlineContext;
+import org.kitesdk.morphline.api.Record;
+import org.kitesdk.morphline.base.Fields;
+
+import org.kitesdk.morphline.base.Notifications;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.*;
+import java.util.stream.*;
+import java.util.concurrent.atomic.*;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.processor.exception.*;
+
+@Tags({"kitesdk", "morphlines"})
+@CapabilityDescription("Provide a description")
+public class MorphlinesProcessor extends AbstractProcessor {
--- End diff --

Please rename processor so that it matches naming conventions


https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#naming-convensions




> Create a Morphlines processor
> -
>
> Key: NIFI-3518
> URL: https://issues.apache.org/jira/browse/NIFI-3518
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: William Nouet
>Priority: Minor
>
> Create a dedicate processor to run Morphlines transformations 
> (http://kitesdk.org/docs/1.1.0/morphlines/morphlines-reference-guide.html) 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1529: NIFI-3518 Create a Morphlines processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1529#discussion_r102860592
  
--- Diff: 
nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-processor/src/main/java/org/apache/nifi/processors/morphlines/Morphlines.java
 ---
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.morphlines;
+
+import com.google.common.base.Preconditions;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.*;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.kitesdk.morphline.api.Command;
+import org.kitesdk.morphline.api.MorphlineContext;
+import org.kitesdk.morphline.api.Record;
+import org.kitesdk.morphline.base.Fields;
+
+import org.kitesdk.morphline.base.Notifications;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.*;
+import java.util.stream.*;
+import java.util.concurrent.atomic.*;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.processor.exception.*;
+
+@Tags({"kitesdk", "morphlines"})
+@CapabilityDescription("Provide a description")
+public class MorphlinesProcessor extends AbstractProcessor {
--- End diff --

Please rename processor so that it matches naming conventions


https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#naming-convensions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3520) HDFS processors experiencing Kerberos "impersonate" errors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881690#comment-15881690
 ] 

ASF GitHub Bot commented on NIFI-3520:
--

GitHub user jtstorck opened a pull request:

https://github.com/apache/nifi/pull/1539

NIFI-3520 Updates classloading for components annotated with Requires…

…InstanceClassLoading to include dependencies in the InstanceClassLoader

Added RequiresInstanceClassLoading annotation to AbstractHadoopProcessor 
and HiveConnectionPool
Removed RequiresInstanceClassLoading from PutHDFS now that the 
AbstractHadoopProcessor uses the annotation
UGI relogins are now performed using doAs
Added debug-level logging for UGI relogins in KerberosTicketRenewer and 
AbstractHadoopProcessor

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jtstorck/nifi NIFI-3520

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1539.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1539


commit 9d91e60641f3ec528786f7add2ec878a8ad56173
Author: Jeff Storck 
Date:   2017-02-24T01:03:40Z

NIFI-3520 Updates classloading for components annotated with 
RequiresInstanceClassLoading to include dependencies in the InstanceClassLoader
Added RequiresInstanceClassLoading annotation to AbstractHadoopProcessor 
and HiveConnectionPool
Removed RequiresInstanceClassLoading from PutHDFS now that the 
AbstractHadoopProcessor uses the annotation
UGI relogins are now performed using doAs
Added debug-level logging for UGI relogins in KerberosTicketRenewer and 
AbstractHadoopProcessor




> HDFS processors experiencing Kerberos "impersonate" errors 
> ---
>
> Key: NIFI-3520
> URL: https://issues.apache.org/jira/browse/NIFI-3520
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 1.1.0, 1.1.1, 1.0.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>
> When multiple Kerberos principals are used between multiple HDFS processors, 
> the processor instances will be able to login to Kerberos with their 
> configured principals initially, but will not properly relogin.  
> For example, if there are two PutHDFS processors, one configured as 
> us...@example.com, and the other as us...@example.com, they will both login 
> with the KDC correctly and be able to transfer files to HDFS.  Once one of 
> the PutHDFS processors attempts to relogin, it may end up being logged in as 
> the principal from the other PutHDFS processor.  The principal contexts end 
> up getting switched, and the hadoop client used by the processor will attempt 
> to proxy requests from one user through another, resulting in the following 
> exception:
> {panel}Failed to write to HDFS due to 
> 

[GitHub] nifi pull request #1539: NIFI-3520 Updates classloading for components annot...

[jtstorck]

GitHub user jtstorck opened a pull request:

https://github.com/apache/nifi/pull/1539

NIFI-3520 Updates classloading for components annotated with Requires…

…InstanceClassLoading to include dependencies in the InstanceClassLoader

Added RequiresInstanceClassLoading annotation to AbstractHadoopProcessor 
and HiveConnectionPool
Removed RequiresInstanceClassLoading from PutHDFS now that the 
AbstractHadoopProcessor uses the annotation
UGI relogins are now performed using doAs
Added debug-level logging for UGI relogins in KerberosTicketRenewer and 
AbstractHadoopProcessor

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jtstorck/nifi NIFI-3520

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1539.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1539


commit 9d91e60641f3ec528786f7add2ec878a8ad56173
Author: Jeff Storck 
Date:   2017-02-24T01:03:40Z

NIFI-3520 Updates classloading for components annotated with 
RequiresInstanceClassLoading to include dependencies in the InstanceClassLoader
Added RequiresInstanceClassLoading annotation to AbstractHadoopProcessor 
and HiveConnectionPool
Removed RequiresInstanceClassLoading from PutHDFS now that the 
AbstractHadoopProcessor uses the annotation
UGI relogins are now performed using doAs
Added debug-level logging for UGI relogins in KerberosTicketRenewer and 
AbstractHadoopProcessor




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3518) Create a Morphlines processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881657#comment-15881657
 ] 

ASF GitHub Bot commented on NIFI-3518:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1529#discussion_r102856855
  
--- Diff: 
nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-processor/src/main/java/org/apache/nifi/processors/morphlines/Morphlines.java
 ---
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.morphlines;
+
+import com.google.common.base.Preconditions;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.*;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.kitesdk.morphline.api.Command;
+import org.kitesdk.morphline.api.MorphlineContext;
+import org.kitesdk.morphline.api.Record;
+import org.kitesdk.morphline.base.Fields;
+
+import org.kitesdk.morphline.base.Notifications;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.*;
+import java.util.stream.*;
+import java.util.concurrent.atomic.*;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.processor.exception.*;
+
+@Tags({"kitesdk", "morphlines"})
+@CapabilityDescription("Provide a description")
--- End diff --

Please add a description to what the processor does


> Create a Morphlines processor
> -
>
> Key: NIFI-3518
> URL: https://issues.apache.org/jira/browse/NIFI-3518
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: William Nouet
>Priority: Minor
>
> Create a dedicate processor to run Morphlines transformations 
> (http://kitesdk.org/docs/1.1.0/morphlines/morphlines-reference-guide.html) 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1529: NIFI-3518 Create a Morphlines processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1529#discussion_r102856855
  
--- Diff: 
nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-processor/src/main/java/org/apache/nifi/processors/morphlines/Morphlines.java
 ---
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.morphlines;
+
+import com.google.common.base.Preconditions;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.*;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.kitesdk.morphline.api.Command;
+import org.kitesdk.morphline.api.MorphlineContext;
+import org.kitesdk.morphline.api.Record;
+import org.kitesdk.morphline.base.Fields;
+
+import org.kitesdk.morphline.base.Notifications;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.*;
+import java.util.stream.*;
+import java.util.concurrent.atomic.*;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.processor.exception.*;
+
+@Tags({"kitesdk", "morphlines"})
+@CapabilityDescription("Provide a description")
--- End diff --

Please add a description to what the processor does


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3518) Create a Morphlines processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881641#comment-15881641
 ] 

ASF GitHub Bot commented on NIFI-3518:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1529
  
@WilliamNouet - thank you for the contribution. Had a quick look and added 
some comments. Other will probably want to have a look as well.


> Create a Morphlines processor
> -
>
> Key: NIFI-3518
> URL: https://issues.apache.org/jira/browse/NIFI-3518
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: William Nouet
>Priority: Minor
>
> Create a dedicate processor to run Morphlines transformations 
> (http://kitesdk.org/docs/1.1.0/morphlines/morphlines-reference-guide.html) 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1529: NIFI-3518 Create a Morphlines processor

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1529
  
@WilliamNouet - thank you for the contribution. Had a quick look and added 
some comments. Other will probably want to have a look as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1529: NIFI-3518 Create a Morphlines processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1529#discussion_r102855578
  
--- Diff: 
nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-processor/pom.xml ---
@@ -0,0 +1,130 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+  
+
+org.apache.nifi
+nifi-morphlines-bundle
+0.1
+
+
+nifi-morphlines-processors
+jar
+
+
+1.0.0
+1.1.0
--- End diff --

not sure you want to hardcode this. It should inherit from project.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3518) Create a Morphlines processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881639#comment-15881639
 ] 

ASF GitHub Bot commented on NIFI-3518:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1529#discussion_r102855578
  
--- Diff: 
nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-processor/pom.xml ---
@@ -0,0 +1,130 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+  
+
+org.apache.nifi
+nifi-morphlines-bundle
+0.1
+
+
+nifi-morphlines-processors
+jar
+
+
+1.0.0
+1.1.0
--- End diff --

not sure you want to hardcode this. It should inherit from project.


> Create a Morphlines processor
> -
>
> Key: NIFI-3518
> URL: https://issues.apache.org/jira/browse/NIFI-3518
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: William Nouet
>Priority: Minor
>
> Create a dedicate processor to run Morphlines transformations 
> (http://kitesdk.org/docs/1.1.0/morphlines/morphlines-reference-guide.html) 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3518) Create a Morphlines processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881630#comment-15881630
 ] 

ASF GitHub Bot commented on NIFI-3518:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1529#discussion_r102854353
  
--- Diff: nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/pom.xml ---
@@ -40,7 +40,21 @@
 org.apache.nifi
 nifi-utils
 
-
+
+org.apache.nifi
--- End diff --

have you tested this using the Cloudera and MapR profiles? Does it pass 
test?


> Create a Morphlines processor
> -
>
> Key: NIFI-3518
> URL: https://issues.apache.org/jira/browse/NIFI-3518
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: William Nouet
>Priority: Minor
>
> Create a dedicate processor to run Morphlines transformations 
> (http://kitesdk.org/docs/1.1.0/morphlines/morphlines-reference-guide.html) 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1529: NIFI-3518 Create a Morphlines processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1529#discussion_r102854353
  
--- Diff: nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/pom.xml ---
@@ -40,7 +40,21 @@
 org.apache.nifi
 nifi-utils
 
-
+
+org.apache.nifi
--- End diff --

have you tested this using the Cloudera and MapR profiles? Does it pass 
test?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3518) Create a Morphlines processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881627#comment-15881627
 ] 

ASF GitHub Bot commented on NIFI-3518:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1529#discussion_r102854253
  
--- Diff: 
nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-nar/pom.xml ---
@@ -0,0 +1,41 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+org.apache.com
+nifi-morphlines-bundle
+0.1
+
+
+nifi-morphlines-nar
+0.1
+nar
+
+true
+true
+
+
+
+
+org.apache.nifi
+nifi-morphlines-processors
+0.1
--- End diff --

not critical but generally the bundled processors versions follow the NiFi 
main version. So at this stage this should be 1.2.0-SNAPSHOT


> Create a Morphlines processor
> -
>
> Key: NIFI-3518
> URL: https://issues.apache.org/jira/browse/NIFI-3518
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: William Nouet
>Priority: Minor
>
> Create a dedicate processor to run Morphlines transformations 
> (http://kitesdk.org/docs/1.1.0/morphlines/morphlines-reference-guide.html) 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1529: NIFI-3518 Create a Morphlines processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1529#discussion_r102854253
  
--- Diff: 
nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-nar/pom.xml ---
@@ -0,0 +1,41 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+org.apache.com
+nifi-morphlines-bundle
+0.1
+
+
+nifi-morphlines-nar
+0.1
+nar
+
+true
+true
+
+
+
+
+org.apache.nifi
+nifi-morphlines-processors
+0.1
--- End diff --

not critical but generally the bundled processors versions follow the NiFi 
main version. So at this stage this should be 1.2.0-SNAPSHOT


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3518) Create a Morphlines processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881625#comment-15881625
 ] 

ASF GitHub Bot commented on NIFI-3518:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1529#discussion_r102854106
  
--- Diff: nifi-nar-bundles/nifi-morphlines-bundle/pom.xml ---
@@ -0,0 +1,39 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+org.apache.nifi
+nifi-nar-bundles
+1.1.1
+
+
+org.apache.nifi
+nifi-morphlines-bundle
+0.1
+pom
+
+
+nifi-morphlines-processors
+nifi-morphlines-nar
+
+
+
+true
--- End diff --

any particular reason why skip is being used?


> Create a Morphlines processor
> -
>
> Key: NIFI-3518
> URL: https://issues.apache.org/jira/browse/NIFI-3518
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: William Nouet
>Priority: Minor
>
> Create a dedicate processor to run Morphlines transformations 
> (http://kitesdk.org/docs/1.1.0/morphlines/morphlines-reference-guide.html) 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1529: NIFI-3518 Create a Morphlines processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1529#discussion_r102854106
  
--- Diff: nifi-nar-bundles/nifi-morphlines-bundle/pom.xml ---
@@ -0,0 +1,39 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+org.apache.nifi
+nifi-nar-bundles
+1.1.1
+
+
+org.apache.nifi
+nifi-morphlines-bundle
+0.1
+pom
+
+
+nifi-morphlines-processors
+nifi-morphlines-nar
+
+
+
+true
--- End diff --

any particular reason why skip is being used?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3518) Create a Morphlines processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881620#comment-15881620
 ] 

ASF GitHub Bot commented on NIFI-3518:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1529#discussion_r102853739
  
--- Diff: 
nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-processor/pom.xml ---
@@ -0,0 +1,130 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+  
+
+org.apache.nifi
+nifi-morphlines-bundle
+0.1
+
+
+nifi-morphlines-processors
+jar
+
+
+1.0.0
+1.1.0
+
+
+
+
+org.apache.nifi
+nifi-api
+
+
+org.apache.nifi
+nifi-processor-utils
+
+
+org.apache.nifi
+nifi-mock
+${nifi.version}
+test
+
+
+org.slf4j
+slf4j-simple
+test
+
+
+junit
+junit
+4.11
+test
+
+
+org.kitesdk
+kite-morphlines-core
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-avro
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-json
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-saxon
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-hadoop-core
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-hadoop-parquet-avro
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-hadoop-sequencefile
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-hadoop-rcfile
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-tika-core
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-tika-decompress
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-twitter
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-maxmind
+${kite.version}
--- End diff --

the relevant LICENSE and DEPENDENCY files must be added/updated to cater to 
all those imports. 


> Create a Morphlines processor
> -
>
> Key: NIFI-3518
> URL: https://issues.apache.org/jira/browse/NIFI-3518
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: William Nouet
>Priority: Minor
>
> Create a dedicate processor to run Morphlines transformations 
> (http://kitesdk.org/docs/1.1.0/morphlines/morphlines-reference-guide.html) 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1529: NIFI-3518 Create a Morphlines processor

[trixpan]

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1529#discussion_r102853739
  
--- Diff: 
nifi-nar-bundles/nifi-morphlines-bundle/nifi-morphlines-processor/pom.xml ---
@@ -0,0 +1,130 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+  
+
+org.apache.nifi
+nifi-morphlines-bundle
+0.1
+
+
+nifi-morphlines-processors
+jar
+
+
+1.0.0
+1.1.0
+
+
+
+
+org.apache.nifi
+nifi-api
+
+
+org.apache.nifi
+nifi-processor-utils
+
+
+org.apache.nifi
+nifi-mock
+${nifi.version}
+test
+
+
+org.slf4j
+slf4j-simple
+test
+
+
+junit
+junit
+4.11
+test
+
+
+org.kitesdk
+kite-morphlines-core
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-avro
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-json
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-saxon
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-hadoop-core
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-hadoop-parquet-avro
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-hadoop-sequencefile
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-hadoop-rcfile
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-tika-core
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-tika-decompress
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-twitter
+${kite.version}
+
+
+org.kitesdk
+kite-morphlines-maxmind
+${kite.version}
--- End diff --

the relevant LICENSE and DEPENDENCY files must be added/updated to cater to 
all those imports. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3518) Create a Morphlines processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881615#comment-15881615
 ] 

ASF GitHub Bot commented on NIFI-3518:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1529
  
@WilliamNouet to squash commit just checkout your branch and run:

`git rebase -i`

and follow the squash process as very well exemplified here:

http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html


> Create a Morphlines processor
> -
>
> Key: NIFI-3518
> URL: https://issues.apache.org/jira/browse/NIFI-3518
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: William Nouet
>Priority: Minor
>
> Create a dedicate processor to run Morphlines transformations 
> (http://kitesdk.org/docs/1.1.0/morphlines/morphlines-reference-guide.html) 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi issue #1529: NIFI-3518 Create a Morphlines processor

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1529
  
@WilliamNouet to squash commit just checkout your branch and run:

`git rebase -i`

and follow the squash process as very well exemplified here:

http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #54: MINIFI-206: Implement Extensible Comm Mechanism

[phrocker]

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/54
  
@apiri FYI the test protocol I used was to test s2s, bidirectionally, in 
secure and non-secure mode. I tested in linux and under mac. It inspired me to 
try and build an integration test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #54: MINIFI-206: Implement Extensible Comm Mechanism

[phrocker]

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/54
  
@apiri I tested it locally. I would like to add tests for it in unit tests 
ala the java client. That may require a little bit more work though to stand up 
a server, but that will be a subsequent PR

@jdye64  I will make this change


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #54: MINIFI-206: Implement Extensible Comm Mech...

[jdye64]

Github user jdye64 commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/54#discussion_r102845154
  
--- Diff: libminifi/CMakeLists.txt ---
@@ -36,12 +36,23 @@ ENDIF(POLICY CMP0048)
 set(CMAKE_CXX_STANDARD 11)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 
+include(CheckCXXCompilerFlag)
+CHECK_CXX_COMPILER_FLAG("-std=c++11" COMPILER_SUPPORTS_CXX11)
+CHECK_CXX_COMPILER_FLAG("-std=c++0x" COMPILER_SUPPORTS_CXX0X)
+if(COMPILER_SUPPORTS_CXX11)
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11 -g")
+elseif(COMPILER_SUPPORTS_CXX0X)
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++0x")
+else()
+ message(STATUS "The compiler ${CMAKE_CXX_COMPILER} has no C++11 support. 
Please use a different C++ compiler.")
+endif()
+
 include_directories(../include)
 include_directories(../thirdparty/yaml-cpp-yaml-cpp-0.5.3/include)
 include_directories(../thirdparty/civetweb-1.9.1/include)
 include_directories(include)
 
-file(GLOB SOURCES "src/*.cpp")
+file(GLOB_RECURSE SOURCES "src/*.cpp")
--- End diff --

Will think conflict with conditional builds like in 
https://github.com/apache/nifi-minifi-cpp/pull/52/files#diff-2672ba1ce10995865ff39a95c67bc5dcR72
 since those features are in fact subdirectories of src?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #52: Support for GPSD integration

[jdye64]

Github user jdye64 commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/52#discussion_r102844031
  
--- Diff: libminifi/test/unit/gps/GetGPSTests.h ---
@@ -0,0 +1,37 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+#ifndef GPS_TESTS
+#define GPS_TESTS
+#include "../../TestBase.h"
+
+#include "../ProvenanceTestHelper.h"
+#include "../Provenance.h"
+#include "../FlowFileRecord.h"
+
+
+TEST_CASE("Test GPSD Create", "[TestGPSEventRecord]"){
--- End diff --

@apiri and @phrocker I'm leaning on you guys for recommendations around the 
tests here. I was planning on using http://www.catb.org/gpsd/gpsfake.html


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #38: MINIFI-175 Create official Apache NiFi MiNiFi C++...

[jdye64]

Github user jdye64 commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/38
  
@apiri I have made some changes. This should solve our problems with 
building and running the docker image.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2876) Refactor TextLineDemarcator and StreamDemarcator into a common abstract class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881368#comment-15881368
 ] 

ASF GitHub Bot commented on NIFI-2876:
--

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102830048
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/AbstractDemarcator.java
 ---
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stream.io.util;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStream;
+
+import org.apache.nifi.stream.io.exception.TokenTooLargeException;
+
+/**
+ * Base class for implementing streaming demarcators.
+ * 
+ * NOTE: Not intended for multi-thread usage hence not Thread-safe.
+ * 
+ */
+abstract class AbstractDemarcator implements Closeable {
+
+final static int INIT_BUFFER_SIZE = 8192;
+
+private final InputStream is;
+
+private final int initialBufferSize;
+
+private final int maxDataSize;
+
+byte[] buffer;
+
+int index;
+
+int mark;
+
+long offset;
+
+int bufferLength;
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size. Each demarcated token must fit within max 
buffer
+ * size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize) {
+this(is, maxDataSize, INIT_BUFFER_SIZE);
+}
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size and initial buffer size. Each demarcated token 
must
+ * fit within max buffer size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize, int 
initialBufferSize) {
+this.validate(is, maxDataSize, initialBufferSize);
+this.is = is;
+this.initialBufferSize = initialBufferSize;
+this.buffer = new byte[initialBufferSize];
+this.maxDataSize = maxDataSize;
+}
+
+@Override
+public void close() throws IOException {
+// noop
+}
+
+/**
+ * Will fill the current buffer from current 'index' position, 
expanding it
+ * and or shuffling it if necessary. If buffer exceeds max buffer size 
a
+ * {@link TokenTooLargeException} will be thrown.
+ *
+ * @throws IOException
+ * if unable to read from the stream
+ */
+void fill() throws IOException {
+if (this.index >= this.buffer.length) {
+if (this.mark == 0) { // expand
+byte[] newBuff = new byte[this.buffer.length + 
this.initialBufferSize];
+System.arraycopy(this.buffer, 0, newBuff, 0, 
this.buffer.length);
+this.buffer = newBuff;
+} else { // shuffle
+int length = this.index - this.mark;
+System.arraycopy(this.buffer, this.mark, this.buffer, 0, 
length);
+this.index = length;
+this.mark = 0;
+}
+}
+
+int bytesRead;
+do {
+bytesRead = this.is.read(this.buffer, this.index, 
this.buffer.length - this.index);
--- End diff --

Good point @mosermw . Mark and I did discuss it earlier and the I kept the 
code as is for exactly that reason. I did add an inline comment to explain the 
do/while loop.


> Refactor TextLineDemarcator and StreamDemarcator into a common abstract class
> -
>
> Key: NIFI-2876
> URL: https://issues.apache.org/jira/browse/NIFI-2876
> Project: Apache NiFi
>   

[GitHub] nifi pull request #1214: NIFI-2876 refactored demarcators into a common abst...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102830048
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/AbstractDemarcator.java
 ---
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stream.io.util;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStream;
+
+import org.apache.nifi.stream.io.exception.TokenTooLargeException;
+
+/**
+ * Base class for implementing streaming demarcators.
+ * 
+ * NOTE: Not intended for multi-thread usage hence not Thread-safe.
+ * 
+ */
+abstract class AbstractDemarcator implements Closeable {
+
+final static int INIT_BUFFER_SIZE = 8192;
+
+private final InputStream is;
+
+private final int initialBufferSize;
+
+private final int maxDataSize;
+
+byte[] buffer;
+
+int index;
+
+int mark;
+
+long offset;
+
+int bufferLength;
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size. Each demarcated token must fit within max 
buffer
+ * size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize) {
+this(is, maxDataSize, INIT_BUFFER_SIZE);
+}
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size and initial buffer size. Each demarcated token 
must
+ * fit within max buffer size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize, int 
initialBufferSize) {
+this.validate(is, maxDataSize, initialBufferSize);
+this.is = is;
+this.initialBufferSize = initialBufferSize;
+this.buffer = new byte[initialBufferSize];
+this.maxDataSize = maxDataSize;
+}
+
+@Override
+public void close() throws IOException {
+// noop
+}
+
+/**
+ * Will fill the current buffer from current 'index' position, 
expanding it
+ * and or shuffling it if necessary. If buffer exceeds max buffer size 
a
+ * {@link TokenTooLargeException} will be thrown.
+ *
+ * @throws IOException
+ * if unable to read from the stream
+ */
+void fill() throws IOException {
+if (this.index >= this.buffer.length) {
+if (this.mark == 0) { // expand
+byte[] newBuff = new byte[this.buffer.length + 
this.initialBufferSize];
+System.arraycopy(this.buffer, 0, newBuff, 0, 
this.buffer.length);
+this.buffer = newBuff;
+} else { // shuffle
+int length = this.index - this.mark;
+System.arraycopy(this.buffer, this.mark, this.buffer, 0, 
length);
+this.index = length;
+this.mark = 0;
+}
+}
+
+int bytesRead;
+do {
+bytesRead = this.is.read(this.buffer, this.index, 
this.buffer.length - this.index);
--- End diff --

Good point @mosermw . Mark and I did discuss it earlier and the I kept the 
code as is for exactly that reason. I did add an inline comment to explain the 
do/while loop.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2876) Refactor TextLineDemarcator and StreamDemarcator into a common abstract class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881360#comment-15881360
 ] 

ASF GitHub Bot commented on NIFI-2876:
--

Github user mosermw commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102828860
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/AbstractDemarcator.java
 ---
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stream.io.util;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStream;
+
+import org.apache.nifi.stream.io.exception.TokenTooLargeException;
+
+/**
+ * Base class for implementing streaming demarcators.
+ * 
+ * NOTE: Not intended for multi-thread usage hence not Thread-safe.
+ * 
+ */
+abstract class AbstractDemarcator implements Closeable {
+
+final static int INIT_BUFFER_SIZE = 8192;
+
+private final InputStream is;
+
+private final int initialBufferSize;
+
+private final int maxDataSize;
+
+byte[] buffer;
+
+int index;
+
+int mark;
+
+long offset;
+
+int bufferLength;
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size. Each demarcated token must fit within max 
buffer
+ * size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize) {
+this(is, maxDataSize, INIT_BUFFER_SIZE);
+}
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size and initial buffer size. Each demarcated token 
must
+ * fit within max buffer size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize, int 
initialBufferSize) {
+this.validate(is, maxDataSize, initialBufferSize);
+this.is = is;
+this.initialBufferSize = initialBufferSize;
+this.buffer = new byte[initialBufferSize];
+this.maxDataSize = maxDataSize;
+}
+
+@Override
+public void close() throws IOException {
+// noop
+}
+
+/**
+ * Will fill the current buffer from current 'index' position, 
expanding it
+ * and or shuffling it if necessary. If buffer exceeds max buffer size 
a
+ * {@link TokenTooLargeException} will be thrown.
+ *
+ * @throws IOException
+ * if unable to read from the stream
+ */
+void fill() throws IOException {
+if (this.index >= this.buffer.length) {
+if (this.mark == 0) { // expand
+byte[] newBuff = new byte[this.buffer.length + 
this.initialBufferSize];
+System.arraycopy(this.buffer, 0, newBuff, 0, 
this.buffer.length);
+this.buffer = newBuff;
+} else { // shuffle
+int length = this.index - this.mark;
+System.arraycopy(this.buffer, this.mark, this.buffer, 0, 
length);
+this.index = length;
+this.mark = 0;
+}
+}
+
+int bytesRead;
+do {
+bytesRead = this.is.read(this.buffer, this.index, 
this.buffer.length - this.index);
--- End diff --

Generic InputStreams that are backed by a network socket implementation 
actually can return 0 temporarily, before continuing to return good data.  Not 
sure if this class will ever be used in that situation, though.


> Refactor TextLineDemarcator and StreamDemarcator into a common abstract class
> -
>
> Key: NIFI-2876
> URL: 

[GitHub] nifi pull request #1214: NIFI-2876 refactored demarcators into a common abst...

[mosermw]

Github user mosermw commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102828860
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/AbstractDemarcator.java
 ---
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stream.io.util;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStream;
+
+import org.apache.nifi.stream.io.exception.TokenTooLargeException;
+
+/**
+ * Base class for implementing streaming demarcators.
+ * 
+ * NOTE: Not intended for multi-thread usage hence not Thread-safe.
+ * 
+ */
+abstract class AbstractDemarcator implements Closeable {
+
+final static int INIT_BUFFER_SIZE = 8192;
+
+private final InputStream is;
+
+private final int initialBufferSize;
+
+private final int maxDataSize;
+
+byte[] buffer;
+
+int index;
+
+int mark;
+
+long offset;
+
+int bufferLength;
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size. Each demarcated token must fit within max 
buffer
+ * size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize) {
+this(is, maxDataSize, INIT_BUFFER_SIZE);
+}
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size and initial buffer size. Each demarcated token 
must
+ * fit within max buffer size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize, int 
initialBufferSize) {
+this.validate(is, maxDataSize, initialBufferSize);
+this.is = is;
+this.initialBufferSize = initialBufferSize;
+this.buffer = new byte[initialBufferSize];
+this.maxDataSize = maxDataSize;
+}
+
+@Override
+public void close() throws IOException {
+// noop
+}
+
+/**
+ * Will fill the current buffer from current 'index' position, 
expanding it
+ * and or shuffling it if necessary. If buffer exceeds max buffer size 
a
+ * {@link TokenTooLargeException} will be thrown.
+ *
+ * @throws IOException
+ * if unable to read from the stream
+ */
+void fill() throws IOException {
+if (this.index >= this.buffer.length) {
+if (this.mark == 0) { // expand
+byte[] newBuff = new byte[this.buffer.length + 
this.initialBufferSize];
+System.arraycopy(this.buffer, 0, newBuff, 0, 
this.buffer.length);
+this.buffer = newBuff;
+} else { // shuffle
+int length = this.index - this.mark;
+System.arraycopy(this.buffer, this.mark, this.buffer, 0, 
length);
+this.index = length;
+this.mark = 0;
+}
+}
+
+int bytesRead;
+do {
+bytesRead = this.is.read(this.buffer, this.index, 
this.buffer.length - this.index);
--- End diff --

Generic InputStreams that are backed by a network socket implementation 
actually can return 0 temporarily, before continuing to return good data.  Not 
sure if this class will ever be used in that situation, though.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-3484) GenerateTableFetch Should Allow for Right Boundary

[Peter Wicks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Wicks updated NIFI-3484:
--
Fix Version/s: (was: 1.2.0)

> GenerateTableFetch Should Allow for Right Boundary
> --
>
> Key: NIFI-3484
> URL: https://issues.apache.org/jira/browse/NIFI-3484
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Peter Wicks
>Assignee: Peter Wicks
>Priority: Minor
>
> When using GenerateTableFetch it places no right hand boundary on pages of 
> data.  This can lead to issues when the statement says to get the next 1000 
> records greater then a specific key, but records were added to the table 
> between the time the processor executed and when the SQL is being executed. 
> As a result it pulls in records that did not exist when the processor was 
> run.  On the next execution of the processor these records will be pulled in 
> a second time.
> Example:
> Partition Size = 1000
> First run (no state): Count(*)=4700 and MAX(ID)=4700.
> 5 FlowFiles are generated, the last one will say to fetch 1000, not 700. (But 
> I don't think this is really a bug, just an observation).
> 5 Flow Files are now in queue to be executed by ExecuteSQL.  Before the 5th 
> file can execute 400 new rows are added to the table.  When the final SQL 
> statement is executed 300 extra records, with higher ID values, will also be 
> pulled into NiFi.
> Second run (state: ID=4700).  Count(*) ID>4700 = 400 and MAX(ID)=5100.
> 1 Flow File is generated, but includes 300 records already pulled into NiFI.
> The solution is to have an optional property that will let users use the new 
> MAX(ID) as a right boundary when generating queries.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3494) GenerateTableFetch Some JDBC Drivers Have Trouble With Name vs Label

[Peter Wicks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Wicks updated NIFI-3494:
--
Fix Version/s: (was: 1.2.0)

> GenerateTableFetch Some JDBC Drivers Have Trouble With Name vs Label
> 
>
> Key: NIFI-3494
> URL: https://issues.apache.org/jira/browse/NIFI-3494
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Peter Wicks
>Assignee: Peter Wicks
>Priority: Minor
>
> When GenerateTableFetch retrieves the COUNT(*) and the Max Values for the 
> columns, some JDBC drivers (specifically SAP HANA) in this case have trouble 
> interpreting the name of the column if the "table" is actually a view.
> Since all max value columns are aliased with the original column name I'm 
> going to change the code to try Label first, and fall back to Name if it's 
> not there.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-2829) PutSQL assumes all Date and Time values are provided in Epoch

[Peter Wicks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Wicks updated NIFI-2829:
--
Fix Version/s: (was: 1.2.0)

> PutSQL assumes all Date and Time values are provided in Epoch
> -
>
> Key: NIFI-2829
> URL: https://issues.apache.org/jira/browse/NIFI-2829
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Paul Gibeault
>Assignee: Peter Wicks
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> This bug is the same as NIFI-2576 only extended to data types DATE and TIME.
> https://issues.apache.org/jira/browse/NIFI-2576
> When PutSQL sees a DATE or TIME data type it assumes that it's being provided 
> as a Long in Epoch format.
> This doesn't make much sense since the Query Database tools that return Avro 
> return DATES and TIME values as strings; and thus following the 
> Avro->JSON->JSON To SQL Route leads to DATE and TIME fields as being strings.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3481) DB Adapter for MS SQL

[Peter Wicks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Wicks updated NIFI-3481:
--
Fix Version/s: (was: 1.2.0)

> DB Adapter for MS SQL
> -
>
> Key: NIFI-3481
> URL: https://issues.apache.org/jira/browse/NIFI-3481
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Peter Wicks
>Assignee: Peter Wicks
>Priority: Minor
>
> Adding a new DB Adapter for MS SQL.  MS SQL does paging of queries 
> differently then other RDBMS', including a lack of "LIMIT".
> This adds functionality for TOP and OFFSET/FETCH for paging of results with 
> the GenerateTableFetch. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (NIFI-3526) Update secure site-to-site section of User Guide that references Access Control

[Andrew Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Lim reassigned NIFI-3526:


Assignee: Andrew Lim

> Update secure site-to-site section of User Guide that references Access 
> Control
> ---
>
> Key: NIFI-3526
> URL: https://issues.apache.org/jira/browse/NIFI-3526
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>
> The Access Control paragraph in the "Configure Site-to-Site server NiFi 
> instance" section of the User Guide [1] needs to be updated for 1.x 
> functionality.
> [1] 
> https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#configure-site-to-site-server-nifi-instance



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3481) DB Adapter for MS SQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881256#comment-15881256
 ] 

ASF GitHub Bot commented on NIFI-3481:
--

Github user patricker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1510#discussion_r102817584
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/MSSQLDatabaseAdapter.java
 ---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db.impl;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.processors.standard.db.DatabaseAdapter;
+
+/**
+ * A database adapter that generates MS SQL Compatible SQL.
+ */
+public class MSSQLDatabaseAdapter implements DatabaseAdapter {
+@Override
+public String getName() {
+return "MS SQL";
+}
+
+@Override
+public String getDescription() {
+return "Generates MS SQL Compatible SQL, for version 2012 or 
greater";
+}
+
+@Override
+public String getSelectStatement(String tableName, String columnNames, 
String whereClause, String orderByClause, Long limit, Long offset) {
+if (StringUtils.isEmpty(tableName)) {
+throw new IllegalArgumentException("Table name cannot be null 
or empty");
+}
+final StringBuilder query = new StringBuilder("SELECT ");
+
+//If this is a limit query and not a paging query then use TOP in 
MS SQL
+if (limit != null && offset == null){
+query.append("TOP ");
--- End diff --

@SQLGuyChuck `TOP` in this case is supposed to emulate a `FETCH` with 
`OFFSET` 0. In order to maintain that functionality I think WITH TIES would 
probably have unintended consequences, including data duplication in the next 
page of data (the tied record(s) would be in page 1 and it would also be the 
first record(s) in page 2).

Thoughts?

Also, if we wanted to add TOP WITH TIES style functionality in general that 
would be a separate ticket, as we'd need to touch every DatabaseAdapter (since 
they all do top/limit differently).


> DB Adapter for MS SQL
> -
>
> Key: NIFI-3481
> URL: https://issues.apache.org/jira/browse/NIFI-3481
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Peter Wicks
>Assignee: Peter Wicks
>Priority: Minor
> Fix For: 1.2.0
>
>
> Adding a new DB Adapter for MS SQL.  MS SQL does paging of queries 
> differently then other RDBMS', including a lack of "LIMIT".
> This adds functionality for TOP and OFFSET/FETCH for paging of results with 
> the GenerateTableFetch. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1510: NIFI-3481 DB Adapter for MS SQL

[patricker]

Github user patricker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1510#discussion_r102817584
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/MSSQLDatabaseAdapter.java
 ---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db.impl;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.processors.standard.db.DatabaseAdapter;
+
+/**
+ * A database adapter that generates MS SQL Compatible SQL.
+ */
+public class MSSQLDatabaseAdapter implements DatabaseAdapter {
+@Override
+public String getName() {
+return "MS SQL";
+}
+
+@Override
+public String getDescription() {
+return "Generates MS SQL Compatible SQL, for version 2012 or 
greater";
+}
+
+@Override
+public String getSelectStatement(String tableName, String columnNames, 
String whereClause, String orderByClause, Long limit, Long offset) {
+if (StringUtils.isEmpty(tableName)) {
+throw new IllegalArgumentException("Table name cannot be null 
or empty");
+}
+final StringBuilder query = new StringBuilder("SELECT ");
+
+//If this is a limit query and not a paging query then use TOP in 
MS SQL
+if (limit != null && offset == null){
+query.append("TOP ");
--- End diff --

@SQLGuyChuck `TOP` in this case is supposed to emulate a `FETCH` with 
`OFFSET` 0. In order to maintain that functionality I think WITH TIES would 
probably have unintended consequences, including data duplication in the next 
page of data (the tied record(s) would be in page 1 and it would also be the 
first record(s) in page 2).

Thoughts?

Also, if we wanted to add TOP WITH TIES style functionality in general that 
would be a separate ticket, as we'd need to touch every DatabaseAdapter (since 
they all do top/limit differently).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #54: MINIFI-206: Implement Extensible Comm Mechanism

[apiri]

Github user apiri commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/54
  
reviewing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2941) Clicking the relationship name should toggle the checkbox status in processor settings

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan updated NIFI-2941:
--
Status: Patch Available  (was: In Progress)

> Clicking the relationship name should toggle the checkbox status in processor 
> settings
> --
>
> Key: NIFI-2941
> URL: https://issues.apache.org/jira/browse/NIFI-2941
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Andy LoPresto
>Assignee: Scott Aslan
>Priority: Trivial
>  Labels: beginner, ui
>
> When configuring a processor, the available relationships 
> ({{success}}/{{failure}}, {{matched}}/{{unmatched}}, dynamic relationships) 
> can be selected to auto-terminate. However, clicking the label does not 
> toggle the appropriate checkbox. Only clicks within the checkbox are 
> recognized. To improve the user experience, I suggest increasing the target 
> area to include the labels as many web forms do. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2941) Clicking the relationship name should toggle the checkbox status in processor settings

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881205#comment-15881205
 ] 

ASF GitHub Bot commented on NIFI-2941:
--

GitHub user scottyaslan opened a pull request:

https://github.com/apache/nifi/pull/1538

[NIFI-2941] allow users to click checkbox labels to toggle the checkbox

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/scottyaslan/nifi NIFI-2941

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1538.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1538






> Clicking the relationship name should toggle the checkbox status in processor 
> settings
> --
>
> Key: NIFI-2941
> URL: https://issues.apache.org/jira/browse/NIFI-2941
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Andy LoPresto
>Assignee: Scott Aslan
>Priority: Trivial
>  Labels: beginner, ui
>
> When configuring a processor, the available relationships 
> ({{success}}/{{failure}}, {{matched}}/{{unmatched}}, dynamic relationships) 
> can be selected to auto-terminate. However, clicking the label does not 
> toggle the appropriate checkbox. Only clicks within the checkbox are 
> recognized. To improve the user experience, I suggest increasing the target 
> area to include the labels as many web forms do. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3426) Add ability to set JDBC properties via dynamic properties in DBCPConnectionPool

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-3426:
---
Fix Version/s: 1.2.0

> Add ability to set JDBC properties via dynamic properties in 
> DBCPConnectionPool
> ---
>
> Key: NIFI-3426
> URL: https://issues.apache.org/jira/browse/NIFI-3426
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Minor
> Fix For: 1.2.0
>
>
> Although many JDBC drivers allow for the setting of connection-specific 
> properties via the JDBC URL, some JDBC drivers only accept specific 
> configuration properties for a connection via programmatic calls to add 
> properties to the Connection object; those same properties will not be parsed 
> from the URL.
> In general, a proposed improvement is to add support for Dynamic Properties 
> to the DBCPConnectionPool controller service. Each dynamic property would be 
> set as a property on the DataSource when the controller service is enabled. 
> Note that Expression Language can be supported, but as no flow file will be 
> available at enable-time, flow file attributes cannot be used in EL 
> expressions. However the Variable Registry could be leveraged to provide 
> different values in different environments (dev, test, production, e.g.).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1538: [NIFI-2941] allow users to click checkbox labels to...

[scottyaslan]

GitHub user scottyaslan opened a pull request:

https://github.com/apache/nifi/pull/1538

[NIFI-2941] allow users to click checkbox labels to toggle the checkbox

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/scottyaslan/nifi NIFI-2941

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1538.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1538






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-3268) Add AUTO_INCREMENT column in GenerateTableFetch to benefit index

[qfdk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

qfdk updated NIFI-3268:
---
Status: Open  (was: Patch Available)

> Add AUTO_INCREMENT column in GenerateTableFetch to benefit index
> 
>
> Key: NIFI-3268
> URL: https://issues.apache.org/jira/browse/NIFI-3268
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.1.1
> Environment: - ubuntu 16.04
> - java version "1.8.0_111"
> - Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
> - Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
>Reporter: qfdk
>  Labels: easyfix
> Fix For: 1.2.0
>
>
> I added AUTO_INCREMENT column in  GenerateTableFetch to benefit index column
> By default this processor uses OFFSET, i have  problems with large data. 
> somme column has index so we could use index to speed up query time.
> I posted question here :
> https://community.hortonworks.com/questions/72586/how-can-i-use-an-array-with-putelasticsearch.html
> If you indexed un column (id), you could use this sql
> ```
> select xxx
> From x
> where 20=>id
> order by id
> limit 20
> ```
> “OFFSET is bad for skipping previous rows.” [Online]. Available: 
> http://Use-The-Index-Luke.com/sql/partial-results/fetch-next-page. [Accessed: 
> 27-Dec-2016].
> Thank you in advance



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3268) Add AUTO_INCREMENT column in GenerateTableFetch to benefit index

[qfdk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

qfdk updated NIFI-3268:
---
Status: Patch Available  (was: Open)

> Add AUTO_INCREMENT column in GenerateTableFetch to benefit index
> 
>
> Key: NIFI-3268
> URL: https://issues.apache.org/jira/browse/NIFI-3268
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.1.1
> Environment: - ubuntu 16.04
> - java version "1.8.0_111"
> - Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
> - Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
>Reporter: qfdk
>  Labels: easyfix
> Fix For: 1.2.0
>
>
> I added AUTO_INCREMENT column in  GenerateTableFetch to benefit index column
> By default this processor uses OFFSET, i have  problems with large data. 
> somme column has index so we could use index to speed up query time.
> I posted question here :
> https://community.hortonworks.com/questions/72586/how-can-i-use-an-array-with-putelasticsearch.html
> If you indexed un column (id), you could use this sql
> ```
> select xxx
> From x
> where 20=>id
> order by id
> limit 20
> ```
> “OFFSET is bad for skipping previous rows.” [Online]. Available: 
> http://Use-The-Index-Luke.com/sql/partial-results/fetch-next-page. [Accessed: 
> 27-Dec-2016].
> Thank you in advance



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (NIFI-3527) ExecuteScript body editor is not multiline and does not accept ctrl+enter

[dmitry zakharov (JIRA)]

dmitry zakharov created NIFI-3527:
-

 Summary: ExecuteScript body editor is not multiline and does not 
accept ctrl+enter
 Key: NIFI-3527
 URL: https://issues.apache.org/jira/browse/NIFI-3527
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core UI
Affects Versions: 1.1.1
Reporter: dmitry zakharov






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3526) Update secure site-to-site section of User Guide that references Access Control

[Andrew Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Lim updated NIFI-3526:
-
Summary: Update secure site-to-site section of User Guide that references 
Access Control  (was: Update secure site-to-site section of User Guide)

> Update secure site-to-site section of User Guide that references Access 
> Control
> ---
>
> Key: NIFI-3526
> URL: https://issues.apache.org/jira/browse/NIFI-3526
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Andrew Lim
>
> The Access Control paragraph in the "Configure Site-to-Site server NiFi 
> instance" section of the User Guide [1] needs to be updated for 1.x 
> functionality.
> [1] 
> https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#configure-site-to-site-server-nifi-instance



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (NIFI-3526) Update secure site-to-site section of User Guide

[Andrew Lim (JIRA)]

Andrew Lim created NIFI-3526:


 Summary: Update secure site-to-site section of User Guide
 Key: NIFI-3526
 URL: https://issues.apache.org/jira/browse/NIFI-3526
 Project: Apache NiFi
  Issue Type: Bug
  Components: Documentation & Website
Affects Versions: 1.0.0
Reporter: Andrew Lim


The Access Control paragraph in the "Configure Site-to-Site server NiFi 
instance" section of the User Guide [1] needs to be updated for 1.x 
functionality.

[1] 
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#configure-site-to-site-server-nifi-instance



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3525) QueryDatabaseTable processor adds table name to output FlowFile Attributes

[Ambud Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ambud Sharma updated NIFI-3525:
---
Summary: QueryDatabaseTable processor adds table name to output FlowFile 
Attributes  (was: QueryDatabaseTable processor adds table and column names to 
output FlowFile Attributes)

> QueryDatabaseTable processor adds table name to output FlowFile Attributes
> --
>
> Key: NIFI-3525
> URL: https://issues.apache.org/jira/browse/NIFI-3525
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Ambud Sharma
>
> QueryDatabaseTable processor adds table name to output FlowFile Attributes to 
> avoid having to use another processor (updateattribute) to add table name for 
> routing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-3525) QueryDatabaseTable processor adds table and column names to output FlowFile Attributes

[Ambud Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ambud Sharma updated NIFI-3525:
---
Description: QueryDatabaseTable processor adds table name to output 
FlowFile Attributes to avoid having to use another processor (updateattribute) 
to add table name for routing.  (was: QueryDatabaseTable processor adds table 
and column names to output FlowFile Attributes to avoid having to use another 
processor (updateattribute) to add table name for routing.)

> QueryDatabaseTable processor adds table and column names to output FlowFile 
> Attributes
> --
>
> Key: NIFI-3525
> URL: https://issues.apache.org/jira/browse/NIFI-3525
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Ambud Sharma
>
> QueryDatabaseTable processor adds table name to output FlowFile Attributes to 
> avoid having to use another processor (updateattribute) to add table name for 
> routing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3525) QueryDatabaseTable processor adds table name to output FlowFile Attributes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881027#comment-15881027
 ] 

ASF GitHub Bot commented on NIFI-3525:
--

GitHub user ambud opened a pull request:

https://github.com/apache/nifi/pull/1537

Add table attributes to flow files

https://issues.apache.org/jira/browse/NIFI-3525

Adding column names only



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ambud/nifi master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1537.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1537


commit c19e8997d71067a079c8b2d08278fd3bf9bc35c5
Author: ambud 
Date:   2017-02-23T18:51:58Z

Add table attributes to flow files
https://issues.apache.org/jira/browse/NIFI-3525

Adding column names only




> QueryDatabaseTable processor adds table name to output FlowFile Attributes
> --
>
> Key: NIFI-3525
> URL: https://issues.apache.org/jira/browse/NIFI-3525
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Ambud Sharma
>
> QueryDatabaseTable processor adds table name to output FlowFile Attributes to 
> avoid having to use another processor (updateattribute) to add table name for 
> routing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1537: Add table attributes to flow files

[ambud]

GitHub user ambud opened a pull request:

https://github.com/apache/nifi/pull/1537

Add table attributes to flow files

https://issues.apache.org/jira/browse/NIFI-3525

Adding column names only



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ambud/nifi master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1537.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1537


commit c19e8997d71067a079c8b2d08278fd3bf9bc35c5
Author: ambud 
Date:   2017-02-23T18:51:58Z

Add table attributes to flow files
https://issues.apache.org/jira/browse/NIFI-3525

Adding column names only




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-3525) QueryDatabaseTable processor adds table and column names to output FlowFile Attributes

[Ambud Sharma (JIRA)]

Ambud Sharma created NIFI-3525:
--

 Summary: QueryDatabaseTable processor adds table and column names 
to output FlowFile Attributes
 Key: NIFI-3525
 URL: https://issues.apache.org/jira/browse/NIFI-3525
 Project: Apache NiFi
  Issue Type: New Feature
Reporter: Ambud Sharma


QueryDatabaseTable processor adds table and column names to output FlowFile 
Attributes to avoid having to use another processor (updateattribute) to add 
table name for routing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (NIFI-3509) Creating templates of copied and pasted PGs can cause issues

[Oleg Zhurakousky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Zhurakousky reassigned NIFI-3509:
--

Assignee: Oleg Zhurakousky

> Creating templates of copied and pasted PGs can cause issues 
> -
>
> Key: NIFI-3509
> URL: https://issues.apache.org/jira/browse/NIFI-3509
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.1.1
>Reporter: Joseph Percivall
>Assignee: Oleg Zhurakousky
>Priority: Critical
> Attachments: Screen Shot 2017-02-20 at 7.38.49 PM.png, Screen Shot 
> 2017-02-20 at 7.53.21 PM.png, Screen Shot 2017-02-20 at 7.53.34 PM.png
>
>
> Using the following steps I was able to create a template which instantiates 
> with connections from one input port to another input port of another PG. See 
> the template here[1]. 
> See the first attached screenshot for the seed flow.  After importing, it 
> originally gives an error "Cannot add Connection to Process Group because its 
> destination does not belong to this Process Group" and nothing appears to be 
> added but after I refresh the canvas parts of the original flow appear 
> (screenshot 2 and 3).
> What is most odd, is that there are connections going from the input port 
> "input" of pg "test" to the input ports of other PGs. I didn't think that was 
> possible.
> The steps I used to create it:
> 1: create a PG "test" with input port "input",  UpdateAttribute and output 
> port "output". Creating connections between them.
> 2: Go to Parent group and copy "test".
> 3: Go back into "test" and paste.
> 4: Copy and paste "test" multiple times, creating the flow.
> 5: Create a template of the flow
> 6: When using the template, see the issue described.
> The root cause of the issues, I believe, is that the id for the parent group 
> starts with "f3ac1eb5-1010-115a" and when I copied and pasted "test" into 
> itself it created the "test" pg with an id starting with the same value.
> [1] https://gist.github.com/JPercivall/923519ef09500886c50a8f574f7662ca



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2876) Refactor TextLineDemarcator and StreamDemarcator into a common abstract class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880939#comment-15880939
 ] 

ASF GitHub Bot commented on NIFI-2876:
--

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102775809
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/AbstractDemarcator.java
 ---
@@ -98,22 +136,26 @@ void fill() throws IOException {
 }
 
 int bytesRead;
+/*
+ * The do/while pattern is used here similar to the way it is used 
in
+ * BufferedReader essentially protecting from assuming the EOS 
until it
+ * actually is since not every implementation of InputStream 
guarantees
+ * that bytes are always available while the stream is open.
+ */
 do {
 bytesRead = this.is.read(this.buffer, this.index, 
this.buffer.length - this.index);
 } while (bytesRead == 0);
-this.bufferLength = bytesRead != -1 ? this.index + bytesRead : -1;
-if (this.bufferLength > this.maxDataSize) {
+this.availableBytesLength = bytesRead != -1 ? this.index + 
bytesRead : -1;
+if (this.availableBytesLength > this.maxDataSize) {
--- End diff --

Hmm, good catch. It should only be compared to a token as it's being 
computed, will fix



> Refactor TextLineDemarcator and StreamDemarcator into a common abstract class
> -
>
> Key: NIFI-2876
> URL: https://issues.apache.org/jira/browse/NIFI-2876
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Oleg Zhurakousky
>Assignee: Oleg Zhurakousky
>Priority: Minor
> Fix For: 1.2.0
>
>
> Based on the work that has been performed as part of the NIFI-2851 we now 
> have a new class with a significantly faster logic to perform demarcation of 
> the InputStream (TextLineDemarcator). This new class's initial starting point 
> was the existing LineDemarcator. They both now share ~60-70% of common code 
> which would be important to extract into a common abstract class as well as 
> incorporate the new (faster) demarcation logic int StreamDemarcator.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1510: NIFI-3481 DB Adapter for MS SQL

[SQLGuyChuck]

Github user SQLGuyChuck commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1510#discussion_r102772033
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/MSSQLDatabaseAdapter.java
 ---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db.impl;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.processors.standard.db.DatabaseAdapter;
+
+/**
+ * A database adapter that generates MS SQL Compatible SQL.
+ */
+public class MSSQLDatabaseAdapter implements DatabaseAdapter {
+@Override
+public String getName() {
+return "MS SQL";
+}
+
+@Override
+public String getDescription() {
+return "Generates MS SQL Compatible SQL, for version 2012 or 
greater";
+}
+
+@Override
+public String getSelectStatement(String tableName, String columnNames, 
String whereClause, String orderByClause, Long limit, Long offset) {
+if (StringUtils.isEmpty(tableName)) {
+throw new IllegalArgumentException("Table name cannot be null 
or empty");
+}
+final StringBuilder query = new StringBuilder("SELECT ");
+
+//If this is a limit query and not a paging query then use TOP in 
MS SQL
+if (limit != null && offset == null){
+query.append("TOP ");
--- End diff --

I think something to consider is potentially adding WITH TIES option as by 
default, it won't show tie rows, I think this is true of most all of the RDMSs. 
http://stackoverflow.com/questions/9629953/postgresql-equivalent-for-top-n-with-ties-limit-with-ties
 is related question on this subject.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1214: NIFI-2876 refactored demarcators into a common abst...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102775809
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/AbstractDemarcator.java
 ---
@@ -98,22 +136,26 @@ void fill() throws IOException {
 }
 
 int bytesRead;
+/*
+ * The do/while pattern is used here similar to the way it is used 
in
+ * BufferedReader essentially protecting from assuming the EOS 
until it
+ * actually is since not every implementation of InputStream 
guarantees
+ * that bytes are always available while the stream is open.
+ */
 do {
 bytesRead = this.is.read(this.buffer, this.index, 
this.buffer.length - this.index);
 } while (bytesRead == 0);
-this.bufferLength = bytesRead != -1 ? this.index + bytesRead : -1;
-if (this.bufferLength > this.maxDataSize) {
+this.availableBytesLength = bytesRead != -1 ? this.index + 
bytesRead : -1;
+if (this.availableBytesLength > this.maxDataSize) {
--- End diff --

Hmm, good catch. It should only be compared to a token as it's being 
computed, will fix



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2876) Refactor TextLineDemarcator and StreamDemarcator into a common abstract class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880935#comment-15880935
 ] 

ASF GitHub Bot commented on NIFI-2876:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102774788
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/AbstractDemarcator.java
 ---
@@ -98,22 +136,26 @@ void fill() throws IOException {
 }
 
 int bytesRead;
+/*
+ * The do/while pattern is used here similar to the way it is used 
in
+ * BufferedReader essentially protecting from assuming the EOS 
until it
+ * actually is since not every implementation of InputStream 
guarantees
+ * that bytes are always available while the stream is open.
+ */
 do {
 bytesRead = this.is.read(this.buffer, this.index, 
this.buffer.length - this.index);
 } while (bytesRead == 0);
-this.bufferLength = bytesRead != -1 ? this.index + bytesRead : -1;
-if (this.bufferLength > this.maxDataSize) {
+this.availableBytesLength = bytesRead != -1 ? this.index + 
bytesRead : -1;
+if (this.availableBytesLength > this.maxDataSize) {
--- End diff --

This logic seems wrong to me. It is saying 'If we have more data available 
in the buffer than the largest token that is allowed, we should fail?? The 
following unit test fails, though I believe it should pass:

```
@Test
public void testLargeBufferSmallMaxSize() throws IOException {
final byte[] inputData = "A Great Benefit To Us 
All".getBytes(StandardCharsets.UTF_8);

try (final InputStream is = new ByteArrayInputStream(inputData);
final StreamDemarcator demarcator = new StreamDemarcator(is, 
"B".getBytes(StandardCharsets.UTF_8), 24, 4096)) {

final byte[] first = demarcator.nextToken();
assertNotNull(first);
assertEquals("A Great ", new String(first));

final byte[] second = demarcator.nextToken();
assertNotNull(second);
assertEquals("enefit To Us All", new String(second));

assertNull(demarcator.nextToken());

}
}
```


> Refactor TextLineDemarcator and StreamDemarcator into a common abstract class
> -
>
> Key: NIFI-2876
> URL: https://issues.apache.org/jira/browse/NIFI-2876
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Oleg Zhurakousky
>Assignee: Oleg Zhurakousky
>Priority: Minor
> Fix For: 1.2.0
>
>
> Based on the work that has been performed as part of the NIFI-2851 we now 
> have a new class with a significantly faster logic to perform demarcation of 
> the InputStream (TextLineDemarcator). This new class's initial starting point 
> was the existing LineDemarcator. They both now share ~60-70% of common code 
> which would be important to extract into a common abstract class as well as 
> incorporate the new (faster) demarcation logic int StreamDemarcator.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1214: NIFI-2876 refactored demarcators into a common abst...

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102774788
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/AbstractDemarcator.java
 ---
@@ -98,22 +136,26 @@ void fill() throws IOException {
 }
 
 int bytesRead;
+/*
+ * The do/while pattern is used here similar to the way it is used 
in
+ * BufferedReader essentially protecting from assuming the EOS 
until it
+ * actually is since not every implementation of InputStream 
guarantees
+ * that bytes are always available while the stream is open.
+ */
 do {
 bytesRead = this.is.read(this.buffer, this.index, 
this.buffer.length - this.index);
 } while (bytesRead == 0);
-this.bufferLength = bytesRead != -1 ? this.index + bytesRead : -1;
-if (this.bufferLength > this.maxDataSize) {
+this.availableBytesLength = bytesRead != -1 ? this.index + 
bytesRead : -1;
+if (this.availableBytesLength > this.maxDataSize) {
--- End diff --

This logic seems wrong to me. It is saying 'If we have more data available 
in the buffer than the largest token that is allowed, we should fail?? The 
following unit test fails, though I believe it should pass:

```
@Test
public void testLargeBufferSmallMaxSize() throws IOException {
final byte[] inputData = "A Great Benefit To Us 
All".getBytes(StandardCharsets.UTF_8);

try (final InputStream is = new ByteArrayInputStream(inputData);
final StreamDemarcator demarcator = new StreamDemarcator(is, 
"B".getBytes(StandardCharsets.UTF_8), 24, 4096)) {

final byte[] first = demarcator.nextToken();
assertNotNull(first);
assertEquals("A Great ", new String(first));

final byte[] second = demarcator.nextToken();
assertNotNull(second);
assertEquals("enefit To Us All", new String(second));

assertNull(demarcator.nextToken());

}
}
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3481) DB Adapter for MS SQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880908#comment-15880908
 ] 

ASF GitHub Bot commented on NIFI-3481:
--

Github user SQLGuyChuck commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1510#discussion_r102772033
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/MSSQLDatabaseAdapter.java
 ---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db.impl;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.processors.standard.db.DatabaseAdapter;
+
+/**
+ * A database adapter that generates MS SQL Compatible SQL.
+ */
+public class MSSQLDatabaseAdapter implements DatabaseAdapter {
+@Override
+public String getName() {
+return "MS SQL";
+}
+
+@Override
+public String getDescription() {
+return "Generates MS SQL Compatible SQL, for version 2012 or 
greater";
+}
+
+@Override
+public String getSelectStatement(String tableName, String columnNames, 
String whereClause, String orderByClause, Long limit, Long offset) {
+if (StringUtils.isEmpty(tableName)) {
+throw new IllegalArgumentException("Table name cannot be null 
or empty");
+}
+final StringBuilder query = new StringBuilder("SELECT ");
+
+//If this is a limit query and not a paging query then use TOP in 
MS SQL
+if (limit != null && offset == null){
+query.append("TOP ");
--- End diff --

I think something to consider is potentially adding WITH TIES option as by 
default, it won't show tie rows, I think this is true of most all of the RDMSs. 
http://stackoverflow.com/questions/9629953/postgresql-equivalent-for-top-n-with-ties-limit-with-ties
 is related question on this subject.


> DB Adapter for MS SQL
> -
>
> Key: NIFI-3481
> URL: https://issues.apache.org/jira/browse/NIFI-3481
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Peter Wicks
>Assignee: Peter Wicks
>Priority: Minor
> Fix For: 1.2.0
>
>
> Adding a new DB Adapter for MS SQL.  MS SQL does paging of queries 
> differently then other RDBMS', including a lack of "LIMIT".
> This adds functionality for TOP and OFFSET/FETCH for paging of results with 
> the GenerateTableFetch. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi-minifi-cpp pull request #52: Support for GPSD integration

[apiri]

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/52#discussion_r102765909
  
--- Diff: libminifi/test/unit/gps/GetGPSTests.h ---
@@ -0,0 +1,37 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+#ifndef GPS_TESTS
+#define GPS_TESTS
+#include "../../TestBase.h"
+
+#include "../ProvenanceTestHelper.h"
+#include "../Provenance.h"
+#include "../FlowFileRecord.h"
+
+
+TEST_CASE("Test GPSD Create", "[TestGPSEventRecord]"){
--- End diff --

Might make sense to roll in a mocking framework?  
https://github.com/eranpeer/FakeIt seems similarly lightweight to catch.  
@phrocker any recommendations or preferences?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #52: Support for GPSD integration

[apiri]

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/52#discussion_r102765612
  
--- Diff: README.md ---
@@ -110,6 +118,9 @@ $ brew install cmake \
   leveldb \
   ossp-uuid \
   boost \ openssl
+
+# If building with GPS support
+$ brew install gps
--- End diff --

Hmm... gps doesn't show up as a package for me in brew.  
```
brew search gps
bgpstream gpsbabel  gpsd ✔gpsim 
osm-gps-map
Caskroom/cask/gpsdump
```

Am I overlooking something?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #52: Support for GPSD integration

[jdye64]

Github user jdye64 commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/52#discussion_r102765412
  
--- Diff: libminifi/src/FlowController.cpp ---
@@ -289,6 +289,12 @@ Processor 
*FlowControllerImpl::createProcessor(std::string name, uuid_t uuid) {
return NULL;
}
 
+   #ifdef BUILD_GPS
--- End diff --

This was not very smart of me ... this will cause an error to be thrown 
that the processor of type GetGPS does not exist at runtime because the last 
else condition will be triggered before this. I need to move this higher up the 
condition before the last else.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #52: Support for GPSD integration

[jdye64]

Github user jdye64 commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/52#discussion_r102764886
  
--- Diff: README.md ---
@@ -110,6 +118,9 @@ $ brew install cmake \
   leveldb \
   ossp-uuid \
   boost \ openssl
+
+# If building with GPS support
+$ brew install gps
--- End diff --

Its actually is just gps for OS X. Its the gps.h headers and such that is 
needed. However it might make sense to also install gpsd so I'll add that


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #52: Support for GPSD integration

[apiri]

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/52#discussion_r102764856
  
--- Diff: README.md ---
@@ -91,6 +93,9 @@ $ yum install cmake \
   leveldb-devel leveldb \
   libuuid libuuid-devel \
   boost-devel \ libssl-dev
+
+# If building with GPS support
+$ yum install libgps-devel
--- End diff --

Believe this should gpsd-devel for yum based distros


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #52: Support for GPSD integration

[apiri]

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/52#discussion_r102764070
  
--- Diff: libminifi/src/gps/GetGPS.cpp ---
@@ -0,0 +1,158 @@
+/**
+ * @file GetGPS.cpp
+ * GetGPS class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include "TimeUtil.h"
+#include "GetGPS.h"
+#include "ProcessContext.h"
+#include "ProcessSession.h"
+
+#include 
--- End diff --

Having issues on OS X 10.11 with the following:
```
In file included from 
/Users/apiri/Development/code/nifi-minifi-cpp/libminifi/src/gps/GetGPS.cpp:43:
In file included from /usr/local/include/libgpsmm.h:12:
/usr/local/include/gps.h:1887:8: error: definition of type 'policy_t' 
conflicts with
  typedef of the same name
struct policy_t {
   ^
/usr/include/mach/policy.h:79:17: note: 'policy_t' declared here
typedef int policy_t;
^
In file included from 
/Users/apiri/Development/code/nifi-minifi-cpp/libminifi/src/gps/GetGPS.cpp:43:
In file included from /usr/local/include/libgpsmm.h:12:
/usr/local/include/gps.h:2014:12: error: elaborated type refers to a typedef
struct policy_t policy; /* our listening policy */
   ^
/usr/include/mach/policy.h:79:17: note: declared here
typedef int policy_t;
^
2 errors generated.
make[2]: *** [libminifi/CMakeFiles/minifi_gps.dir/src/gps/GetGPS.cpp.o] 
Error 1

```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #52: Support for GPSD integration

[apiri]

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/52#discussion_r102762571
  
--- Diff: README.md ---
@@ -110,6 +118,9 @@ $ brew install cmake \
   leveldb \
   ossp-uuid \
   boost \ openssl
+
+# If building with GPS support
+$ brew install gps
--- End diff --

Believe this should be gpsd


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #52: Support for GPSD integration

[jdye64]

Github user jdye64 commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/52#discussion_r102758892
  
--- Diff: libminifi/test/unit/gps/GetGPSTests.h ---
@@ -0,0 +1,37 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+#ifndef GPS_TESTS
+#define GPS_TESTS
+#include "../../TestBase.h"
+
+#include "../ProvenanceTestHelper.h"
+#include "../Provenance.h"
+#include "../FlowFileRecord.h"
+
+
+TEST_CASE("Test GPSD Create", "[TestGPSEventRecord]"){
--- End diff --

certainly. I'm wondering the best way to do this at the moment. I want to 
use gpsfake for this but thinking through the best way to do so. Using gpsfake 
would require another system dependency since it is an extra GPSD service 
itself.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp issue #52: Support for GPSD integration

[apiri]

Github user apiri commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/52
  
Could we also get travis updated to include this in the build by default?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #52: Support for GPSD integration

[apiri]

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/52#discussion_r102753365
  
--- Diff: libminifi/test/unit/gps/GetGPSTests.h ---
@@ -0,0 +1,37 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+#ifndef GPS_TESTS
+#define GPS_TESTS
+#include "../../TestBase.h"
+
+#include "../ProvenanceTestHelper.h"
+#include "../Provenance.h"
+#include "../FlowFileRecord.h"
+
+
+TEST_CASE("Test GPSD Create", "[TestGPSEventRecord]"){
--- End diff --

Would we be able to create some processor specific tests?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3332) Bug in ListXXX causes matching timestamps to be ignored on later runs

[Joe Skora (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880743#comment-15880743
 ] 

Joe Skora commented on NIFI-3332:
-

[~ijokarumawak], I understand the concerns with the storage of the file list, 
but we can't guarantee all files will be picked up without it unless we add 
some artificial constraint like your "_SUCCESS" marker file.  The "_SUCCESS" 
marker can be created by a Hadoop batch process, but other processes don't have 
a natural way to do that, do you see that being created and maintained 
internally or externally to the flow?

For use cases that are guaranteed to have multiple files with the same 
timestamp, these processors are unusable.  My concern is that any flow can 
potentially have multiple files for a timestamp and thus trigger this problem.

> Bug in ListXXX causes matching timestamps to be ignored on later runs
> -
>
> Key: NIFI-3332
> URL: https://issues.apache.org/jira/browse/NIFI-3332
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.7.1, 1.1.1
>Reporter: Joe Skora
>Assignee: Koji Kawamura
>Priority: Critical
> Attachments: Test-showing-ListFile-timestamp-bug.log, 
> Test-showing-ListFile-timestamp-bug.patch
>
>
> The new state implementation for the ListXXX processors based on 
> AbstractListProcessor creates a race conditions when processor runs occur 
> while a batch of files is being written with the same timestamp.
> The changes to state management dropped tracking of the files processed for a 
> given timestamp.  Without the record of files processed, the remainder of the 
> batch is ignored on the next processor run since their timestamp is not 
> greater than the one timestamp stored in processor state.  With the file 
> tracking it was possible to process files that matched the timestamp exactly 
> and exclude the previously processed files.
> A basic time goes as follows.
>   T0 - system creates or receives batch of files with Tx timestamp where Tx 
> is more than the current timestamp in processor state.
>   T1 - system writes 1st half of Tx batch to the ListFile source directory.
>   T2 - ListFile runs picking up 1st half of Tx batch and stores Tx timestamp 
> in processor state.
>   T3 - system writes 2nd half of Tx batch to ListFile source directory.
>   T4 - ListFile runs ignoring any files with T <= Tx, eliminating 2nd half Tx 
> timestamp batch.
> I've attached a patch[1] for TestListFile.java that adds an instrumented unit 
> test demonstrates the problem and a log[2] of the output from one such run.  
> The test writes 3 files each in two batches with processor runs after each 
> batch.  Batch 2 writes files with timestamps older than, equal to, and newer 
> than the timestamp stored when batch 1 was processed, but only the newer file 
> is picked up.  The older file is correctly ignored but file with the matchin 
> timestamp file should have been processed.
> [1] Test-showing-ListFile-timestamp-bug.patch
> [2] Test-showing-ListFile-timestamp-bug.log



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3497) ScanAttribute should support tagging a flowfile with metadata value from the supplied dictionary

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880699#comment-15880699
 ] 

ASF GitHub Bot commented on NIFI-3497:
--

GitHub user joetrite opened a pull request:

https://github.com/apache/nifi/pull/1536

NIFI-3497 ScanAttribute Tagging

Adding tagging to ScanAttribute.  

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/joetrite/nifi master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1536.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1536


commit f52e1f2a064b31f87d4165af6075716fa7d55046
Author: Joe Trite 
Date:   2017-02-22T21:36:08Z

NIFI-3497 - Added metadata option

Added optional to post additional metadata as new attributed if a match is 
found in the dictionary.

commit 8eb54a50193897cf564eb7d222aae35481168af4
Author: Joe Trite 
Date:   2017-02-22T21:46:13Z

NIFI-3497 test cases for metadata updates

Adding test cases to support metadata option update.

commit a7a7b6ace80380416c342809ce95a4f4087bb066
Author: Joe Trite 
Date:   2017-02-22T21:48:10Z

NIFI-3497 - New dictionary files for test

Adding new dictionary files to support metadata dictionary option.

commit d71426037b142da8ca04dae38952c164d1614806
Author: Joe Trite 
Date:   2017-02-23T15:19:01Z

NIFI-3497 - excluding test files

Adding new test data files to exclude list.




> ScanAttribute should support tagging a flowfile with metadata value from the 
> supplied dictionary
> 
>
> Key: NIFI-3497
> URL: https://issues.apache.org/jira/browse/NIFI-3497
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Witt
>
> Today ScanAttribute just looks through the supplied dictionary and given 
> object for a string matching hit.  If it hits then it is a match otherwise it 
> is a 'not found'.  However, when a hit occurs it can often be quite useful to 
> gather additional metadata about that hit.  This makes cases like 
> enrichment/tagging much easier.
> So, plan is to have ScanAttribute support a dictionary value demarcator which 
> would separate the dictionary term from some string response that will be 
> added to the flowfile.  For instance a dictionary might have
> apples:These are red or green
> bananas:These are yellow unless you should toss them or make bread
> Then if a hit occurs on 'apples' the flowfile that contained such an 
> attribute would have a new attribute such as 'dictionary.hit.term' = 'apple' 
> and 'dictionary.hit.metadata' = 'These are red or green'.
> This means downstream processors could extract that metadata and do 
> interesting things with it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1536: NIFI-3497 ScanAttribute Tagging

[joetrite]

GitHub user joetrite opened a pull request:

https://github.com/apache/nifi/pull/1536

NIFI-3497 ScanAttribute Tagging

Adding tagging to ScanAttribute.  

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/joetrite/nifi master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1536.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1536


commit f52e1f2a064b31f87d4165af6075716fa7d55046
Author: Joe Trite 
Date:   2017-02-22T21:36:08Z

NIFI-3497 - Added metadata option

Added optional to post additional metadata as new attributed if a match is 
found in the dictionary.

commit 8eb54a50193897cf564eb7d222aae35481168af4
Author: Joe Trite 
Date:   2017-02-22T21:46:13Z

NIFI-3497 test cases for metadata updates

Adding test cases to support metadata option update.

commit a7a7b6ace80380416c342809ce95a4f4087bb066
Author: Joe Trite 
Date:   2017-02-22T21:48:10Z

NIFI-3497 - New dictionary files for test

Adding new dictionary files to support metadata dictionary option.

commit d71426037b142da8ca04dae38952c164d1614806
Author: Joe Trite 
Date:   2017-02-23T15:19:01Z

NIFI-3497 - excluding test files

Adding new test data files to exclude list.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1535: [NIFI-2891] persist selected tab for connection con...

[scottyaslan]

GitHub user scottyaslan opened a pull request:

https://github.com/apache/nifi/pull/1535

[NIFI-2891] persist selected tab for connection configuration dialog,…

… except for when creating a new connection the dialog will open with the 
details tab visible since users will need to select a connection relationship

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/scottyaslan/nifi NIFI-2891

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1535.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1535


commit 3f90ab8d8ee11988e764a7663a41496729b87af4
Author: Scott Aslan 
Date:   2017-02-23T15:40:58Z

[NIFI-2891] persist selected tab for connection configuration dialog, 
except for when creating a new connection the dialog will open with the details 
tab visible since users will need to select a connection relationship




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2891) The last tab selected/edited should be remembered when configuring a connection

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan updated NIFI-2891:
--
Status: Patch Available  (was: In Progress)

> The last tab selected/edited should be remembered when configuring a 
> connection
> ---
>
> Key: NIFI-2891
> URL: https://issues.apache.org/jira/browse/NIFI-2891
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Affects Versions: 0.7.0, 1.0.0
>Reporter: Andrew Lim
>Assignee: Scott Aslan
>Priority: Trivial
>
> When you configure a processor, the Configure Processor window opens with the 
> tab that was most recently selected.
> This same behavior should occur when configuring a connection.  For example, 
> if a user changes the back pressure settings on a connection, the next 
> Configuration Connection dialog should open with the Settings tab selected.   
> The current behavior is that the Details tab is always the default.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2891) The last tab selected/edited should be remembered when configuring a connection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880664#comment-15880664
 ] 

ASF GitHub Bot commented on NIFI-2891:
--

GitHub user scottyaslan opened a pull request:

https://github.com/apache/nifi/pull/1535

[NIFI-2891] persist selected tab for connection configuration dialog,…

… except for when creating a new connection the dialog will open with the 
details tab visible since users will need to select a connection relationship

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/scottyaslan/nifi NIFI-2891

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1535.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1535


commit 3f90ab8d8ee11988e764a7663a41496729b87af4
Author: Scott Aslan 
Date:   2017-02-23T15:40:58Z

[NIFI-2891] persist selected tab for connection configuration dialog, 
except for when creating a new connection the dialog will open with the details 
tab visible since users will need to select a connection relationship




> The last tab selected/edited should be remembered when configuring a 
> connection
> ---
>
> Key: NIFI-2891
> URL: https://issues.apache.org/jira/browse/NIFI-2891
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Andrew Lim
>Assignee: Scott Aslan
>Priority: Trivial
>
> When you configure a processor, the Configure Processor window opens with the 
> tab that was most recently selected.
> This same behavior should occur when configuring a connection.  For example, 
> if a user changes the back pressure settings on a connection, the next 
> Configuration Connection dialog should open with the Settings tab selected.   
> The current behavior is that the Details tab is always the default.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (NIFI-2891) The last tab selected/edited should be remembered when configuring a connection

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan reassigned NIFI-2891:
-

Assignee: Scott Aslan  (was: Matt Gilman)

> The last tab selected/edited should be remembered when configuring a 
> connection
> ---
>
> Key: NIFI-2891
> URL: https://issues.apache.org/jira/browse/NIFI-2891
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Andrew Lim
>Assignee: Scott Aslan
>Priority: Trivial
>
> When you configure a processor, the Configure Processor window opens with the 
> tab that was most recently selected.
> This same behavior should occur when configuring a connection.  For example, 
> if a user changes the back pressure settings on a connection, the next 
> Configuration Connection dialog should open with the Settings tab selected.   
> The current behavior is that the Details tab is always the default.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1534: [NIFI-3038] Update cluster summary dialog layout

[scottyaslan]

GitHub user scottyaslan opened a pull request:

https://github.com/apache/nifi/pull/1534

[NIFI-3038] Update cluster summary dialog layout

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/scottyaslan/nifi NIFI-3038

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1534.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1534


commit 49f7c5873439228016d3ffde8675e7cf86fad470
Author: Scott Aslan 
Date:   2017-02-23T14:48:38Z

[NIFI-3038] Update cluster summary dialog layout




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-3038) Update layout of Cluster Summary window

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan updated NIFI-3038:
--
Status: Patch Available  (was: In Progress)

> Update layout of Cluster  Summary window
> ---
>
> Key: NIFI-3038
> URL: https://issues.apache.org/jira/browse/NIFI-3038
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Rob Moran
>Assignee: Scott Aslan
>Priority: Minor
>
> To make the Cluster  Summary window more consistent with other 
> areas of the application, move the selected component details to align left 
> and move the refresh action and timestamp to the bottom left (below the 
> table).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3038) Update layout of Cluster Summary window

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880580#comment-15880580
 ] 

ASF GitHub Bot commented on NIFI-3038:
--

GitHub user scottyaslan opened a pull request:

https://github.com/apache/nifi/pull/1534

[NIFI-3038] Update cluster summary dialog layout

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/scottyaslan/nifi NIFI-3038

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1534.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1534


commit 49f7c5873439228016d3ffde8675e7cf86fad470
Author: Scott Aslan 
Date:   2017-02-23T14:48:38Z

[NIFI-3038] Update cluster summary dialog layout




> Update layout of Cluster  Summary window
> ---
>
> Key: NIFI-3038
> URL: https://issues.apache.org/jira/browse/NIFI-3038
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Rob Moran
>Assignee: Scott Aslan
>Priority: Minor
>
> To make the Cluster  Summary window more consistent with other 
> areas of the application, move the selected component details to align left 
> and move the refresh action and timestamp to the bottom left (below the 
> table).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1214: NIFI-2876 refactored demarcators into a common abst...

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102725986
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/TextLineDemarcator.java
 ---
@@ -95,52 +71,61 @@ public OffsetInfo nextOffsetInfo() {
  *
  * @return offset info
  */
-public OffsetInfo nextOffsetInfo(byte[] startsWith) {
+public OffsetInfo nextOffsetInfo(byte[] startsWith) throws IOException 
{
--- End diff --

I'm not sure that i understand the concept here of passing in a `byte[] 
startsWith`. The method returns an OffsetInfo regardless. Whether or not it 
starts with some arbitrary byte array seems like a very separate concern. Seems 
like that should be externalized and let the caller check if it starts with 
some pattern... or offering a `startsWith(byte[])` method on OffsetInfo?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2876) Refactor TextLineDemarcator and StreamDemarcator into a common abstract class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880554#comment-15880554
 ] 

ASF GitHub Bot commented on NIFI-2876:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102725986
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/TextLineDemarcator.java
 ---
@@ -95,52 +71,61 @@ public OffsetInfo nextOffsetInfo() {
  *
  * @return offset info
  */
-public OffsetInfo nextOffsetInfo(byte[] startsWith) {
+public OffsetInfo nextOffsetInfo(byte[] startsWith) throws IOException 
{
--- End diff --

I'm not sure that i understand the concept here of passing in a `byte[] 
startsWith`. The method returns an OffsetInfo regardless. Whether or not it 
starts with some arbitrary byte array seems like a very separate concern. Seems 
like that should be externalized and let the caller check if it starts with 
some pattern... or offering a `startsWith(byte[])` method on OffsetInfo?


> Refactor TextLineDemarcator and StreamDemarcator into a common abstract class
> -
>
> Key: NIFI-2876
> URL: https://issues.apache.org/jira/browse/NIFI-2876
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Oleg Zhurakousky
>Assignee: Oleg Zhurakousky
>Priority: Minor
> Fix For: 1.2.0
>
>
> Based on the work that has been performed as part of the NIFI-2851 we now 
> have a new class with a significantly faster logic to perform demarcation of 
> the InputStream (TextLineDemarcator). This new class's initial starting point 
> was the existing LineDemarcator. They both now share ~60-70% of common code 
> which would be important to extract into a common abstract class as well as 
> incorporate the new (faster) demarcation logic int StreamDemarcator.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2876) Refactor TextLineDemarcator and StreamDemarcator into a common abstract class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880550#comment-15880550
 ] 

ASF GitHub Bot commented on NIFI-2876:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102725352
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/TextLineDemarcator.java
 ---
@@ -270,5 +180,10 @@ public boolean isStartsWithMatch() {
 void setStartsWithMatch(int startsWithMatch) {
--- End diff --

Why does this take in an `int` and then assign a `boolean` based on it? Why 
not just take in an `boolean`?


> Refactor TextLineDemarcator and StreamDemarcator into a common abstract class
> -
>
> Key: NIFI-2876
> URL: https://issues.apache.org/jira/browse/NIFI-2876
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Oleg Zhurakousky
>Assignee: Oleg Zhurakousky
>Priority: Minor
> Fix For: 1.2.0
>
>
> Based on the work that has been performed as part of the NIFI-2851 we now 
> have a new class with a significantly faster logic to perform demarcation of 
> the InputStream (TextLineDemarcator). This new class's initial starting point 
> was the existing LineDemarcator. They both now share ~60-70% of common code 
> which would be important to extract into a common abstract class as well as 
> incorporate the new (faster) demarcation logic int StreamDemarcator.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] nifi pull request #1214: NIFI-2876 refactored demarcators into a common abst...

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102712235
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/AbstractDemarcator.java
 ---
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stream.io.util;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStream;
+
+import org.apache.nifi.stream.io.exception.TokenTooLargeException;
+
+/**
+ * Base class for implementing streaming demarcators.
+ * 
+ * NOTE: Not intended for multi-thread usage hence not Thread-safe.
+ * 
+ */
+abstract class AbstractDemarcator implements Closeable {
+
+final static int INIT_BUFFER_SIZE = 8192;
+
+private final InputStream is;
+
+private final int initialBufferSize;
+
+private final int maxDataSize;
+
+byte[] buffer;
+
+int index;
+
+int mark;
+
+long offset;
+
+int bufferLength;
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size. Each demarcated token must fit within max 
buffer
+ * size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize) {
+this(is, maxDataSize, INIT_BUFFER_SIZE);
+}
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size and initial buffer size. Each demarcated token 
must
+ * fit within max buffer size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize, int 
initialBufferSize) {
+this.validate(is, maxDataSize, initialBufferSize);
+this.is = is;
+this.initialBufferSize = initialBufferSize;
+this.buffer = new byte[initialBufferSize];
+this.maxDataSize = maxDataSize;
+}
+
+@Override
+public void close() throws IOException {
+// noop
--- End diff --

This appears to be a resource leak, as it is not closing the given 
InputStream


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2876) Refactor TextLineDemarcator and StreamDemarcator into a common abstract class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880556#comment-15880556
 ] 

ASF GitHub Bot commented on NIFI-2876:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102712235
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/AbstractDemarcator.java
 ---
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stream.io.util;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStream;
+
+import org.apache.nifi.stream.io.exception.TokenTooLargeException;
+
+/**
+ * Base class for implementing streaming demarcators.
+ * 
+ * NOTE: Not intended for multi-thread usage hence not Thread-safe.
+ * 
+ */
+abstract class AbstractDemarcator implements Closeable {
+
+final static int INIT_BUFFER_SIZE = 8192;
+
+private final InputStream is;
+
+private final int initialBufferSize;
+
+private final int maxDataSize;
+
+byte[] buffer;
+
+int index;
+
+int mark;
+
+long offset;
+
+int bufferLength;
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size. Each demarcated token must fit within max 
buffer
+ * size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize) {
+this(is, maxDataSize, INIT_BUFFER_SIZE);
+}
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size and initial buffer size. Each demarcated token 
must
+ * fit within max buffer size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize, int 
initialBufferSize) {
+this.validate(is, maxDataSize, initialBufferSize);
+this.is = is;
+this.initialBufferSize = initialBufferSize;
+this.buffer = new byte[initialBufferSize];
+this.maxDataSize = maxDataSize;
+}
+
+@Override
+public void close() throws IOException {
+// noop
--- End diff --

This appears to be a resource leak, as it is not closing the given 
InputStream


> Refactor TextLineDemarcator and StreamDemarcator into a common abstract class
> -
>
> Key: NIFI-2876
> URL: https://issues.apache.org/jira/browse/NIFI-2876
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Oleg Zhurakousky
>Assignee: Oleg Zhurakousky
>Priority: Minor
> Fix For: 1.2.0
>
>
> Based on the work that has been performed as part of the NIFI-2851 we now 
> have a new class with a significantly faster logic to perform demarcation of 
> the InputStream (TextLineDemarcator). This new class's initial starting point 
> was the existing LineDemarcator. They both now share ~60-70% of common code 
> which would be important to extract into a common abstract class as well as 
> incorporate the new (faster) demarcation logic int StreamDemarcator.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2876) Refactor TextLineDemarcator and StreamDemarcator into a common abstract class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880551#comment-15880551
 ] 

ASF GitHub Bot commented on NIFI-2876:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102713215
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/AbstractDemarcator.java
 ---
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stream.io.util;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStream;
+
+import org.apache.nifi.stream.io.exception.TokenTooLargeException;
+
+/**
+ * Base class for implementing streaming demarcators.
+ * 
+ * NOTE: Not intended for multi-thread usage hence not Thread-safe.
+ * 
+ */
+abstract class AbstractDemarcator implements Closeable {
+
+final static int INIT_BUFFER_SIZE = 8192;
+
+private final InputStream is;
+
+private final int initialBufferSize;
+
+private final int maxDataSize;
+
+byte[] buffer;
+
+int index;
+
+int mark;
+
+long offset;
+
+int bufferLength;
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size. Each demarcated token must fit within max 
buffer
+ * size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize) {
+this(is, maxDataSize, INIT_BUFFER_SIZE);
+}
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size and initial buffer size. Each demarcated token 
must
+ * fit within max buffer size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize, int 
initialBufferSize) {
+this.validate(is, maxDataSize, initialBufferSize);
+this.is = is;
+this.initialBufferSize = initialBufferSize;
+this.buffer = new byte[initialBufferSize];
+this.maxDataSize = maxDataSize;
+}
+
+@Override
+public void close() throws IOException {
+// noop
+}
+
+/**
+ * Will fill the current buffer from current 'index' position, 
expanding it
+ * and or shuffling it if necessary. If buffer exceeds max buffer size 
a
+ * {@link TokenTooLargeException} will be thrown.
+ *
+ * @throws IOException
+ * if unable to read from the stream
+ */
+void fill() throws IOException {
+if (this.index >= this.buffer.length) {
+if (this.mark == 0) { // expand
+byte[] newBuff = new byte[this.buffer.length + 
this.initialBufferSize];
+System.arraycopy(this.buffer, 0, newBuff, 0, 
this.buffer.length);
+this.buffer = newBuff;
+} else { // shuffle
+int length = this.index - this.mark;
+System.arraycopy(this.buffer, this.mark, this.buffer, 0, 
length);
+this.index = length;
+this.mark = 0;
+}
+}
+
+int bytesRead;
+do {
+bytesRead = this.is.read(this.buffer, this.index, 
this.buffer.length - this.index);
--- End diff --

I don't believe there's any need for a do/while block here. InputStream 
either returns -1 indicating that the stream is finished or blocks until data 
is available, so 0 is never a valid return value.


> Refactor TextLineDemarcator and StreamDemarcator into a common abstract class
> -
>
> Key: NIFI-2876
> URL: https://issues.apache.org/jira/browse/NIFI-2876
>

[jira] [Commented] (NIFI-2876) Refactor TextLineDemarcator and StreamDemarcator into a common abstract class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880553#comment-15880553
 ] 

ASF GitHub Bot commented on NIFI-2876:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102718539
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/StreamDemarcator.java
 ---
@@ -102,99 +83,53 @@ public StreamDemarcator(InputStream is, byte[] 
delimiterBytes, int maxDataSize,
  * @throws IOException if unable to read from the stream
  */
 public byte[] nextToken() throws IOException {
-byte[] data = null;
+byte[] token = null;
 int j = 0;
-
-while (data == null && this.buffer != null) {
-if (this.index >= this.readAheadLength) {
+nextTokenLoop:
+while (token == null && this.bufferLength != -1) {
+if (this.index >= this.bufferLength) {
 this.fill();
 }
-if (this.index >= this.readAheadLength) {
-data = this.extractDataToken(0);
-this.buffer = null;
-} else {
-byte byteVal = this.buffer[this.index++];
-if (this.delimiterBytes != null && this.delimiterBytes[j] 
== byteVal) {
-if (++j == this.delimiterBytes.length) {
-data = 
this.extractDataToken(this.delimiterBytes.length);
+if (this.bufferLength != -1) {
+byte byteVal;
+int i;
+for (i = this.index; i < this.bufferLength; i++) {
+byteVal = this.buffer[i];
+
+boolean delimiterFound = false;
+if (this.delimiterBytes != null && 
this.delimiterBytes[j] == byteVal) {
--- End diff --

This seems to be buggy. If this.delimiterBytes[j] == byteVal, we increment 
j. But the next byte does not match, we have already incremented j and it won't 
get reset. As a result, if we find all bytes in the delimiter in the proper 
order, we return that token, even if the bytes are not contiguous. Please add 
the following unit test to the test case and you will see the failure:

```
@Test
public void testOnPartialMatchThenSubsequentPartialMatch() throws 
IOException {
final byte[] inputData = "A Great Big 
Boy".getBytes(StandardCharsets.UTF_8);
final byte[] delimBytes = "AB".getBytes(StandardCharsets.UTF_8);

try (final InputStream is = new ByteArrayInputStream(inputData);
final StreamDemarcator demarcator = new StreamDemarcator(is, 
delimBytes, 4096)) {

final byte[] bytes = demarcator.nextToken();
assertArrayEquals(inputData, bytes);

assertNull(demarcator.nextToken());
}
}

```


> Refactor TextLineDemarcator and StreamDemarcator into a common abstract class
> -
>
> Key: NIFI-2876
> URL: https://issues.apache.org/jira/browse/NIFI-2876
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Oleg Zhurakousky
>Assignee: Oleg Zhurakousky
>Priority: Minor
> Fix For: 1.2.0
>
>
> Based on the work that has been performed as part of the NIFI-2851 we now 
> have a new class with a significantly faster logic to perform demarcation of 
> the InputStream (TextLineDemarcator). This new class's initial starting point 
> was the existing LineDemarcator. They both now share ~60-70% of common code 
> which would be important to extract into a common abstract class as well as 
> incorporate the new (faster) demarcation logic int StreamDemarcator.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2876) Refactor TextLineDemarcator and StreamDemarcator into a common abstract class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880552#comment-15880552
 ] 

ASF GitHub Bot commented on NIFI-2876:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102726151
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/TextLineDemarcator.java
 ---
@@ -95,52 +71,61 @@ public OffsetInfo nextOffsetInfo() {
  *
  * @return offset info
  */
-public OffsetInfo nextOffsetInfo(byte[] startsWith) {
+public OffsetInfo nextOffsetInfo(byte[] startsWith) throws IOException 
{
 OffsetInfo offsetInfo = null;
-int lineLength = 0;
-byte[] token = null;
-lineLoop:
-while (this.bufferLength != -1) {
+byte previousByteVal = 0;
+byte[] data = null;
+nextTokenLoop: 
+while (data == null && this.bufferLength != -1) {
 if (this.index >= this.bufferLength) {
 this.fill();
 }
+int delimiterSize = 0;
 if (this.bufferLength != -1) {
-int i;
 byte byteVal;
+int i;
 for (i = this.index; i < this.bufferLength; i++) {
 byteVal = this.buffer[i];
-lineLength++;
-int crlfLength = computeEol(byteVal, i + 1);
-if (crlfLength > 0) {
-i += crlfLength;
-if (crlfLength == 2) {
-lineLength++;
-}
-offsetInfo = new OffsetInfo(this.offset, 
lineLength, crlfLength);
+
+if (byteVal == 10) {
+delimiterSize = previousByteVal == 13 ? 2 : 1;
+} else if (previousByteVal == 13) {
+delimiterSize = 1;
+i--;
+}
+previousByteVal = byteVal;
+if (delimiterSize > 0) {
+this.index = i + 1;
+int size = Math.max(1, this.index - this.mark);
+offsetInfo = new OffsetInfo(this.offset, size, 
delimiterSize);
+this.offset += size;
 if (startsWith != null) {
-token = this.extractDataToken(lineLength);
+data = this.extractDataToken(size);
 }
 this.mark = this.index;
-break lineLoop;
+break nextTokenLoop;
 }
 }
 this.index = i;
+} else {
+delimiterSize = previousByteVal == 13 || previousByteVal 
== 10 ? 1 : 0;
+if (offsetInfo == null) {
+int size = this.index - this.mark;
+if (size > 0) {
+offsetInfo = new OffsetInfo(this.offset, size, 
delimiterSize);
+this.offset += size;
+}
+}
+if (startsWith != null) {
+data = this.extractDataToken(this.index - this.mark);
+}
 }
 }
-// EOF where last char(s) are not CRLF.
-if (lineLength > 0 && offsetInfo == null) {
-offsetInfo = new OffsetInfo(this.offset, lineLength, 0);
-if (startsWith != null) {
-token = this.extractDataToken(lineLength);
-}
-}
-this.offset += lineLength;
 
-// checks if the new line starts with 'startsWith' chars
-if (startsWith != null) {
+if (startsWith != null && data != null) {
 for (int i = 0; i < startsWith.length; i++) {
 byte sB = startsWith[i];
-if (token != null && sB != token[i]) {
+if (data != null && sB != data[i]) {
--- End diff --

`data` is guaranteed non-null here because it is checked above.


> Refactor TextLineDemarcator and StreamDemarcator into a common abstract class
> -
>
> Key: NIFI-2876
> URL: https://issues.apache.org/jira/browse/NIFI-2876
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Oleg Zhurakousky
>Assignee: Oleg Zhurakousky
>

[jira] [Commented] (NIFI-2876) Refactor TextLineDemarcator and StreamDemarcator into a common abstract class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880555#comment-15880555
 ] 

ASF GitHub Bot commented on NIFI-2876:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102713854
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/AbstractDemarcator.java
 ---
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stream.io.util;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStream;
+
+import org.apache.nifi.stream.io.exception.TokenTooLargeException;
+
+/**
+ * Base class for implementing streaming demarcators.
+ * 
+ * NOTE: Not intended for multi-thread usage hence not Thread-safe.
+ * 
+ */
+abstract class AbstractDemarcator implements Closeable {
+
+final static int INIT_BUFFER_SIZE = 8192;
+
+private final InputStream is;
+
+private final int initialBufferSize;
+
+private final int maxDataSize;
+
+byte[] buffer;
+
+int index;
+
+int mark;
+
+long offset;
+
+int bufferLength;
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size. Each demarcated token must fit within max 
buffer
+ * size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize) {
+this(is, maxDataSize, INIT_BUFFER_SIZE);
+}
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size and initial buffer size. Each demarcated token 
must
+ * fit within max buffer size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize, int 
initialBufferSize) {
+this.validate(is, maxDataSize, initialBufferSize);
+this.is = is;
+this.initialBufferSize = initialBufferSize;
+this.buffer = new byte[initialBufferSize];
+this.maxDataSize = maxDataSize;
+}
+
+@Override
+public void close() throws IOException {
+// noop
+}
+
+/**
+ * Will fill the current buffer from current 'index' position, 
expanding it
+ * and or shuffling it if necessary. If buffer exceeds max buffer size 
a
+ * {@link TokenTooLargeException} will be thrown.
+ *
+ * @throws IOException
+ * if unable to read from the stream
+ */
+void fill() throws IOException {
+if (this.index >= this.buffer.length) {
+if (this.mark == 0) { // expand
+byte[] newBuff = new byte[this.buffer.length + 
this.initialBufferSize];
+System.arraycopy(this.buffer, 0, newBuff, 0, 
this.buffer.length);
+this.buffer = newBuff;
+} else { // shuffle
+int length = this.index - this.mark;
+System.arraycopy(this.buffer, this.mark, this.buffer, 0, 
length);
+this.index = length;
+this.mark = 0;
+}
+}
+
+int bytesRead;
+do {
+bytesRead = this.is.read(this.buffer, this.index, 
this.buffer.length - this.index);
+} while (bytesRead == 0);
+this.bufferLength = bytesRead != -1 ? this.index + bytesRead : -1;
+if (this.bufferLength > this.maxDataSize) {
+throw new TokenTooLargeException("A message in the stream 
exceeds the maximum allowed message size of "
++ this.maxDataSize + " bytes.");
+}
+}
+
+/**
+ * Will extract data token of the provided length from the current 
buffer.
+ * The 

[GitHub] nifi pull request #1214: NIFI-2876 refactored demarcators into a common abst...

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102718767
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/TextLineDemarcator.java
 ---
@@ -95,52 +71,61 @@ public OffsetInfo nextOffsetInfo() {
  *
  * @return offset info
  */
-public OffsetInfo nextOffsetInfo(byte[] startsWith) {
+public OffsetInfo nextOffsetInfo(byte[] startsWith) throws IOException 
{
 OffsetInfo offsetInfo = null;
-int lineLength = 0;
-byte[] token = null;
-lineLoop:
-while (this.bufferLength != -1) {
+byte previousByteVal = 0;
+byte[] data = null;
+nextTokenLoop: 
+while (data == null && this.bufferLength != -1) {
 if (this.index >= this.bufferLength) {
 this.fill();
 }
+int delimiterSize = 0;
 if (this.bufferLength != -1) {
-int i;
 byte byteVal;
+int i;
 for (i = this.index; i < this.bufferLength; i++) {
 byteVal = this.buffer[i];
-lineLength++;
-int crlfLength = computeEol(byteVal, i + 1);
-if (crlfLength > 0) {
-i += crlfLength;
-if (crlfLength == 2) {
-lineLength++;
-}
-offsetInfo = new OffsetInfo(this.offset, 
lineLength, crlfLength);
+
+if (byteVal == 10) {
--- End diff --

Can we get rid of inline constants 10 & 13 here and instead make them 
static final member variables to make the code a little more readable?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1214: NIFI-2876 refactored demarcators into a common abst...

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1214#discussion_r102713215
  
--- Diff: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/AbstractDemarcator.java
 ---
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stream.io.util;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStream;
+
+import org.apache.nifi.stream.io.exception.TokenTooLargeException;
+
+/**
+ * Base class for implementing streaming demarcators.
+ * 
+ * NOTE: Not intended for multi-thread usage hence not Thread-safe.
+ * 
+ */
+abstract class AbstractDemarcator implements Closeable {
+
+final static int INIT_BUFFER_SIZE = 8192;
+
+private final InputStream is;
+
+private final int initialBufferSize;
+
+private final int maxDataSize;
+
+byte[] buffer;
+
+int index;
+
+int mark;
+
+long offset;
+
+int bufferLength;
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size. Each demarcated token must fit within max 
buffer
+ * size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize) {
+this(is, maxDataSize, INIT_BUFFER_SIZE);
+}
+
+/**
+ * Constructs an instance of demarcator with provided {@link 
InputStream}
+ * and max buffer size and initial buffer size. Each demarcated token 
must
+ * fit within max buffer size, otherwise the exception will be raised.
+ */
+AbstractDemarcator(InputStream is, int maxDataSize, int 
initialBufferSize) {
+this.validate(is, maxDataSize, initialBufferSize);
+this.is = is;
+this.initialBufferSize = initialBufferSize;
+this.buffer = new byte[initialBufferSize];
+this.maxDataSize = maxDataSize;
+}
+
+@Override
+public void close() throws IOException {
+// noop
+}
+
+/**
+ * Will fill the current buffer from current 'index' position, 
expanding it
+ * and or shuffling it if necessary. If buffer exceeds max buffer size 
a
+ * {@link TokenTooLargeException} will be thrown.
+ *
+ * @throws IOException
+ * if unable to read from the stream
+ */
+void fill() throws IOException {
+if (this.index >= this.buffer.length) {
+if (this.mark == 0) { // expand
+byte[] newBuff = new byte[this.buffer.length + 
this.initialBufferSize];
+System.arraycopy(this.buffer, 0, newBuff, 0, 
this.buffer.length);
+this.buffer = newBuff;
+} else { // shuffle
+int length = this.index - this.mark;
+System.arraycopy(this.buffer, this.mark, this.buffer, 0, 
length);
+this.index = length;
+this.mark = 0;
+}
+}
+
+int bytesRead;
+do {
+bytesRead = this.is.read(this.buffer, this.index, 
this.buffer.length - this.index);
--- End diff --

I don't believe there's any need for a do/while block here. InputStream 
either returns -1 indicating that the stream is finished or blocks until data 
is available, so 0 is never a valid return value.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---