date:20160616

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/239
  
Thanks for noticing that, I didn't realize it would cross-link.  That's not 
what I was hoping for.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #502: Nifi-1972 Apache Ignite Put Cache Processor

2016-06-16 Thread mans2singh

Github user mans2singh commented on the issue:

https://github.com/apache/nifi/pull/502
  
Hey @pvillard31 - Thanks for your comments and I will try to address to 
your questions below:

1.  In order to run integration tests, we need to run apache ignite 1.6 
locally (or create the proper xml configuration file - just like under the 
src/test/resources directory).  
2. There are two integration tests (and a suite of unit tests) with the 
module.  The integration tests validate the data the contents of the flow with 
the data in the ignite cache.
3. If the config file is not provided the processor creates a ignite client 
which tries to discover the ignite cluster on the localhost.  This could be 
used in case a single Nifi node is collocated with Ignite Cache node. When I 
run the processor using Nifi application, I can see the entry count 
corresponding the flow files in ignite cache by using the ignitevisorcmd.sh 
application.  If you see any problems with it, please let me know and I will to 
reproduce the scenario locally.
4. Regarding provenance, Ignite tries to shield the clients from the 
servers by using a discovery spi.  While we can find out all the remote or 
local nodes in the cluster using this spi, we do not know which nodes will host 
the entry.  Getting all the nodes in the cluster will also impact the 
performance if we do it for the provenance of each flow file.  So, I've changed 
provenance to provide cache name which is how the client should access the 
entries.  Let me know your thoughts on it.
5. The logs for Ignite can be handled by changing the log levels and I will 
look into it.
6. I've been able to start/stop the cache and don't see the exceptions 
you've mentioned. If you have specific steps to reproduce them, please let me 
know.

@apiri - My thought was that this processor is a egress component for 
streaming large volume/velocity data to the ignite cache.  If you have any 
thought/recommendation for me, please let me know.

Thanks for all your advice.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #532: NIFI-1941: Child group contents in exported template...

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/532


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #532: NIFI-1941: Child group contents in exported templates

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/532
  
Thanks for tracking that down, it makes sense now that I see your changes.  
I tested the new changes, did a full build with contrib check (thanks for the 
earlier fix), and will merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #539: NIFI-1549: Fix JMSConsumer exception

2016-06-16 Thread rekhajoshm

GitHub user rekhajoshm opened a pull request:

https://github.com/apache/nifi/pull/539

NIFI-1549: Fix JMSConsumer exception

NIFI-1549: Fix JMSConsumer exception

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rekhajoshm/nifi NIFI-1549

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/539.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #539


commit 4a418127c8c1154255c737a7991ca11734a1eafd
Author: Joshi 
Date:   2016-06-17T03:51:49Z

NIFI-1549: Fix JMSConsumer exception




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #511: NIFI-1850 - JSON-to-JSON Schema Converter Editor

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/511
  
@YolandaMDavis While reading over your PR I noticed that the 
ControllerService and ReportingTask URI's in the 
StandandNiFiWebConfigurationContext are incorrect. They shouldn't contain the 
path segment `/node` anymore. Could you fix that in your PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #511: NIFI-1850 - JSON-to-JSON Schema Converter Editor

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/511#discussion_r67456561
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java
 ---
@@ -1574,7 +1574,7 @@ public ProcessGroupEntity createArchive() {
 }
 
 @Override
-public ProcessorEntity setProcessorAnnotationData(final Revision 
revision, final String processorId, final String annotationData) {
--- End diff --

I don't think we need this method anymore. We should be able to user 
updateProcessor instead. This is what I was referring to in the PR for 0.x.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #511: NIFI-1850 - JSON-to-JSON Schema Converter Editor

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/511#discussion_r67456463
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/pom.xml
 ---
@@ -355,10 +355,5 @@
 spock-core
 test
 
-
--- End diff --

Was this dependency no longer used or does this PR make it unnecessary now?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #538: NIFI-1774: AbstractControllerService deprecated anno...

2016-06-16 Thread rekhajoshm

GitHub user rekhajoshm opened a pull request:

https://github.com/apache/nifi/pull/538

NIFI-1774: AbstractControllerService deprecated annotation

NIFI-1774: AbstractControllerService deprecated annotation

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rekhajoshm/nifi NIFI-1774

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/538.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #538


commit 7ecee930cdc8211eadd5e387c574c9fdbde628e1
Author: Joshi 
Date:   2016-06-17T03:21:08Z

NIFI-1774: AbstractControllerService deprecated annotation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #537: NIFI-1955: Deprecate IntegerHolder, LongHolder, Bool...

2016-06-16 Thread rekhajoshm

GitHub user rekhajoshm opened a pull request:

https://github.com/apache/nifi/pull/537

NIFI-1955: Deprecate IntegerHolder, LongHolder, BooleanHolder

NIFI-1955: Deprecate IntegerHolder, LongHolder, BooleanHolder

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rekhajoshm/nifi NIFI-1955

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/537.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #537


commit ed4a3a6e6f6b65539b19bb01820f03be21c5728f
Author: Joshi 
Date:   2016-06-17T02:49:00Z

NIFI-1955: Deprecate IntegerHolder, LongHolder, BooleanHolder




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #533: Nifi 1690 0.x

Github user apiri commented on the issue:

https://github.com/apache/nifi/pull/533
  
Hey @olegz,

Looks like the magic words to close this PR were missed in the commit.  
Could you please close this when you have a chance?

Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #502: Nifi-1972 Apache Ignite Put Cache Processor

Github user apiri commented on the issue:

https://github.com/apache/nifi/pull/502
  
Hey @mans2singh,

A bit late to the game but was doing some scanning over issues and 
performing cleanup.

Just curious if any thought was given to treating this as an alternate 
implementation for a DistributedCache controller service.  Such an 
implementation would allow existing processors to make use of this backend as 
well.  Also not overly familiar with Ignite, so if there are any 
gotchas/caveats that would prevent this apologies for the lack of knowledge on 
my behalf.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #506: NIFI-1660 - Enhance the expression language with jso...

2016-06-16 Thread ckmcd

Github user ckmcd closed the pull request at:

https://github.com/apache/nifi/pull/506


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #303: NIFI-1660 - Enhance the expression language with jso...

2016-06-16 Thread ckmcd

Github user ckmcd closed the pull request at:

https://github.com/apache/nifi/pull/303


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #397: NIFI-1815

2016-06-16 Thread mattyb149

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/397#discussion_r67451461
  
--- Diff: 
nifi-nar-bundles/nifi-ocr-bundle/nifi-ocr-processors/src/test/resources/tessdata/tessconfigs/nobatch
 ---
@@ -0,0 +1 @@
+
--- End diff --

Is this supposed to be an empty file or one that was created (which is 
probably better located in target/)?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #303: NIFI-1660 - Enhance the expression language with jsonPath f...

Github user apiri commented on the issue:

https://github.com/apache/nifi/pull/303
  
Hey @ckmcd!  Looks like we missed the magical incantation to have this 
closed on the commit.  Would you mind closing this when you have the 
opportunity?

Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #285: NIFI-1636: Print Stacktrace When Unexpected OnTrigger Excep...

Github user apiri commented on the issue:

https://github.com/apache/nifi/pull/285
  
@rickysaltzer Was combing over PRs in rounding up items for 0.7.0.  Curious 
if the effort highlighted by Oleg makes this particular PR OBE.  Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #534: Fix for NIFI-1838 & NIFI-1152

Github user PuspenduBanerjee commented on the issue:

https://github.com/apache/nifi/pull/534
  
@pvillard31 Thanks for your effort to review this. 
I have incorporated your comments as much as I could.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #534: Fix for NIFI-1838 & NIFI-1152

Github user PuspenduBanerjee commented on a diff in the pull request:

https://github.com/apache/nifi/pull/534#discussion_r67446868
  
--- Diff: 
nifi-mock/src/test/java/org/apache/nifi/util/TestMockProcessSession.java ---
@@ -36,7 +36,7 @@ public void testPenalizeFlowFileFromProcessor() {
 
 protected static class PoorlyBehavedProcessor extends 
AbstractProcessor {
 
-private static final Relationship REL_FAILURE = new 
Relationship.Builder()
+protected static final Relationship REL_FAILURE = new 
Relationship.Builder()
--- End diff --

No not needed anymore. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #536: NIFI-1037 Ported processor for HDFS' inotify events to 0.x.

2016-06-16 Thread jjmeyer0

Github user jjmeyer0 commented on the issue:

https://github.com/apache/nifi/pull/536
  
@pvillard31 fixed java 7 issues of this processor as requested.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #534: Fix for NIFI-1838 & NIFI-1152

Github user PuspenduBanerjee commented on a diff in the pull request:

https://github.com/apache/nifi/pull/534#discussion_r67446830
  
--- Diff: 
nifi-mock/src/main/java/org/apache/nifi/util/MockProcessSession.java ---
@@ -793,13 +798,21 @@ private void validateState(final FlowFile flowFile) {
 throw new FlowFileHandlingException(flowFile + " is not the 
most recent version of this flow file within this session");
 }
 
+
 for (final List flowFiles : transferMap.values()) {
 if (flowFiles.contains(flowFile)) {
 throw new IllegalStateException(flowFile + " has already 
been transferred");
 }
 }
 }
 
+private void validateDestinationRelation(final Relationship 
relationship) {
+if (!relationships.contains(relationship)) {
+rollback();
--- End diff --

I think, though it's for testing purpose , we should emulate as real as we 
can and rollback is the best possible way to revert back to a stable state. 
And, in a real context we may face such issue in case of orchestration, where 
an attribute in a flowfile may finally dictate where it should be transferred.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #532: NIFI-1941: Child group contents in exported templates

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/532
  
@jvwing Just addressed the issue and rebased to current master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #532: NIFI-1941: Child group contents in exported templates

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/532
  
@jvwing Actually, I think I see what the issue is. Going to verify my 
change and if all looks good, I'll update the PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #239: Nifi 1540 - AWS Kinesis Get and Put Processors

Github user apiri commented on the issue:

https://github.com/apache/nifi/pull/239
  
Hey @jvwing & @joewitt 

James, your comment listing the associated JIRA legal issue actually had 
the nice side effect of getting included in the associated issue.  A response 
is here: 
https://issues.apache.org/jira/browse/LEGAL-198?focusedCommentId=15321636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15321636

Unfortunately, it seems like this falls under Category X and cannot be 
required for the build (meaning we would not be able to bundle it) from my 
interpretation of Justin's comment.  Seems we are a bit stuck with regards to 
NiFi as a default inclusion with our assembly and would likely need help as to 
how build separately until we are able to get an extension registry that would 
perhaps provide some additional flexibility.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #532: NIFI-1941: Child group contents in exported templates

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/532
  
@jvwing Thanks for jumping in to review! I am not aware of this specific 
issue though I know we have a number of outside JIRAs surrounding templates and 
the 1.0.0 release. It doesn't surprise me that the UUID is not in the 
serialized template as the component's generate new UUIDs upon instantiation. 
This allows us to drop the same template on the canvas as many times as we want 
to.

The issue you mentioned may even lay in some REST API changes for starting 
components. Want me to open a new JIRA to address this issue as it's outside 
the scope of the original? 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r67443261
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/test/java/org/apache/nifi/processors/windows/event/log/EventSubscribeCantLoadTest.java
 ---
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+
+/**
+ * Test scenario where we can't load the native libraries
+ */
+@RunWith(JNAFailJUnitRunner.class)
+public class EventSubscribeCantLoadTest {
+
+@Test
+public void testValidate() {
+TestRunner testRunner = 
TestRunners.newTestRunner(EvtSubscribe.class);
+testRunner.assertNotValid();
--- End diff --

Was a dumb question, lol, it's due to "@RunWith(JNAFailJUnitRunner.class)"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r67443110
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/test/java/org/apache/nifi/processors/windows/event/log/EvtSubscribeTest.java
 ---
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.google.common.net.MediaType;
+import com.sun.jna.Pointer;
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.W32Errors;
+import com.sun.jna.platform.win32.WinDef;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.MockProcessContext;
+import org.apache.nifi.util.ReflectionUtils;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mock;
+
+import java.io.ByteArrayOutputStream;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Consumer;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyInt;
+import static org.mockito.Matchers.anyString;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Matchers.isA;
+import static org.mockito.Matchers.isNull;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@RunWith(JNAJUnitRunner.class)
+public class EvtSubscribeTest {
+@Mock
+Kernel32 kernel32;
+
+@Mock
+WEvtApi wEvtApi;
+
+@Mock
+ProcessContext processContext;
+
+@Mock
+ProcessSessionFactory processSessionFactory;
+
+@Mock
+ProcessSession processSession;
+
+private EvtSubscribe evtSubscribe;
+
+@Before
+public void setup() {
+
when(processSessionFactory.createSession()).thenReturn(processSession);
+evtSubscribe = new EvtSubscribe(wEvtApi, kernel32);
+}
+
+@Test
+public void testFlow() throws Exception {
+int maxEventQueue = 1;
+int maxBuffer = 1024;
+String testChannel = "testChannel";
+String testQuery = "testQuery";
+String testXml3 = "TestXml3";
+String testXml4 = "TestXml4";
+
+FlowFile flowFile1 = mock(FlowFile.class);
+FlowFile flowFile2 = mock(FlowFile.class);
+FlowFile flowFile3 = mock(FlowFile.class);
+
+FlowFile flowFile4 = mock(FlowFile.class);
+FlowFile flowFile5 = mock(FlowFile.class);
+FlowFile flowFile6 = mock(FlowFile.class);
+
+ByteArrayOutputStream byteArrayOutputStream1 = new 
ByteArrayOutputStream();
+ByteArrayOutputStream byteArrayOutputStream2 = new

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r67443126
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/test/java/org/apache/nifi/processors/windows/event/log/EvtSubscribeTest.java
 ---
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.google.common.net.MediaType;
+import com.sun.jna.Pointer;
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.W32Errors;
+import com.sun.jna.platform.win32.WinDef;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.MockProcessContext;
+import org.apache.nifi.util.ReflectionUtils;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mock;
+
+import java.io.ByteArrayOutputStream;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Consumer;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyInt;
+import static org.mockito.Matchers.anyString;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Matchers.isA;
+import static org.mockito.Matchers.isNull;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@RunWith(JNAJUnitRunner.class)
+public class EvtSubscribeTest {
+@Mock
+Kernel32 kernel32;
+
+@Mock
+WEvtApi wEvtApi;
+
+@Mock
+ProcessContext processContext;
+
+@Mock
+ProcessSessionFactory processSessionFactory;
+
+@Mock
+ProcessSession processSession;
+
+private EvtSubscribe evtSubscribe;
+
+@Before
+public void setup() {
+
when(processSessionFactory.createSession()).thenReturn(processSession);
+evtSubscribe = new EvtSubscribe(wEvtApi, kernel32);
+}
+
+@Test
+public void testFlow() throws Exception {
+int maxEventQueue = 1;
+int maxBuffer = 1024;
+String testChannel = "testChannel";
+String testQuery = "testQuery";
+String testXml3 = "TestXml3";
+String testXml4 = "TestXml4";
+
+FlowFile flowFile1 = mock(FlowFile.class);
+FlowFile flowFile2 = mock(FlowFile.class);
+FlowFile flowFile3 = mock(FlowFile.class);
+
+FlowFile flowFile4 = mock(FlowFile.class);
+FlowFile flowFile5 = mock(FlowFile.class);
+FlowFile flowFile6 = mock(FlowFile.class);
+
+ByteArrayOutputStream byteArrayOutputStream1 = new 
ByteArrayOutputStream();
+ByteArrayOutputStream byteArrayOutputStream2 = new

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r67442740
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/test/java/org/apache/nifi/processors/windows/event/log/EvtSubscribeTest.java
 ---
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.google.common.net.MediaType;
+import com.sun.jna.Pointer;
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.W32Errors;
+import com.sun.jna.platform.win32.WinDef;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.MockProcessContext;
+import org.apache.nifi.util.ReflectionUtils;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mock;
+
+import java.io.ByteArrayOutputStream;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Consumer;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyInt;
+import static org.mockito.Matchers.anyString;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Matchers.isA;
+import static org.mockito.Matchers.isNull;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@RunWith(JNAJUnitRunner.class)
+public class EvtSubscribeTest {
+@Mock
+Kernel32 kernel32;
+
+@Mock
+WEvtApi wEvtApi;
+
+@Mock
+ProcessContext processContext;
+
+@Mock
+ProcessSessionFactory processSessionFactory;
+
+@Mock
+ProcessSession processSession;
+
+private EvtSubscribe evtSubscribe;
+
+@Before
+public void setup() {
+
when(processSessionFactory.createSession()).thenReturn(processSession);
+evtSubscribe = new EvtSubscribe(wEvtApi, kernel32);
+}
+
+@Test
+public void testFlow() throws Exception {
--- End diff --

What is the purpose of this test? It seems like a lot of things are mocked 
out and passed around which could just be done with the TestRunner but maybe 
I'm just missing something.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r67442458
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/test/java/org/apache/nifi/processors/windows/event/log/EvtSubscribeTest.java
 ---
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.google.common.net.MediaType;
+import com.sun.jna.Pointer;
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.W32Errors;
+import com.sun.jna.platform.win32.WinDef;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.MockProcessContext;
+import org.apache.nifi.util.ReflectionUtils;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mock;
+
+import java.io.ByteArrayOutputStream;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Consumer;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyInt;
+import static org.mockito.Matchers.anyString;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Matchers.isA;
+import static org.mockito.Matchers.isNull;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+@RunWith(JNAJUnitRunner.class)
+public class EvtSubscribeTest {
+@Mock
+Kernel32 kernel32;
+
+@Mock
+WEvtApi wEvtApi;
+
+@Mock
+ProcessContext processContext;
+
+@Mock
+ProcessSessionFactory processSessionFactory;
+
+@Mock
+ProcessSession processSession;
+
+private EvtSubscribe evtSubscribe;
+
+@Before
+public void setup() {
+
when(processSessionFactory.createSession()).thenReturn(processSession);
+evtSubscribe = new EvtSubscribe(wEvtApi, kernel32);
+}
+
+@Test
+public void testFlow() throws Exception {
+int maxEventQueue = 1;
+int maxBuffer = 1024;
+String testChannel = "testChannel";
+String testQuery = "testQuery";
+String testXml3 = "TestXml3";
+String testXml4 = "TestXml4";
+
+FlowFile flowFile1 = mock(FlowFile.class);
--- End diff --

Why mock the FlowFiles manually instead of using the MockFlowFile class?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r67442237
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/test/java/org/apache/nifi/processors/windows/event/log/EventSubscribeCantLoadTest.java
 ---
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+
+/**
+ * Test scenario where we can't load the native libraries
+ */
+@RunWith(JNAFailJUnitRunner.class)
+public class EventSubscribeCantLoadTest {
+
+@Test
+public void testValidate() {
+TestRunner testRunner = 
TestRunners.newTestRunner(EvtSubscribe.class);
+testRunner.assertNotValid();
--- End diff --

May be a dumb question but why won't this be valid on windows systems?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #532: NIFI-1941: Child group contents in exported templates

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/532

I ran through the following steps to test this fix:

1. Confirm the bug
a. Built a template on master/1.0 branch with a process group
b. Exported the template
c. Confirmed that process group content was not in the exported XML file
d. Confirmed that template could not be imported into a different NiFi (it
imports into the same NiFi OK, because the IDs match?)

2. Verify the fix
a. Exported the same template using the fix code
b. Verified that the exported XML contained the process groups
c. Verified that the template can be imported into a second NiFi
d. Verified that the template can be run in the second NiFi

Everything worked great up to step 2d, running the imported template in the
second NiFi. After CTRL-A selecting everything and clicking Run, I got this
nastygram: `LocalPort[name=stuff to
log,id=9c17cb62-4df3-4122-ab36-44f290dc2bce] is not a member of this Process
Group`

Which is a reference to an input port in a Process Group nested in a
Process Group. Manually navigating around the flow and starting all of the
components worked fine without errors. I get a similar error stopping the
flow.

The id "9c17..." is not in the template XML file. From poking around at
the API it does exist, it is the "stuff to log" input port as expected in my
template. The parent-child relationships between Process Groups and Input
Ports looked OK to the untrained eye. What make this weirder is that I get a
comparable error on the original NiFi I exported the template from. So I'm not
sure this is a problem specific to this fix.

[TestNiFi1941FixTemplate.xml.txt](https://github.com/apache/nifi/files/319518/TestNiFi1941FixTemplate.xml.txt)
Matt, are you aware of this issue, and do you experience this importing and
running the attached template?

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r67440428
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/jna/WEvtApi.java
 ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log.jna;
+
+import com.sun.jna.Native;
+import com.sun.jna.Pointer;
+import com.sun.jna.platform.win32.WinDef;
+import com.sun.jna.platform.win32.WinNT;
+import com.sun.jna.win32.StdCallLibrary;
+import com.sun.jna.win32.W32APIOptions;
+
+/**
+ * JNA will create the instance of this interface with 
Native.loadLibrary().
+ * Please see 
https://msdn.microsoft.com/en-us/library/windows/desktop/aa385772(v=vs.85).aspx 
for documentation on the methods and data structures.
+ */
+public interface WEvtApi extends StdCallLibrary {
+WEvtApi INSTANCE = (WEvtApi) Native.loadLibrary("wevtapi", 
WEvtApi.class, W32APIOptions.DEFAULT_OPTIONS);
+
+WinNT.HANDLE EvtSubscribe(WinNT.HANDLE session, WinNT.HANDLE 
signalEvent, String channelName, String xpathQuery,
+  WinNT.HANDLE bookmark, WinDef.PVOID context, 
EVT_SUBSCRIBE_CALLBACK evtSubscribeCallback, int flags);
+
+boolean EvtRender(WinNT.HANDLE context, WinNT.HANDLE fragment, int 
flags, int bufferSize, Pointer buffer, Pointer bufferUsed, Pointer 
propertyCount);
+
+interface EvtSubscribeNotifyAction {
+int ERROR = 0;
+int DELIVER = 1;
+}
+
+interface EvtSubscribeFlags {
--- End diff --

What is the purpose of this and EvtRenderFlags? Only the 
SUBSCRIBE_TO_FUTURE and EVENT_XML values are used.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #328: NIFI-1690 Changed MonitorMemory to use allowable values for...

2016-06-16 Thread alopresto

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/328
  
Merged and closed. Will do the same for [PR 
533](https://github.com/apache/nifi/pull/533). 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r67440149
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/EvtSubscribe.java
 ---
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.net.MediaType;
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"ingest", "event", "windows"})
+@CapabilityDescription("Registers a Windows Event Log Subscribe Callback 
to receive FlowFiles from Events on Windows.  These can be filtered via channel 
and XPath.")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Will set 
a MIME type value of application/xml.")
+})
+public class EvtSubscribe extends AbstractSessionFactoryProcessor {
--- End diff --

This should be renamed to ConsumeWindowsEventLog. There should be little 
doubt the scope and purpose of the processor from the name and, in my opinion 
(which could be totally wrong), Evt doesn't uniquely identify the processor 
enough. 

As for the "Consume" part, there was a long discussion @joewitt had with 
others but the gist is that we aren't just subscribing, we are consuming the 
events.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #328: NIFI-1690 Changed MonitorMemory to use allowable val...

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/328


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r67439511
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/EvtSubscribe.java
 ---
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.net.MediaType;
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"ingest", "event", "windows"})
+@CapabilityDescription("Registers a Windows Event Log Subscribe Callback 
to receive FlowFiles from Events on Windows.  These can be filtered via channel 
and XPath.")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Will set 
a MIME type value of application/xml.")
+})
+public class EvtSubscribe extends AbstractSessionFactoryProcessor {
+public static final String DEFAULT_CHANNEL = "System";
+public static final String DEFAULT_XPATH = "*";
+public static final int DEFAULT_MAX_BUFFER = 1024 * 1024;
+public static final int DEFAULT_MAX_QUEUE_SIZE = 1024;
+
+public static final PropertyDescriptor CHANNEL = new 
PropertyDescriptor.Builder()
+.name("channel")
+.displayName("Channel")
+.required(true)
+.defaultValue(DEFAULT_CHANNEL)
+.description("The Windows Event Log Channel to listen to.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor QUERY = new 
PropertyDescriptor.Builder()
+.name("query")
+.displayName("XPath Query")
+.required(true)
+.defaultValue(DEFAULT_XPATH)
+.description("XPath Query to filter events. (See 
https://msdn.microsoft.com/en-us/library/windows/desktop/dd996910(v=vs.85).aspx 
for examples.)")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor MAX_BUFFER_SIZE = new

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r67439340
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/jna/EventSubscribeXmlRenderingCallback.java
 ---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log.jna;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.sun.jna.Memory;
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.W32Errors;
+import com.sun.jna.platform.win32.WinDef;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.logging.ComponentLog;
+
+import java.util.function.Consumer;
+
+/**
+ * Callback that will render the XML representation of the event using 
native Windows API
+ */
+public class EventSubscribeXmlRenderingCallback implements 
WEvtApi.EVT_SUBSCRIBE_CALLBACK {
+public static final String RECEIVED_THE_FOLLOWING_WIN32_ERROR = 
"Received the following Win32 error: ";
+public static final int INITIAL_BUFFER_SIZE = 1024;
+
+private final ComponentLog logger;
+private final Consumer consumer;
+private final int maxBufferSize;
+private final WEvtApi wEvtApi;
+private final Kernel32 kernel32;
+
+private int size;
+private Memory buffer;
+private Memory used;
+private Memory propertyCount;
+
+public EventSubscribeXmlRenderingCallback(ComponentLog logger, 
Consumer consumer, int maxBufferSize, WEvtApi wEvtApi, Kernel32 
kernel32) {
+this.logger = logger;
+this.consumer = consumer;
+this.maxBufferSize = maxBufferSize;
+this.wEvtApi = wEvtApi;
+this.kernel32 = kernel32;
+this.size = Math.min(maxBufferSize, INITIAL_BUFFER_SIZE);
+this.buffer = new Memory(size);
+this.used = new Memory(4);
+this.propertyCount = new Memory(4);
+}
+
+@Override
+public synchronized int onEvent(int evtSubscribeNotifyAction, 
WinDef.PVOID userContext, WinNT.HANDLE eventHandle) {
--- End diff --

This method has a return type of int but only ever returns 0, what is the 
purpose of that?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #534: Fix for NIFI-1838 & NIFI-1152

Github user PuspenduBanerjee commented on a diff in the pull request:

https://github.com/apache/nifi/pull/534#discussion_r67439195
  
--- Diff: 
nifi-mock/src/test/java/org/apache/nifi/util/TestMockProcessSession.java ---
@@ -55,4 +55,35 @@ public void onTrigger(final ProcessContext ctx, final 
ProcessSession session) th
 }
 
 }
+
+
+@Test(expected = AssertionError.class)
+public void testNonExistentRelationFromProcessor() {
+
TestRunners.newTestRunner(NonExistentRelationTestProcessor.class).run();
+}
+
+protected static class NonExistentRelationTestProcessor extends 
PoorlyBehavedProcessor {
+
+static final Relationship REL_NON_EXISTENT = new 
Relationship.Builder()
+.name("NonExistent Relation")
+.build();
+
+public Set relationships = 
Collections.singleton(REL_FAILURE);
+
+@Override
+public Set getRelationships() {
+return relationships;
--- End diff --

Good finding. Can't really remember why did I add that. removed now and 
checking in.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r67438939
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/EvtSubscribe.java
 ---
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.net.MediaType;
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"ingest", "event", "windows"})
+@CapabilityDescription("Registers a Windows Event Log Subscribe Callback 
to receive FlowFiles from Events on Windows.  These can be filtered via channel 
and XPath.")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Will set 
a MIME type value of application/xml.")
+})
+public class EvtSubscribe extends AbstractSessionFactoryProcessor {
+public static final String DEFAULT_CHANNEL = "System";
+public static final String DEFAULT_XPATH = "*";
+public static final int DEFAULT_MAX_BUFFER = 1024 * 1024;
+public static final int DEFAULT_MAX_QUEUE_SIZE = 1024;
+
+public static final PropertyDescriptor CHANNEL = new 
PropertyDescriptor.Builder()
+.name("channel")
+.displayName("Channel")
+.required(true)
+.defaultValue(DEFAULT_CHANNEL)
+.description("The Windows Event Log Channel to listen to.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor QUERY = new 
PropertyDescriptor.Builder()
+.name("query")
+.displayName("XPath Query")
+.required(true)
+.defaultValue(DEFAULT_XPATH)
+.description("XPath Query to filter events. (See 
https://msdn.microsoft.com/en-us/library/windows/desktop/dd996910(v=vs.85).aspx 
for examples.)")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor MAX_BUFFER_SIZE = new

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r67438756
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/EvtSubscribe.java
 ---
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.net.MediaType;
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"ingest", "event", "windows"})
+@CapabilityDescription("Registers a Windows Event Log Subscribe Callback 
to receive FlowFiles from Events on Windows.  These can be filtered via channel 
and XPath.")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Will set 
a MIME type value of application/xml.")
+})
+public class EvtSubscribe extends AbstractSessionFactoryProcessor {
+public static final String DEFAULT_CHANNEL = "System";
+public static final String DEFAULT_XPATH = "*";
+public static final int DEFAULT_MAX_BUFFER = 1024 * 1024;
+public static final int DEFAULT_MAX_QUEUE_SIZE = 1024;
+
+public static final PropertyDescriptor CHANNEL = new 
PropertyDescriptor.Builder()
+.name("channel")
+.displayName("Channel")
+.required(true)
+.defaultValue(DEFAULT_CHANNEL)
+.description("The Windows Event Log Channel to listen to.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor QUERY = new 
PropertyDescriptor.Builder()
+.name("query")
+.displayName("XPath Query")
+.required(true)
+.defaultValue(DEFAULT_XPATH)
+.description("XPath Query to filter events. (See 
https://msdn.microsoft.com/en-us/library/windows/desktop/dd996910(v=vs.85).aspx 
for examples.)")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor MAX_BUFFER_SIZE = new

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r67438723
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/EvtSubscribe.java
 ---
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.net.MediaType;
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"ingest", "event", "windows"})
+@CapabilityDescription("Registers a Windows Event Log Subscribe Callback 
to receive FlowFiles from Events on Windows.  These can be filtered via channel 
and XPath.")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Will set 
a MIME type value of application/xml.")
+})
+public class EvtSubscribe extends AbstractSessionFactoryProcessor {
+public static final String DEFAULT_CHANNEL = "System";
+public static final String DEFAULT_XPATH = "*";
+public static final int DEFAULT_MAX_BUFFER = 1024 * 1024;
+public static final int DEFAULT_MAX_QUEUE_SIZE = 1024;
+
+public static final PropertyDescriptor CHANNEL = new 
PropertyDescriptor.Builder()
+.name("channel")
+.displayName("Channel")
+.required(true)
+.defaultValue(DEFAULT_CHANNEL)
+.description("The Windows Event Log Channel to listen to.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor QUERY = new 
PropertyDescriptor.Builder()
+.name("query")
+.displayName("XPath Query")
+.required(true)
+.defaultValue(DEFAULT_XPATH)
+.description("XPath Query to filter events. (See 
https://msdn.microsoft.com/en-us/library/windows/desktop/dd996910(v=vs.85).aspx 
for examples.)")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor MAX_BUFFER_SIZE = new

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r67437336
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/EvtSubscribe.java
 ---
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.net.MediaType;
+import com.sun.jna.platform.win32.Kernel32;
+import com.sun.jna.platform.win32.WinNT;
+import org.apache.commons.io.Charsets;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import 
org.apache.nifi.processors.windows.event.log.jna.EventSubscribeXmlRenderingCallback;
+import org.apache.nifi.processors.windows.event.log.jna.WEvtApi;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@Tags({"ingest", "event", "windows"})
+@CapabilityDescription("Registers a Windows Event Log Subscribe Callback 
to receive FlowFiles from Events on Windows.  These can be filtered via channel 
and XPath.")
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description = "Will set 
a MIME type value of application/xml.")
+})
+public class EvtSubscribe extends AbstractSessionFactoryProcessor {
+public static final String DEFAULT_CHANNEL = "System";
+public static final String DEFAULT_XPATH = "*";
+public static final int DEFAULT_MAX_BUFFER = 1024 * 1024;
+public static final int DEFAULT_MAX_QUEUE_SIZE = 1024;
+
+public static final PropertyDescriptor CHANNEL = new 
PropertyDescriptor.Builder()
+.name("channel")
+.displayName("Channel")
+.required(true)
+.defaultValue(DEFAULT_CHANNEL)
+.description("The Windows Event Log Channel to listen to.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor QUERY = new 
PropertyDescriptor.Builder()
+.name("query")
+.displayName("XPath Query")
+.required(true)
+.defaultValue(DEFAULT_XPATH)
+.description("XPath Query to filter events. (See 
https://msdn.microsoft.com/en-us/library/windows/desktop/dd996910(v=vs.85).aspx 
for examples.)")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor MAX_BUFFER_SIZE = new

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r67435351
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-processors/src/main/java/org/apache/nifi/processors/windows/event/log/EvtSubscribe.java
 ---
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.windows.event.log;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.net.MediaType;
--- End diff --

Are these two imports the only ones from google guava? Neither seem to 
warrant bringing in an additional dependency and should be removed if able


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] release line management

2016-06-16 Thread Andy LoPresto

I agree with all those points, but a big +1 to this:

> By making any release (incremental, major, or minor) we should be
> saying that it is in a mature state and we should certainly be saying
> there is no known "loss of capability or risk to their data".  For the
> examples you provide I believe those fall under the guidance that
> states we would backport and produce a release.
> 
> Users will never uptake new versions as fast as we'd like.  That is
> why we should focus on producing higher and higher quality releases
> and why we should focus on making the process of upgrading as smooth
> as possible.

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jun 16, 2016, at 1:14 PM, Joe Witt  wrote:
> 
> JoeS,
> 
> Understood.  Some responses for the points you are raising.
> 
> ---
> "At a minimum, everything needed to upgrade from X to Y needs to be
> explicitly covered in a migration document"
> 
> Agreed.  That is the intent of this document [1] primarily.  There is
> also useful information related to how the community has discussed
> handling such things here [2] and here [3].
> 
> ---
> "At a minimum, everything needed to upgrade from X to Y needs to
> be...where possible and practical...simplified by tools or scripts"
> 
> Agreed.  That I think is well reflected in JIRAs and feature proposals
> which can be disruptive in nature and also in techniques that have
> been adopted for some time.  For some concrete examples of the
> commitment to easing and promoting upgrades:
> 
> 1) The old flow configuration is designed to be automatically
> converted/honored on startup
> 2) The old authorization file configuration is automatically converted
> to the new policies model on startup.
> 3) Placement/dimensions of components on the old flow configuration
> are automatically scaled to look nice in the new UI.
> 4) Key features such as site to site and flow file repository have
> been build to store and honor versions so that new versions can read
> and port them.  This will also occur with the flow configuration
> itself going forward.
> 
> ---
> "I think any change to reduce support for 0.x should depend on first
> reaching a mature state with 1.x"
> 
> By making any release (incremental, major, or minor) we should be
> saying that it is in a mature state and we should certainly be saying
> there is no known "loss of capability or risk to their data".  For the
> examples you provide I believe those fall under the guidance that
> states we would backport and produce a release.
> 
> Users will never uptake new versions as fast as we'd like.  That is
> why we should focus on producing higher and higher quality releases
> and why we should focus on making the process of upgrading as smooth
> as possible.
> 
> [1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
> [2] https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi
> [3] https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme
> [4] 
> https://cwiki.apache.org/confluence/display/NIFI/Git+Branching+and+Release+Line+Management
> 
> Thanks
> Joe
> 
> On Wed, Jun 15, 2016 at 3:59 PM, Joe Skora  wrote:
>> I think any change to reduce support for 0.x should depend on first
>> reaching a mature state with 1.x.  Only after the new release is stable can
>> 0.x users upgrade to 1.x without loss of capability or risk to their data.
>> 
>> I don't think the guidance needs significant changes, but from an
>> enterprise use perspective this line concerns.
>> 
>>> When the release line on master has no releases we will back port...
>> 
>> I'd prefer something along the lines of this.
>> 
>>> Until the release line on master has reached a mature release we will back
>>> port...
>>> 
>> 
>> Once an enterprise commits to a deployment of something like NiFi, changes
>> are expensive and disruptive.  At a minimum, everything needed to upgrade
>> from X to Y needs to be explicitly covered in a migration document, and
>> where possible and practical simplified by tools or scripts.  I'm not sure
>> how to put that commitment in writing since it can't be an absolute, but a
>> statement to that effect seems missing in the guidance.
>> 
>> On Wed, Jun 15, 2016 at 3:22 PM, Tony Kurc  wrote:
>> 
>>> Yes, I think we need to probably tighten the language to take as much
>>> inference (and hence misunderstanding) out as possible.
>>> On Jun 15, 2016 5:59 PM, "Joe Witt"  wrote:
>>> 
 
 
 
 Tony, Joe,
 
 It sounds as though you would like to propose a change to this [1]
 release line management guidance that we generated as a result of the
 various discussions.
 
 Can you please propose some changes to that guidance?
 
 Of course, we must always work to support easy migration for
 developers and users any time we move minor or major releases

[GitHub] nifi issue #493: NIFI-1037 Created processor that handles HDFS' inotify even...

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/493
  
@jjmeyer0 LGTM +1 tested with workflows, mvn build OK.
Merged in master, but would you mind opening a specific PR for 0.x branch 
(Java 7 compatible):
GetHDFSEvents.java:[208,78] error: local variable e is accessed from within 
inner class; needs to be declared final


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #525: NIFI-1976 - Windows Event Log native processor

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/525#discussion_r67433123
  
--- Diff: 
nifi-nar-bundles/nifi-windows-event-log-bundle/nifi-windows-event-log-nar/src/main/resources/META-INF/LICENSE
 ---
@@ -0,0 +1,239 @@
+
+ Apache License
+   Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+  "License" shall mean the terms and conditions for use, reproduction,
+  and distribution as defined by Sections 1 through 9 of this document.
+
+  "Licensor" shall mean the copyright owner or entity authorized by
+  the copyright owner that is granting the License.
+
+  "Legal Entity" shall mean the union of the acting entity and all
+  other entities that control, are controlled by, or are under common
+  control with that entity. For the purposes of this definition,
+  "control" means (i) the power, direct or indirect, to cause the
+  direction or management of such entity, whether by contract or
+  otherwise, or (ii) ownership of fifty percent (50%) or more of the
+  outstanding shares, or (iii) beneficial ownership of such entity.
+
+  "You" (or "Your") shall mean an individual or Legal Entity
+  exercising permissions granted by this License.
+
+  "Source" form shall mean the preferred form for making modifications,
+  including but not limited to software source code, documentation
+  source, and configuration files.
+
+  "Object" form shall mean any form resulting from mechanical
+  transformation or translation of a Source form, including but
+  not limited to compiled object code, generated documentation,
+  and conversions to other media types.
+
+  "Work" shall mean the work of authorship, whether in Source or
+  Object form, made available under the License, as indicated by a
+  copyright notice that is included in or attached to the work
+  (an example is provided in the Appendix below).
+
+  "Derivative Works" shall mean any work, whether in Source or Object
+  form, that is based on (or derived from) the Work and for which the
+  editorial revisions, annotations, elaborations, or other 
modifications
+  represent, as a whole, an original work of authorship. For the 
purposes
+  of this License, Derivative Works shall not include works that remain
+  separable from, or merely link (or bind by name) to the interfaces 
of,
+  the Work and Derivative Works thereof.
+
+  "Contribution" shall mean any work of authorship, including
+  the original version of the Work and any modifications or additions
+  to that Work or Derivative Works thereof, that is intentionally
+  submitted to Licensor for inclusion in the Work by the copyright 
owner
+  or by an individual or Legal Entity authorized to submit on behalf of
+  the copyright owner. For the purposes of this definition, "submitted"
+  means any form of electronic, verbal, or written communication sent
+  to the Licensor or its representatives, including but not limited to
+  communication on electronic mailing lists, source code control 
systems,
+  and issue tracking systems that are managed by, or on behalf of, the
+  Licensor for the purpose of discussing and improving the Work, but
+  excluding communication that is conspicuously marked or otherwise
+  designated in writing by the copyright owner as "Not a Contribution."
+
+  "Contributor" shall mean Licensor and any individual or Legal Entity
+  on behalf of whom a Contribution has been received by Licensor and
+  subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  copyright license to reproduce, prepare Derivative Works of,
+  publicly display, publicly perform, sublicense, and distribute the
+  Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  (except as stated in this section) patent license to make, have made,
+  use, offer to sell, sell, import, and otherwise transfer the Work,
+  where such license applies

[GitHub] nifi pull request #493: NIFI-1037 Created processor that handles HDFS' inoti...

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/493


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #252: NIFI-615 - Create a processor to extract WAV file character...

Github user jskora commented on the issue:

https://github.com/apache/nifi/pull/252
  
Ok, I think I've resolved the problems.  The Tika conflict was fixed by 
udpating Tika and Metadata Extractor dependency versions and adjusting 
ExtractImageMetadata for new attribute names in the newer parser.

I've stored my current branch on my Github branch 
[NIFI-615-v2](https://github.com/jskora/nifi/tree/NIFI-615-v2).  I still need 
to rebase that, I probably won't be able to do that until Monday.  Please take 
a look if you get a chance and I'll try to get the pull updated on Monday.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #532: NIFI-1941: Child group contents in exported templates

Github user jvwing commented on the issue:

https://github.com/apache/nifi/pull/532
  
Reviewing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #534: Fix for NIFI-1838 & NIFI-1152

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/534#discussion_r67428396
  
--- Diff: 
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/processors/script/InvokeScriptedProcessor.java
 ---
@@ -92,11 +92,10 @@
 logger.error(message, t);
 }
 }
-} else {
-// Return defaults for now
-relationships.add(REL_SUCCESS);
-relationships.add(REL_FAILURE);
 }
+// Add defaults
+relationships.add(REL_SUCCESS);
+relationships.add(REL_FAILURE);
--- End diff --

@mattyb149 do you mind having a look? I don't think this is what we want 
here, I'd rather say that if we need to change something it would be on tests 
side.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #458: NIFI-1829 - Create new DebugFlow processor.

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/458


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #534: Fix for NIFI-1838 & NIFI-1152

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/534#discussion_r67424962
  
--- Diff: 
nifi-mock/src/test/java/org/apache/nifi/util/TestMockProcessSession.java ---
@@ -55,4 +55,35 @@ public void onTrigger(final ProcessContext ctx, final 
ProcessSession session) th
 }
 
 }
+
+
+@Test(expected = AssertionError.class)
+public void testNonExistentRelationFromProcessor() {
+
TestRunners.newTestRunner(NonExistentRelationTestProcessor.class).run();
+}
+
+protected static class NonExistentRelationTestProcessor extends 
PoorlyBehavedProcessor {
+
+static final Relationship REL_NON_EXISTENT = new 
Relationship.Builder()
+.name("NonExistent Relation")
+.build();
+
+public Set relationships = 
Collections.singleton(REL_FAILURE);
+
+@Override
+public Set getRelationships() {
+return relationships;
--- End diff --

It looks to me this is unrelated. This is a test class absolutely 
independent from scripting processors, you don't need to change the scope of 
the relationship in ``PoorlyBehavedProcessor``and you can return an empty set 
of relationships to execute your unit test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #252: NIFI-615 - Create a processor to extract WAV file character...

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/252
  
Thanks for the updates @jskora, unless if you say otherwise I'm going to 
assume you're gonna get this knocked out for 0.7.0 as soon as possible. If you 
think you need more time and can have it slide then feel free to remove the 
0.7.0 tag.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #517: NIFI-1994: Fixed issues with controller services and templa...

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/517
  
@markap14 When exporting a template with controller services at multiple 
levels, it appears the services all end up at the top level group. These should 
probably retain the same scoping as the source system. With all the services at 
the top level, it is difficult to verify this fix (if I understand it 
correctly).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #534: Fix for NIFI-1838 & NIFI-1152

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/534#discussion_r67423663
  
--- Diff: 
nifi-mock/src/main/java/org/apache/nifi/util/MockProcessSession.java ---
@@ -793,13 +798,21 @@ private void validateState(final FlowFile flowFile) {
 throw new FlowFileHandlingException(flowFile + " is not the 
most recent version of this flow file within this session");
 }
 
+
 for (final List flowFiles : transferMap.values()) {
 if (flowFiles.contains(flowFile)) {
 throw new IllegalStateException(flowFile + " has already 
been transferred");
 }
 }
 }
 
+private void validateDestinationRelation(final Relationship 
relationship) {
+if (!relationships.contains(relationship)) {
+rollback();
--- End diff --

This class is limited to testing purpose, so throwing an exception would 
make fail the associated test. I think this is what we want and in this case 
rollback is not needed. In a real context we should not be in such situation 
where the relationship does not exist.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #252: NIFI-615 - Create a processor to extract WAV file character...

Github user jskora commented on the issue:

https://github.com/apache/nifi/pull/252
  
Ironically, this boils down to a version dependency conflict between 
ExtractMediaMetadata which uses Tika 1.7 which uses Drew Noakes' 
Metadata-Extract 2.6.2 and the existing ExtractImageMetadata which used Drew 
Noakes' Metadata-Extract 2.7.2.

I'm looking into ways to resolve this.  One option may be to return 
ExtractImageMetadata to it's own nifi-image-bundle nar and create a new 
nifi-media-bundle-nar instead the rename of n-image-b-n to n-media-b-n that I 
originally did.  Another is to look into newer Tika versions, but I don't know 
if that will create other problems.  I'll keep you posted.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: putTCP nifi 0.7?

2016-06-16 Thread Joe Percivall

Hello,

It is not currently marked as something that will go in 0.7.0 but I will review 
it to see where it stands. If it is able to be completed shortly (contributor 
addresses of my feedback) then it will make it in but if not then unfortunately 
it would slide to a later release.


This ticket falls into a category of unversioned "patch available" tickets 
(tickets that don't have a fix version and have a status "patch available") 
that need to be addressed. As the Release Manager for 0.7.0 I will take a look 
at the unversioned "patch available" tickets to see which of them can be 
finished off for 0.7.0 and ping the appropriate committers to help out.

Joe
- - - - - - Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com




On Thursday, June 16, 2016 2:00 PM, Ryan Ward  wrote:
Will this one make 0.7?

https://issues.apache.org/jira/browse/NIFI-1834

[GitHub] nifi pull request #535: NIFI-2041: Unable to refresh Controller Services whe...

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/535


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #535: NIFI-2041: Unable to refresh Controller Services when scope...

2016-06-16 Thread bbende

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/535
  
+1 looks good


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #535: NIFI-2041: Unable to refresh Controller Services whe...

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/535

NIFI-2041: Unable to refresh Controller Services when scoped by Process 
Group

- Fixing issue when attempting to refresh Controller Services when scoped 
by Process Group.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-2041

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/535.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #535


commit 0c4de4f81d6a1da6f5ce32978347ecf228036bed
Author: Matt Gilman 
Date:   2016-06-16T20:45:11Z

NIFI-2041:
- Fixing issue when attempting to refresh Controller Services when scoped 
by Process Group.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #252: NIFI-615 - Create a processor to extract WAV file character...

2016-06-16 Thread McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)

Github user jskora commented on the issue:

https://github.com/apache/nifi/pull/252
  
So, the problems you had with the JPG file are two separate issues.

- The default `BodyContentHandler` passed to the parser can only handle 
100,000 bytes, so any file larger than that produces that message and is only 
partially parsed.  **_I fixed this by adding an optional `Content Buffer Size` 
property to allow that to be increased or set to unlimited and related tests._**
- The Tika JPEG parser appears to have a reference error.  It references 
the class `com.drew.lang.BufferReader` but that does not appear to be in the 
Tika package or the version of the Drew Noakes Metadata Extractor package that 
Tika references.  **_I'm looking into whether updating to a newer Tika will fix 
this or if we have dependency conflicts causing the problem._**


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] release line management

2016-06-16 Thread Joe Witt

JoeS,

Understood.  Some responses for the points you are raising.

---
"At a minimum, everything needed to upgrade from X to Y needs to be
explicitly covered in a migration document"

Agreed.  That is the intent of this document [1] primarily.  There is
also useful information related to how the community has discussed
handling such things here [2] and here [3].

---
"At a minimum, everything needed to upgrade from X to Y needs to
be...where possible and practical...simplified by tools or scripts"

Agreed.  That I think is well reflected in JIRAs and feature proposals
which can be disruptive in nature and also in techniques that have
been adopted for some time.  For some concrete examples of the
commitment to easing and promoting upgrades:

1) The old flow configuration is designed to be automatically
converted/honored on startup
2) The old authorization file configuration is automatically converted
to the new policies model on startup.
3) Placement/dimensions of components on the old flow configuration
are automatically scaled to look nice in the new UI.
4) Key features such as site to site and flow file repository have
been build to store and honor versions so that new versions can read
and port them.  This will also occur with the flow configuration
itself going forward.

---
"I think any change to reduce support for 0.x should depend on first
reaching a mature state with 1.x"

By making any release (incremental, major, or minor) we should be
saying that it is in a mature state and we should certainly be saying
there is no known "loss of capability or risk to their data".  For the
examples you provide I believe those fall under the guidance that
states we would backport and produce a release.

Users will never uptake new versions as fast as we'd like.  That is
why we should focus on producing higher and higher quality releases
and why we should focus on making the process of upgrading as smooth
as possible.

[1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
[2] https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi
[3] https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme
[4] 
https://cwiki.apache.org/confluence/display/NIFI/Git+Branching+and+Release+Line+Management

Thanks
Joe

On Wed, Jun 15, 2016 at 3:59 PM, Joe Skora  wrote:
> I think any change to reduce support for 0.x should depend on first
> reaching a mature state with 1.x.  Only after the new release is stable can
> 0.x users upgrade to 1.x without loss of capability or risk to their data.
>
> I don't think the guidance needs significant changes, but from an
> enterprise use perspective this line concerns.
>
>> When the release line on master has no releases we will back port...
>
> I'd prefer something along the lines of this.
>
>> Until the release line on master has reached a mature release we will back
>> port...
>>
>
> Once an enterprise commits to a deployment of something like NiFi, changes
> are expensive and disruptive.  At a minimum, everything needed to upgrade
> from X to Y needs to be explicitly covered in a migration document, and
> where possible and practical simplified by tools or scripts.  I'm not sure
> how to put that commitment in writing since it can't be an absolute, but a
> statement to that effect seems missing in the guidance.
>
> On Wed, Jun 15, 2016 at 3:22 PM, Tony Kurc  wrote:
>
>> Yes, I think we need to probably tighten the language to take as much
>> inference (and hence misunderstanding) out as possible.
>> On Jun 15, 2016 5:59 PM, "Joe Witt"  wrote:
>>
>> > 
>> >
>> >
>> > Tony, Joe,
>> >
>> > It sounds as though you would like to propose a change to this [1]
>> > release line management guidance that we generated as a result of the
>> > various discussions.
>> >
>> > Can you please propose some changes to that guidance?
>> >
>> > Of course, we must always work to support easy migration for
>> > developers and users any time we move minor or major releases out.
>> >
>> > [1]
>> >
>> https://cwiki.apache.org/confluence/display/NIFI/Git+Branching+and+Release+Line+Management
>> >
>> > Thanks
>> > Joe
>> >
>> > On Wed, Jun 15, 2016 at 11:14 AM, Tony Kurc  wrote:
>> > > I remember the thread, but it seems I need to reread the thread -
>> > honestly
>> > > the comment did take me by surprise, I think we may have used a few
>> terms
>> > > that were left open to interpretation.
>> > > On Jun 15, 2016 5:06 PM, "Joe Skora"  wrote:
>> > >
>> > >> I agree with Tony on this.
>> > >>
>> > >> The point of release branching, etc. is to make it possible to
>> maintain
>> > an
>> > >> older version while building the newer version.  Yes, it is a
>> nuisance,
>> > but
>> > >> not nearly as much as a new version is for users if it is has
>> > significant
>> > >> changes and/or bugs.  Except in cases where there is a captive user
>> > base,
>> > >> failure to maintain the prior version while

log event processing

2016-06-16 Thread vinbalan

Among all entries, I want to capture some specific log events from the
application log and process it continuously. What processor should be used ?



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/log-event-processing-tp11646.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: GetJMSQueue does not detect dead connections

I’ll do it.  Thanks.

Chris McDermott
 
Remote Business Analytics
STaTS/StoreFront Remote
HPE Storage
Hewlett Packard Enterprise
Mobile: +1 978-697-5315
 
https://www.storefrontremote.com

On 6/16/16, 3:19 PM, "Oleg Zhurakousky"  wrote:

>Yes, that is documentation bug.
>
>Chris would you mind raising a JIRA or I can do it.
>
>Cheers
>Oleg
>
>> On Jun 16, 2016, at 3:15 PM, Joe Witt  wrote:
>> 
>> Oleg - so to Chris' other comment about docs suggesting the ActiveMQ
>> lib path is not needed - is that a doc bug?
>> 
>> On Thu, Jun 16, 2016 at 3:13 PM, Oleg Zhurakousky
>>  wrote:
>>> Chris
>>> 
>>> That is correct.
>>> The idea was to make sure that we can support multiple clients and multiple 
>>> vendors since Get/Put only supported AMQP and only one version. The new JMS 
>>> support allows you to use any JMS vendor and the only extra work we are 
>>> asking you to do is to provide ConnectionFactory JAR(s).
>>> 
>>> Does that clarify?
>>> 
>>> Also, yeh tests I was referring to are 
>>> https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/test/java/org/apache/nifi/jms/processors
>>> 
>>> Let me know if you need more help
>>> Cheers
>>> Oleg
>>> On Jun 16, 2016, at 3:08 PM, McDermott, Chris Kevin (MSDU - 
>>> STaTS/StorefrontRemote) 
>>> > wrote:
>>> 
>>> So does that mean that I cannot use the AMQ client packaged with NiFi but 
>>> rather provide my own?
>>> 
>>> Sorry if I an being obtuse.
>>> 
>>> Chris McDermott
>>> 
>>> Remote Business Analytics
>>> STaTS/StoreFront Remote
>>> HPE Storage
>>> Hewlett Packard Enterprise
>>> Mobile: +1 978-697-5315
>>> 
>>> https://www.storefrontremote.com
>>> 
>>> On 6/16/16, 2:53 PM, "Oleg Zhurakousky"  
>>> wrote:
>>> 
>>> Yes, you can probably look at the test case for it since it uses embedded 
>>> AMQP.
>>> 
>>> Let m know if you need more help with it.
>>> 
>>> Cheers
>>> Oleg
>>> On Jun 16, 2016, at 2:50 PM, McDermott, Chris Kevin (MSDU - 
>>> STaTS/StorefrontRemote)  wrote:
>>> 
>>> Thanks, Oleg.
>>> 
>>> Do you have an example of how to configure the JMSConnectionFactoryProvider 
>>> to work with AMQ?
>>> 
>>> The documentation says that the MQ Client Libraries path is optional with 
>>> org.apache.activemq.ActiveMQConnectionFactory but I am find that is not the 
>>> case.
>>> 
>>> Thanks,
>>> 
>>> Chris McDermott
>>> 
>>> Remote Business Analytics
>>> STaTS/StoreFront Remote
>>> HPE Storage
>>> Hewlett Packard Enterprise
>>> Mobile: +1 978-697-5315
>>> 
>>> https://www.storefrontremote.com
>>> 
>>> On 6/16/16, 1:43 PM, "Oleg Zhurakousky"  
>>> wrote:
>>> 
>>> Chris
>>> 
>>> Given that we are deprecating Get/PutJMS* in favor of Publish/SubscribeJMS, 
>>> I’d suggest start using those once.
>>> 
>>> Cheers
>>> Oleg
>>> 
>>> 
>>> On Jun 16, 2016, at 1:34 PM, McDermott, Chris Kevin (MSDU - 
>>> STaTS/StorefrontRemote)  wrote:
>>> 
>>> Folks,
>>> 
>>> I’ve been trying to test my GetJMSQueue configuration so that it detects a 
>>> dead broker connection and fails over to an alternate broker.  When I say 
>>> dead connection I mean TCP connection that has not been closed but is no 
>>> longer passing traffic.  In the real world this typically happens when 
>>> broker server crashes and so it does not reset the open connections.  For 
>>> my test case I am using iptables to block traffic.
>>> 
>>> This is the connection URI I am using
>>> 
>>> failover:(tcp://host2:61616,tcp://host1:61616)?randomize=false=3000=3=3=1=0
>>> 
>>> They key parameters here are soTimeout=3 and soWriteTimeout=3
>>> 
>>> These set a 30 second timeout on socket reads and writes.  I’m not sure if 
>>> these are necessary since I believe the JMSConsumer classes specifies its 
>>> own timeout according to the processor configuration.  The important thing 
>>> to note is that when one of these timeouts occurs the AMQ client does not 
>>> close the connection.
>>> 
>>> I believe the deficiency here is that JMSConsumer does not consider the 
>>> possibility that the connection is dead.   The problem with this is that an 
>>> attempt to reconnect and failover to an alternate broker is not made.
>>> 
>>> I think the fix would involve counting the number of sequential empty 
>>> responses on the connection and then closing the connection once that 
>>> number crosses some threshold.  Then subsequent onTrigger() would cause a 
>>> new connection attempt.
>>> 
>>> Thoughts?
>>> 
>>> Chris McDermott
>>> 
>>> Remote Business Analytics
>>> STaTS/StoreFront Remote
>>> HPE Storage
>>> Hewlett Packard Enterprise
>>> Mobile: +1 978-697-5315
>>> 
>>> https://www.storefrontremote.com
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>

Re: GetJMSQueue does not detect dead connections

Yes, that is documentation bug.

Chris would you mind raising a JIRA or I can do it.

Cheers
Oleg

> On Jun 16, 2016, at 3:15 PM, Joe Witt  wrote:
> 
> Oleg - so to Chris' other comment about docs suggesting the ActiveMQ
> lib path is not needed - is that a doc bug?
> 
> On Thu, Jun 16, 2016 at 3:13 PM, Oleg Zhurakousky
>  wrote:
>> Chris
>> 
>> That is correct.
>> The idea was to make sure that we can support multiple clients and multiple 
>> vendors since Get/Put only supported AMQP and only one version. The new JMS 
>> support allows you to use any JMS vendor and the only extra work we are 
>> asking you to do is to provide ConnectionFactory JAR(s).
>> 
>> Does that clarify?
>> 
>> Also, yeh tests I was referring to are 
>> https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/test/java/org/apache/nifi/jms/processors
>> 
>> Let me know if you need more help
>> Cheers
>> Oleg
>> On Jun 16, 2016, at 3:08 PM, McDermott, Chris Kevin (MSDU - 
>> STaTS/StorefrontRemote) 
>> > wrote:
>> 
>> So does that mean that I cannot use the AMQ client packaged with NiFi but 
>> rather provide my own?
>> 
>> Sorry if I an being obtuse.
>> 
>> Chris McDermott
>> 
>> Remote Business Analytics
>> STaTS/StoreFront Remote
>> HPE Storage
>> Hewlett Packard Enterprise
>> Mobile: +1 978-697-5315
>> 
>> https://www.storefrontremote.com
>> 
>> On 6/16/16, 2:53 PM, "Oleg Zhurakousky"  wrote:
>> 
>> Yes, you can probably look at the test case for it since it uses embedded 
>> AMQP.
>> 
>> Let m know if you need more help with it.
>> 
>> Cheers
>> Oleg
>> On Jun 16, 2016, at 2:50 PM, McDermott, Chris Kevin (MSDU - 
>> STaTS/StorefrontRemote)  wrote:
>> 
>> Thanks, Oleg.
>> 
>> Do you have an example of how to configure the JMSConnectionFactoryProvider 
>> to work with AMQ?
>> 
>> The documentation says that the MQ Client Libraries path is optional with 
>> org.apache.activemq.ActiveMQConnectionFactory but I am find that is not the 
>> case.
>> 
>> Thanks,
>> 
>> Chris McDermott
>> 
>> Remote Business Analytics
>> STaTS/StoreFront Remote
>> HPE Storage
>> Hewlett Packard Enterprise
>> Mobile: +1 978-697-5315
>> 
>> https://www.storefrontremote.com
>> 
>> On 6/16/16, 1:43 PM, "Oleg Zhurakousky"  wrote:
>> 
>> Chris
>> 
>> Given that we are deprecating Get/PutJMS* in favor of Publish/SubscribeJMS, 
>> I’d suggest start using those once.
>> 
>> Cheers
>> Oleg
>> 
>> 
>> On Jun 16, 2016, at 1:34 PM, McDermott, Chris Kevin (MSDU - 
>> STaTS/StorefrontRemote)  wrote:
>> 
>> Folks,
>> 
>> I’ve been trying to test my GetJMSQueue configuration so that it detects a 
>> dead broker connection and fails over to an alternate broker.  When I say 
>> dead connection I mean TCP connection that has not been closed but is no 
>> longer passing traffic.  In the real world this typically happens when 
>> broker server crashes and so it does not reset the open connections.  For my 
>> test case I am using iptables to block traffic.
>> 
>> This is the connection URI I am using
>> 
>> failover:(tcp://host2:61616,tcp://host1:61616)?randomize=false=3000=3=3=1=0
>> 
>> They key parameters here are soTimeout=3 and soWriteTimeout=3
>> 
>> These set a 30 second timeout on socket reads and writes.  I’m not sure if 
>> these are necessary since I believe the JMSConsumer classes specifies its 
>> own timeout according to the processor configuration.  The important thing 
>> to note is that when one of these timeouts occurs the AMQ client does not 
>> close the connection.
>> 
>> I believe the deficiency here is that JMSConsumer does not consider the 
>> possibility that the connection is dead.   The problem with this is that an 
>> attempt to reconnect and failover to an alternate broker is not made.
>> 
>> I think the fix would involve counting the number of sequential empty 
>> responses on the connection and then closing the connection once that number 
>> crosses some threshold.  Then subsequent onTrigger() would cause a new 
>> connection attempt.
>> 
>> Thoughts?
>> 
>> Chris McDermott
>> 
>> Remote Business Analytics
>> STaTS/StoreFront Remote
>> HPE Storage
>> Hewlett Packard Enterprise
>> Mobile: +1 978-697-5315
>> 
>> https://www.storefrontremote.com
>> 
>> 
>> 
>> 
>> 
>

[GitHub] nifi issue #526: NIFI-1952 Create REST endpoints for user/group/policy manag...

2016-06-16 Thread jtstorck

Github user jtstorck commented on the issue:

https://github.com/apache/nifi/pull/526
  
@mcgilman I've addressed most of your comments on the PR aside from 
renaming Group to UserGroup, which I still plan on doing soon.  Please take a 
look at my latest commit on the PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: GetJMSQueue does not detect dead connections

2016-06-16 Thread Joe Witt

Oleg - so to Chris' other comment about docs suggesting the ActiveMQ
lib path is not needed - is that a doc bug?

On Thu, Jun 16, 2016 at 3:13 PM, Oleg Zhurakousky
 wrote:
> Chris
>
> That is correct.
> The idea was to make sure that we can support multiple clients and multiple 
> vendors since Get/Put only supported AMQP and only one version. The new JMS 
> support allows you to use any JMS vendor and the only extra work we are 
> asking you to do is to provide ConnectionFactory JAR(s).
>
> Does that clarify?
>
> Also, yeh tests I was referring to are 
> https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/test/java/org/apache/nifi/jms/processors
>
> Let me know if you need more help
> Cheers
> Oleg
> On Jun 16, 2016, at 3:08 PM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote) 
> > wrote:
>
> So does that mean that I cannot use the AMQ client packaged with NiFi but 
> rather provide my own?
>
> Sorry if I an being obtuse.
>
> Chris McDermott
>
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
>
> https://www.storefrontremote.com
>
> On 6/16/16, 2:53 PM, "Oleg Zhurakousky"  wrote:
>
> Yes, you can probably look at the test case for it since it uses embedded 
> AMQP.
>
> Let m know if you need more help with it.
>
> Cheers
> Oleg
> On Jun 16, 2016, at 2:50 PM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote)  wrote:
>
> Thanks, Oleg.
>
> Do you have an example of how to configure the JMSConnectionFactoryProvider 
> to work with AMQ?
>
> The documentation says that the MQ Client Libraries path is optional with 
> org.apache.activemq.ActiveMQConnectionFactory but I am find that is not the 
> case.
>
> Thanks,
>
> Chris McDermott
>
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
>
> https://www.storefrontremote.com
>
> On 6/16/16, 1:43 PM, "Oleg Zhurakousky"  wrote:
>
> Chris
>
> Given that we are deprecating Get/PutJMS* in favor of Publish/SubscribeJMS, 
> I’d suggest start using those once.
>
> Cheers
> Oleg
>
>
> On Jun 16, 2016, at 1:34 PM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote)  wrote:
>
> Folks,
>
> I’ve been trying to test my GetJMSQueue configuration so that it detects a 
> dead broker connection and fails over to an alternate broker.  When I say 
> dead connection I mean TCP connection that has not been closed but is no 
> longer passing traffic.  In the real world this typically happens when broker 
> server crashes and so it does not reset the open connections.  For my test 
> case I am using iptables to block traffic.
>
> This is the connection URI I am using
>
> failover:(tcp://host2:61616,tcp://host1:61616)?randomize=false=3000=3=3=1=0
>
> They key parameters here are soTimeout=3 and soWriteTimeout=3
>
> These set a 30 second timeout on socket reads and writes.  I’m not sure if 
> these are necessary since I believe the JMSConsumer classes specifies its own 
> timeout according to the processor configuration.  The important thing to 
> note is that when one of these timeouts occurs the AMQ client does not close 
> the connection.
>
> I believe the deficiency here is that JMSConsumer does not consider the 
> possibility that the connection is dead.   The problem with this is that an 
> attempt to reconnect and failover to an alternate broker is not made.
>
> I think the fix would involve counting the number of sequential empty 
> responses on the connection and then closing the connection once that number 
> crosses some threshold.  Then subsequent onTrigger() would cause a new 
> connection attempt.
>
> Thoughts?
>
> Chris McDermott
>
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
>
> https://www.storefrontremote.com
>
>
>
>
>

Re: GetJMSQueue does not detect dead connections

2016-06-16 Thread McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)

Chris

That is correct.
The idea was to make sure that we can support multiple clients and multiple
vendors since Get/Put only supported AMQP and only one version. The new JMS
support allows you to use any JMS vendor and the only extra work we are asking
you to do is to provide ConnectionFactory JAR(s).

Does that clarify?

Also, yeh tests I was referring to are
https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/test/java/org/apache/nifi/jms/processors

Let me know if you need more help
Cheers
Oleg
On Jun 16, 2016, at 3:08 PM, McDermott, Chris Kevin (MSDU -
STaTS/StorefrontRemote)
> wrote:

So does that mean that I cannot use the AMQ client packaged with NiFi but
rather provide my own?

Sorry if I an being obtuse.

Chris McDermott

Remote Business Analytics
STaTS/StoreFront Remote
HPE Storage
Hewlett Packard Enterprise
Mobile: +1 978-697-5315

https://www.storefrontremote.com

On 6/16/16, 2:53 PM, "Oleg Zhurakousky" wrote:

Yes, you can probably look at the test case for it since it uses embedded AMQP.

Let m know if you need more help with it.

Cheers
Oleg
On Jun 16, 2016, at 2:50 PM, McDermott, Chris Kevin (MSDU -
STaTS/StorefrontRemote) wrote:

Thanks, Oleg.

Do you have an example of how to configure the JMSConnectionFactoryProvider to
work with AMQ?

The documentation says that the MQ Client Libraries path is optional with
org.apache.activemq.ActiveMQConnectionFactory but I am find that is not the
case.

Thanks,

Chris McDermott

Remote Business Analytics
STaTS/StoreFront Remote
HPE Storage
Hewlett Packard Enterprise
Mobile: +1 978-697-5315

https://www.storefrontremote.com

On 6/16/16, 1:43 PM, "Oleg Zhurakousky" wrote:

Chris

Given that we are deprecating Get/PutJMS* in favor of Publish/SubscribeJMS, I’d
suggest start using those once.

Cheers
Oleg

On Jun 16, 2016, at 1:34 PM, McDermott, Chris Kevin (MSDU -
STaTS/StorefrontRemote) wrote:

Folks,

I’ve been trying to test my GetJMSQueue configuration so that it detects a dead
broker connection and fails over to an alternate broker. When I say dead
connection I mean TCP connection that has not been closed but is no longer
passing traffic. In the real world this typically happens when broker server
crashes and so it does not reset the open connections. For my test case I am
using iptables to block traffic.

This is the connection URI I am using

failover:(tcp://host2:61616,tcp://host1:61616)?randomize=false=3000=3=3=1=0

They key parameters here are soTimeout=3 and soWriteTimeout=3

These set a 30 second timeout on socket reads and writes. I’m not sure if
these are necessary since I believe the JMSConsumer classes specifies its own
timeout according to the processor configuration. The important thing to note
is that when one of these timeouts occurs the AMQ client does not close the
connection.

I believe the deficiency here is that JMSConsumer does not consider the
possibility that the connection is dead. The problem with this is that an
attempt to reconnect and failover to an alternate broker is not made.

I think the fix would involve counting the number of sequential empty responses
on the connection and then closing the connection once that number crosses some
threshold. Then subsequent onTrigger() would cause a new connection attempt.

Thoughts?

Chris McDermott

Remote Business Analytics
STaTS/StoreFront Remote
HPE Storage
Hewlett Packard Enterprise
Mobile: +1 978-697-5315

https://www.storefrontremote.com

Re: GetJMSQueue does not detect dead connections

So does that mean that I cannot use the AMQ client packaged with NiFi but 
rather provide my own?

Sorry if I an being obtuse.

Chris McDermott

Remote Business Analytics
STaTS/StoreFront Remote
HPE Storage
Hewlett Packard Enterprise
Mobile: +1 978-697-5315

https://www.storefrontremote.com

On 6/16/16, 2:53 PM, "Oleg Zhurakousky"  wrote:

>Yes, you can probably look at the test case for it since it uses embedded AMQP.
>
>Let m know if you need more help with it.
>
>Cheers
>Oleg
>> On Jun 16, 2016, at 2:50 PM, McDermott, Chris Kevin (MSDU - 
>> STaTS/StorefrontRemote)  wrote:
>> 
>> Thanks, Oleg.  
>> 
>> Do you have an example of how to configure the JMSConnectionFactoryProvider 
>> to work with AMQ?
>> 
>> The documentation says that the MQ Client Libraries path is optional with 
>> org.apache.activemq.ActiveMQConnectionFactory but I am find that is not the 
>> case.
>> 
>> Thanks,
>> 
>> Chris McDermott
>> 
>> Remote Business Analytics
>> STaTS/StoreFront Remote
>> HPE Storage
>> Hewlett Packard Enterprise
>> Mobile: +1 978-697-5315
>> 
>> https://www.storefrontremote.com
>> 
>> On 6/16/16, 1:43 PM, "Oleg Zhurakousky"  wrote:
>> 
>>> Chris 
>>> 
>>> Given that we are deprecating Get/PutJMS* in favor of Publish/SubscribeJMS, 
>>> I’d suggest start using those once.
>>> 
>>> Cheers
>>> Oleg
>>> 
>>> 
 On Jun 16, 2016, at 1:34 PM, McDermott, Chris Kevin (MSDU - 
 STaTS/StorefrontRemote)  wrote:

 Folks,

 I’ve been trying to test my GetJMSQueue configuration so that it detects a 
 dead broker connection and fails over to an alternate broker.  When I say 
 dead connection I mean TCP connection that has not been closed but is no 
 longer passing traffic.  In the real world this typically happens when 
 broker server crashes and so it does not reset the open connections.  For 
 my test case I am using iptables to block traffic.

 This is the connection URI I am using

 failover:(tcp://host2:61616,tcp://host1:61616)?randomize=false=3000=3=3=1=0

 They key parameters here are soTimeout=3 and soWriteTimeout=3

 These set a 30 second timeout on socket reads and writes.  I’m not sure if 
 these are necessary since I believe the JMSConsumer classes specifies its 
 own timeout according to the processor configuration.  The important thing 
 to note is that when one of these timeouts occurs the AMQ client does not 
 close the connection.

 I believe the deficiency here is that JMSConsumer does not consider the 
 possibility that the connection is dead.   The problem with this is that 
 an attempt to reconnect and failover to an alternate broker is not made.

 I think the fix would involve counting the number of sequential empty 
 responses on the connection and then closing the connection once that 
 number crosses some threshold.  Then subsequent onTrigger() would cause a 
 new connection attempt.

 Thoughts?

 Chris McDermott

 Remote Business Analytics
 STaTS/StoreFront Remote
 HPE Storage
 Hewlett Packard Enterprise
 Mobile: +1 978-697-5315

 https://www.storefrontremote.com
>>> 
>> 
>

[GitHub] nifi pull request #252: NIFI-615 - Create a processor to extract WAV file ch...

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/252#discussion_r67402505
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.;)
+@WritesAttributes({@WritesAttribute(attribute = "tika.", 
description = "The extracted content metadata "
++ "will be inserted with the attribute name \"tika.\". 
")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+public static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new 
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to 
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will 
process all of them.")
+.required(false)
+
.addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor FILENAME_FILTER = new 
PropertyDescriptor.Builder()
+.name("File Name Filter")
+.description("A regular expression identifying file names 
which metadata should extracted.  As flowfiles"
+

Re: GetJMSQueue does not detect dead connections

2016-06-16 Thread McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)

Yes, you can probably look at the test case for it since it uses embedded AMQP.

Let m know if you need more help with it.

Cheers
Oleg
> On Jun 16, 2016, at 2:50 PM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote)  wrote:
> 
> Thanks, Oleg.  
> 
> Do you have an example of how to configure the JMSConnectionFactoryProvider 
> to work with AMQ?
> 
> The documentation says that the MQ Client Libraries path is optional with 
> org.apache.activemq.ActiveMQConnectionFactory but I am find that is not the 
> case.
> 
> Thanks,
> 
> Chris McDermott
> 
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
> 
> https://www.storefrontremote.com
> 
> On 6/16/16, 1:43 PM, "Oleg Zhurakousky"  wrote:
> 
>> Chris 
>> 
>> Given that we are deprecating Get/PutJMS* in favor of Publish/SubscribeJMS, 
>> I’d suggest start using those once.
>> 
>> Cheers
>> Oleg
>> 
>> 
>>> On Jun 16, 2016, at 1:34 PM, McDermott, Chris Kevin (MSDU - 
>>> STaTS/StorefrontRemote)  wrote:
>>> 
>>> Folks,
>>> 
>>> I’ve been trying to test my GetJMSQueue configuration so that it detects a 
>>> dead broker connection and fails over to an alternate broker.  When I say 
>>> dead connection I mean TCP connection that has not been closed but is no 
>>> longer passing traffic.  In the real world this typically happens when 
>>> broker server crashes and so it does not reset the open connections.  For 
>>> my test case I am using iptables to block traffic.
>>> 
>>> This is the connection URI I am using
>>> 
>>> failover:(tcp://host2:61616,tcp://host1:61616)?randomize=false=3000=3=3=1=0
>>> 
>>> They key parameters here are soTimeout=3 and soWriteTimeout=3
>>> 
>>> These set a 30 second timeout on socket reads and writes.  I’m not sure if 
>>> these are necessary since I believe the JMSConsumer classes specifies its 
>>> own timeout according to the processor configuration.  The important thing 
>>> to note is that when one of these timeouts occurs the AMQ client does not 
>>> close the connection.
>>> 
>>> I believe the deficiency here is that JMSConsumer does not consider the 
>>> possibility that the connection is dead.   The problem with this is that an 
>>> attempt to reconnect and failover to an alternate broker is not made.
>>> 
>>> I think the fix would involve counting the number of sequential empty 
>>> responses on the connection and then closing the connection once that 
>>> number crosses some threshold.  Then subsequent onTrigger() would cause a 
>>> new connection attempt.
>>> 
>>> Thoughts?
>>> 
>>> Chris McDermott
>>> 
>>> Remote Business Analytics
>>> STaTS/StoreFront Remote
>>> HPE Storage
>>> Hewlett Packard Enterprise
>>> Mobile: +1 978-697-5315
>>> 
>>> https://www.storefrontremote.com
>> 
>

Re: GetJMSQueue does not detect dead connections

Thanks, Oleg.  

Do you have an example of how to configure the JMSConnectionFactoryProvider to 
work with AMQ?

The documentation says that the MQ Client Libraries path is optional with 
org.apache.activemq.ActiveMQConnectionFactory but I am find that is not the 
case.

Thanks,

Chris McDermott
 
Remote Business Analytics
STaTS/StoreFront Remote
HPE Storage
Hewlett Packard Enterprise
Mobile: +1 978-697-5315
 
https://www.storefrontremote.com

On 6/16/16, 1:43 PM, "Oleg Zhurakousky"  wrote:

>Chris 
>
>Given that we are deprecating Get/PutJMS* in favor of Publish/SubscribeJMS, 
>I’d suggest start using those once.
>
>Cheers
>Oleg
>
>
>> On Jun 16, 2016, at 1:34 PM, McDermott, Chris Kevin (MSDU - 
>> STaTS/StorefrontRemote)  wrote:
>> 
>> Folks,
>> 
>> I’ve been trying to test my GetJMSQueue configuration so that it detects a 
>> dead broker connection and fails over to an alternate broker.  When I say 
>> dead connection I mean TCP connection that has not been closed but is no 
>> longer passing traffic.  In the real world this typically happens when 
>> broker server crashes and so it does not reset the open connections.  For my 
>> test case I am using iptables to block traffic.
>> 
>> This is the connection URI I am using
>> 
>> failover:(tcp://host2:61616,tcp://host1:61616)?randomize=false=3000=3=3=1=0
>> 
>> They key parameters here are soTimeout=3 and soWriteTimeout=3
>> 
>> These set a 30 second timeout on socket reads and writes.  I’m not sure if 
>> these are necessary since I believe the JMSConsumer classes specifies its 
>> own timeout according to the processor configuration.  The important thing 
>> to note is that when one of these timeouts occurs the AMQ client does not 
>> close the connection.
>> 
>> I believe the deficiency here is that JMSConsumer does not consider the 
>> possibility that the connection is dead.   The problem with this is that an 
>> attempt to reconnect and failover to an alternate broker is not made.
>> 
>> I think the fix would involve counting the number of sequential empty 
>> responses on the connection and then closing the connection once that number 
>> crosses some threshold.  Then subsequent onTrigger() would cause a new 
>> connection attempt.
>> 
>> Thoughts?
>> 
>> Chris McDermott
>> 
>> Remote Business Analytics
>> STaTS/StoreFront Remote
>> HPE Storage
>> Hewlett Packard Enterprise
>> Mobile: +1 978-697-5315
>> 
>> https://www.storefrontremote.com
>

[GitHub] nifi pull request #534: Fix for NIFI-1838 & NIFI-1152

Github user PuspenduBanerjee commented on a diff in the pull request:

https://github.com/apache/nifi/pull/534#discussion_r67394594
  
--- Diff: 
nifi-mock/src/test/java/org/apache/nifi/util/TestMockProcessSession.java ---
@@ -36,7 +36,7 @@ public void testPenalizeFlowFileFromProcessor() {
 
 protected static class PoorlyBehavedProcessor extends 
AbstractProcessor {
 
-private static final Relationship REL_FAILURE = new 
Relationship.Builder()
+protected static final Relationship REL_FAILURE = new 
Relationship.Builder()
--- End diff --

Directly using it in Test NonExistentRelationTestProcessor public 
Set relationships = Collections.singleton(REL_FAILURE);


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #534: Fix for NIFI-1838 & NIFI-1152

Github user PuspenduBanerjee commented on a diff in the pull request:

https://github.com/apache/nifi/pull/534#discussion_r67394024
  
--- Diff: 
nifi-mock/src/main/java/org/apache/nifi/util/MockProcessSession.java ---
@@ -793,13 +798,21 @@ private void validateState(final FlowFile flowFile) {
 throw new FlowFileHandlingException(flowFile + " is not the 
most recent version of this flow file within this session");
 }
 
+
 for (final List flowFiles : transferMap.values()) {
 if (flowFiles.contains(flowFile)) {
 throw new IllegalStateException(flowFile + " has already 
been transferred");
 }
 }
 }
 
+private void validateDestinationRelation(final Relationship 
relationship) {
+if (!relationships.contains(relationship)) {
+rollback();
--- End diff --

So, if a relationship does not exist what should we do?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

putTCP nifi 0.7?

2016-06-16 Thread Ryan Ward

Will this one make 0.7?

https://issues.apache.org/jira/browse/NIFI-1834

[GitHub] nifi pull request #252: NIFI-615 - Create a processor to extract WAV file ch...

Github user jskora commented on a diff in the pull request:

https://github.com/apache/nifi/pull/252#discussion_r67393537
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.;)
+@WritesAttributes({@WritesAttribute(attribute = "tika.", 
description = "The extracted content metadata "
++ "will be inserted with the attribute name \"tika.\". 
")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+public static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new 
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to 
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will 
process all of them.")
+.required(false)
+
.addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor FILENAME_FILTER = new 
PropertyDescriptor.Builder()
+.name("File Name Filter")
+.description("A regular expression identifying file names 
which metadata should extracted.  As flowfiles"
++

[GitHub] nifi pull request #534: Fix for NIFI-1838 & NIFI-1152

Github user PuspenduBanerjee commented on a diff in the pull request:

https://github.com/apache/nifi/pull/534#discussion_r67393509
  
--- Diff: 
nifi-mock/src/main/java/org/apache/nifi/util/MockProcessSession.java ---
@@ -587,13 +589,15 @@ public void transfer(final FlowFile flowFile, final 
Relationship relationship) {
 }
 
 validateState(flowFile);
+validateDestinationRelation(relationship);
 List list = transferMap.get(relationship);
 if (list == null) {
 list = new ArrayList<>();
 transferMap.put(relationship, list);
 }
 
 beingProcessed.remove(flowFile.getId());
+   // if(sharedState.)
--- End diff --

done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #256: NIFI-1578: Create PutSlack processor

2016-06-16 Thread mattyb149

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/256#discussion_r67392432
  
--- Diff: 
nifi-nar-bundles/nifi-slack-bundle/nifi-slack-processors/src/main/java/org/apache/nifi/processors/slack/PutSlack.java
 ---
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.slack;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.DataOutputStream;
+
+import javax.json.Json;
+import javax.json.JsonObject;
+import javax.json.JsonObjectBuilder;
+import javax.json.JsonWriter;
+import java.io.IOException;
+import java.io.StringWriter;
+import java.net.HttpURLConnection;
+import java.net.URL;
+import java.net.URLEncoder;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"put", "slack", "notify"})
+@CapabilityDescription("Sends a message to your team on slack.com")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class PutSlack extends AbstractProcessor {
+
+public static final PropertyDescriptor WEBHOOK_URL = new 
PropertyDescriptor
+.Builder()
+.name("webhook-url")
+.displayName("Webhook URL")
+.description("The POST URL provided by Slack to send messages 
into a channel.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.sensitive(true)
+.build();
+
+public static final PropertyDescriptor WEBHOOK_TEXT = new 
PropertyDescriptor
+.Builder()
+.name("webhook-text")
+.displayName("Webhook Text")
+.description("The text sent in the webhook message")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor CHANNEL = new PropertyDescriptor
+.Builder()
+.name("channel")
+.displayName("Channel")
+.description("A public channel using #channel or direct 
message using @username. If not specified, " +
+"the default webhook channel as specified in Slack's 
Incoming Webhooks web interface is used.")
+.required(false)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor
+.Builder()
+.name("username")
+.displayName("Username")
+.description("The displayed Slack username")
+.required(false)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor ICON_URL = new 
PropertyDescriptor
+.Builder()
+.name("icon-url")
+.displayName("Icon URL")
+

[GitHub] nifi pull request #534: Fix for NIFI-1838 & NIFI-1152

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/534#discussion_r67392290
  
--- Diff: 
nifi-mock/src/test/java/org/apache/nifi/util/TestMockProcessSession.java ---
@@ -55,4 +55,35 @@ public void onTrigger(final ProcessContext ctx, final 
ProcessSession session) th
 }
 
 }
+
+
+@Test(expected = AssertionError.class)
+public void testNonExistentRelationFromProcessor() {
+
TestRunners.newTestRunner(NonExistentRelationTestProcessor.class).run();
+}
+
+protected static class NonExistentRelationTestProcessor extends 
PoorlyBehavedProcessor {
+
+static final Relationship REL_NON_EXISTENT = new 
Relationship.Builder()
+.name("NonExistent Relation")
+.build();
+
+public Set relationships = 
Collections.singleton(REL_FAILURE);
+
+@Override
+public Set getRelationships() {
+return relationships;
--- End diff --

I think you just have to return an empty set of relationships instead of 
referencing the one of the other class. No?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: GetJMSQueue does not detect dead connections

Chris 

Given that we are deprecating Get/PutJMS* in favor of Publish/SubscribeJMS, I’d 
suggest start using those once.

Cheers
Oleg


> On Jun 16, 2016, at 1:34 PM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote)  wrote:
> 
> Folks,
> 
> I’ve been trying to test my GetJMSQueue configuration so that it detects a 
> dead broker connection and fails over to an alternate broker.  When I say 
> dead connection I mean TCP connection that has not been closed but is no 
> longer passing traffic.  In the real world this typically happens when broker 
> server crashes and so it does not reset the open connections.  For my test 
> case I am using iptables to block traffic.
> 
> This is the connection URI I am using
> 
> failover:(tcp://host2:61616,tcp://host1:61616)?randomize=false=3000=3=3=1=0
> 
> They key parameters here are soTimeout=3 and soWriteTimeout=3
> 
> These set a 30 second timeout on socket reads and writes.  I’m not sure if 
> these are necessary since I believe the JMSConsumer classes specifies its own 
> timeout according to the processor configuration.  The important thing to 
> note is that when one of these timeouts occurs the AMQ client does not close 
> the connection.
> 
> I believe the deficiency here is that JMSConsumer does not consider the 
> possibility that the connection is dead.   The problem with this is that an 
> attempt to reconnect and failover to an alternate broker is not made.
> 
> I think the fix would involve counting the number of sequential empty 
> responses on the connection and then closing the connection once that number 
> crosses some threshold.  Then subsequent onTrigger() would cause a new 
> connection attempt.
> 
> Thoughts?
> 
> Chris McDermott
> 
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
> 
> https://www.storefrontremote.com

[GitHub] nifi pull request #534: Fix for NIFI-1838 & NIFI-1152

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/534#discussion_r67391289
  
--- Diff: 
nifi-mock/src/test/java/org/apache/nifi/util/TestMockProcessSession.java ---
@@ -36,7 +36,7 @@ public void testPenalizeFlowFileFromProcessor() {
 
 protected static class PoorlyBehavedProcessor extends 
AbstractProcessor {
 
-private static final Relationship REL_FAILURE = new 
Relationship.Builder()
+protected static final Relationship REL_FAILURE = new 
Relationship.Builder()
--- End diff --

scope change is not needed, right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #534: Fix for NIFI-1838 & NIFI-1152

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/534#discussion_r67390898
  
--- Diff: 
nifi-mock/src/test/java/org/apache/nifi/util/TestMockProcessSession.java ---
@@ -16,6 +16,9 @@
  */
 package org.apache.nifi.util;
 
+import java.util.Collections;
+import java.util.Set;
+
--- End diff --

I believe your IDE is automatically changing the import order when you save 
your modifications. Could you revert the changes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #534: Fix for NIFI-1838 & NIFI-1152

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/534#discussion_r67390513
  
--- Diff: 
nifi-mock/src/main/java/org/apache/nifi/util/MockProcessSession.java ---
@@ -587,13 +589,15 @@ public void transfer(final FlowFile flowFile, final 
Relationship relationship) {
 }
 
 validateState(flowFile);
+validateDestinationRelation(relationship);
 List list = transferMap.get(relationship);
 if (list == null) {
 list = new ArrayList<>();
 transferMap.put(relationship, list);
 }
 
 beingProcessed.remove(flowFile.getId());
+   // if(sharedState.)
--- End diff --

Could you remove this comment?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #272: NIFI-1620 Allow empty Content-Type in InvokeHTTP processor

2016-06-16 Thread McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/272
  
It should be OK, let me know if not.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

GetJMSQueue does not detect dead connections

Folks,

I’ve been trying to test my GetJMSQueue configuration so that it detects a dead 
broker connection and fails over to an alternate broker.  When I say dead 
connection I mean TCP connection that has not been closed but is no longer 
passing traffic.  In the real world this typically happens when broker server 
crashes and so it does not reset the open connections.  For my test case I am 
using iptables to block traffic.

This is the connection URI I am using

failover:(tcp://host2:61616,tcp://host1:61616)?randomize=false=3000=3=3=1=0

They key parameters here are soTimeout=3 and soWriteTimeout=3

These set a 30 second timeout on socket reads and writes.  I’m not sure if 
these are necessary since I believe the JMSConsumer classes specifies its own 
timeout according to the processor configuration.  The important thing to note 
is that when one of these timeouts occurs the AMQ client does not close the 
connection.

I believe the deficiency here is that JMSConsumer does not consider the 
possibility that the connection is dead.   The problem with this is that an 
attempt to reconnect and failover to an alternate broker is not made.

I think the fix would involve counting the number of sequential empty responses 
on the connection and then closing the connection once that number crosses some 
threshold.  Then subsequent onTrigger() would cause a new connection attempt.

Thoughts?

Chris McDermott

Remote Business Analytics
STaTS/StoreFront Remote
HPE Storage
Hewlett Packard Enterprise
Mobile: +1 978-697-5315

https://www.storefrontremote.com

[GitHub] nifi pull request #252: NIFI-615 - Create a processor to extract WAV file ch...

Github user jskora commented on a diff in the pull request:

https://github.com/apache/nifi/pull/252#discussion_r67387942
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.;)
+@WritesAttributes({@WritesAttribute(attribute = "tika.", 
description = "The extracted content metadata "
++ "will be inserted with the attribute name \"tika.\". 
")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+public static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new 
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to 
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will 
process all of them.")
+.required(false)
+
.addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor FILENAME_FILTER = new 
PropertyDescriptor.Builder()
+.name("File Name Filter")
+.description("A regular expression identifying file names 
which metadata should extracted.  As flowfiles"
++

[GitHub] nifi pull request #252: NIFI-615 - Create a processor to extract WAV file ch...

Github user jskora commented on a diff in the pull request:

https://github.com/apache/nifi/pull/252#discussion_r67387964
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/test/java/org/apache/nifi/processors/media/TestExtractMediaMetadata.java
 ---
@@ -0,0 +1,411 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class TestExtractMediaMetadata {
+
+@Test
+public void testProperties() {
+final TestRunner runner = TestRunners.newTestRunner(new 
ExtractMediaMetadata());
+ProcessContext context = runner.getProcessContext();
+Map propertyValues = 
context.getProperties();
+assertEquals(5, propertyValues.size());
+}
+
+@Test
+public void testRelationShips() {
--- End diff --

Got it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #534: Fix for NIFI-1838 & NIFI-1152

GitHub user PuspenduBanerjee opened a pull request:

https://github.com/apache/nifi/pull/534

Fix for NIFI-1838 & NIFI-1152

Fix for NIFI-1838 & NIFI-1152


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/PuspenduBanerjee/nifi NIFI-1152

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/534.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #534


commit 1ee24388bb54a82d008d4ab7e32dd39c2bb0ff4d
Author: Puspendu Banerjee 
Date:   2016-05-03T01:36:50Z

Fix for NIFI-1838 & NIFI-1152




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #252: NIFI-615 - Create a processor to extract WAV file ch...

Github user jskora commented on a diff in the pull request:

https://github.com/apache/nifi/pull/252#discussion_r67385461
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.;)
+@WritesAttributes({@WritesAttribute(attribute = "tika.", 
description = "The extracted content metadata "
++ "will be inserted with the attribute name \"tika.\". 
")})
--- End diff --

Corrected reference to "" and noted that if not 
provided it's just the attribute name.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #252: NIFI-615 - Create a processor to extract WAV file ch...

Github user jskora commented on a diff in the pull request:

https://github.com/apache/nifi/pull/252#discussion_r67384364
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.;)
+@WritesAttributes({@WritesAttribute(attribute = "tika.", 
description = "The extracted content metadata "
++ "will be inserted with the attribute name \"tika.\". 
")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+public static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new 
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to 
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will 
process all of them.")
+.required(false)
+
.addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor FILENAME_FILTER = new 
PropertyDescriptor.Builder()
+.name("File Name Filter")
+.description("A regular expression identifying file names 
which metadata should extracted.  As flowfiles"
++

[GitHub] nifi pull request #252: NIFI-615 - Create a processor to extract WAV file ch...

Github user jskora commented on a diff in the pull request:

https://github.com/apache/nifi/pull/252#discussion_r67383325
  
--- Diff: 
nifi-nar-bundles/nifi-media-bundle/nifi-media-processors/src/main/java/org/apache/nifi/processors/media/ExtractMediaMetadata.java
 ---
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.media;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.regex.Pattern;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ProcessorLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.ObjectHolder;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.sax.BodyContentHandler;
+import org.xml.sax.SAXException;
+
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"media", "file", "format", "metadata", "audio", "video", "image", 
"document", "pdf"})
+@CapabilityDescription("Extract the content metadata from flowfiles 
containing audio, video, image, and other file "
++ "types.  This processor relies on the Apache Tika project for 
file format detection and parsing.  It "
++ "extracts a long list of metadata types for media files 
including audio, video, and print media "
++ "formats."
++ "For the more details and the list of supported file types, 
visit the library's website "
++ "at http://tika.apache.org/.;)
+@WritesAttributes({@WritesAttribute(attribute = "tika.", 
description = "The extracted content metadata "
++ "will be inserted with the attribute name \"tika.\". 
")})
+@SupportsBatching
+public class ExtractMediaMetadata extends AbstractProcessor {
+
+public static final PropertyDescriptor MAX_NUMBER_OF_ATTRIBUTES = new 
PropertyDescriptor.Builder()
+.name("Max Number of Attributes")
+.description("Specify the max number of attributes to add to 
the flowfile. There is no guarantee in what order"
++ " the tags will be processed. By default it will 
process all of them.")
+.required(false)
+
.addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor FILENAME_FILTER = new 
PropertyDescriptor.Builder()
+.name("File Name Filter")
+.description("A regular expression identifying file names 
which metadata should extracted.  As flowfiles"
++

[GitHub] nifi issue #458: NIFI-1829 - Create new DebugFlow processor.

2016-06-16 Thread mosermw

Github user mosermw commented on the issue:

https://github.com/apache/nifi/pull/458
  
Completed review, latest changes look good, unit testing and integration 
testing worked as expected, code passes -Pcontrib-check.

+1 I will squash commits and merge to 0.x and master shortly.  Thanks 
@jskora.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

RE: Attribute for current Processor/Connection

2016-06-16 Thread Saxena, Shishir

Thanks Joe for quick response. I agree it can be implemented with either of the 
approach you suggested. But both approaches are more cumbersome to implement.

Do you think it will be worth to explore adding such an attribute ? I think it 
will be of value to other developers also during development of flow. I am new 
to JAVA development but I can help in contributing to this development.

On a separate note - Is there a way to access provenance trail for this 
information dynamically ?

Thanks.
Shishir

-Original Message-
From: Joe Witt [mailto:joe.w...@gmail.com] 
Sent: Thursday, June 16, 2016 11:35 AM
To: dev@nifi.apache.org
Subject: Re: Attribute for current Processor/Connection

Hello

We do not automatically tag the flow file attributes with sourcing information. 
 That is captured in the provenance trail though.

For your case you might consider adding an update attribute processor after 
each transition point where you want to capture something of interest so you 
can tag the attributes.  Then your single group for error handling will have 
the needed context.  Alternatively, you can put input ports on your error 
handling process group which are inherently about the context of data they 
intend to receive.

Thanks
JOe

On Thu, Jun 16, 2016 at 11:32 AM, Saxena, Shishir  
wrote:
> Hi,
>   Is there any attribute available on every flow that will give details on 
> what was the last processor that processed the flow ? I have a flow where I 
> am routing all failures to a single process group for error handling. It will 
> be convenient to have an attribute for current processor, so any error 
> messages can be generated based on which processor failure occurred.
>
> Thanks for any help.
>
> Shishir
>
>
>
>
>
> This email transmission and any accompanying attachments may contain CSX 
> privileged and confidential information intended only for the use of the 
> intended addressee. Any dissemination, distribution, copying or action taken 
> in reliance on the contents of this email by anyone other than the intended 
> recipient is strictly prohibited. If you have received this email in error 
> please immediately delete it and notify sender at the above CSX email 
> address. Sender and CSX accept no liability for any damage caused directly or 
> indirectly by receipt of this email.

Re: Attribute for current Processor/Connection

2016-06-16 Thread Joe Witt

Hello

We do not automatically tag the flow file attributes with sourcing
information.  That is captured in the provenance trail though.

For your case you might consider adding an update attribute processor
after each transition point where you want to capture something of
interest so you can tag the attributes.  Then your single group for
error handling will have the needed context.  Alternatively, you can
put input ports on your error handling process group which are
inherently about the context of data they intend to receive.

Thanks
JOe

On Thu, Jun 16, 2016 at 11:32 AM, Saxena, Shishir
 wrote:
> Hi,
>   Is there any attribute available on every flow that will give details on 
> what was the last processor that processed the flow ? I have a flow where I 
> am routing all failures to a single process group for error handling. It will 
> be convenient to have an attribute for current processor, so any error 
> messages can be generated based on which processor failure occurred.
>
> Thanks for any help.
>
> Shishir
>
>
>
>
>
> This email transmission and any accompanying attachments may contain CSX 
> privileged and confidential information intended only for the use of the 
> intended addressee. Any dissemination, distribution, copying or action taken 
> in reliance on the contents of this email by anyone other than the intended 
> recipient is strictly prohibited. If you have received this email in error 
> please immediately delete it and notify sender at the above CSX email 
> address. Sender and CSX accept no liability for any damage caused directly or 
> indirectly by receipt of this email.

Attribute for current Processor/Connection

2016-06-16 Thread Saxena, Shishir

Hi,
  Is there any attribute available on every flow that will give details on what 
was the last processor that processed the flow ? I have a flow where I am 
routing all failures to a single process group for error handling. It will be 
convenient to have an attribute for current processor, so any error messages 
can be generated based on which processor failure occurred.

Thanks for any help.

Shishir





This email transmission and any accompanying attachments may contain CSX 
privileged and confidential information intended only for the use of the 
intended addressee. Any dissemination, distribution, copying or action taken in 
reliance on the contents of this email by anyone other than the intended 
recipient is strictly prohibited. If you have received this email in error 
please immediately delete it and notify sender at the above CSX email address. 
Sender and CSX accept no liability for any damage caused directly or indirectly 
by receipt of this email.

Re: About Unit Tesing

2016-06-16 Thread Andy LoPresto

Rahul,

Here is a section of the developer guide [1] which discusses the testing 
framework available to developers.

[1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#testing 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jun 16, 2016, at 12:26 AM, rahul pagare  wrote:
> 
> Hello sir,
> 
> sir i am working as a tester in one company i want to know about the unit
> testing on niifi can you send me some document about testing .
> 
> Thank you



signature.asc
Description: Message signed with OpenPGP using GPGMail

[GitHub] nifi issue #458: NIFI-1829 - Create new DebugFlow processor.