date:20160609

RE: Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl

2016-06-09 Thread Kumiko Yada

I found out that I didn’t have specify this dependency in pom file that it 
would be packaged as part of below by finding the jar file in 
…\nifi-azure-dlstore-nar\target\classes\META-INF\bundled-dependencies.  I 
verified that it is packaged with a new Nar, and I even copied this jar file in 
…\nifi-0.7.0-SNAPSHOT\lib folder, but I’m still getting the exception: 
java.lang.NoClassDefFoundError: retrofit2/BaseUrl.



adx-snapshots
Azure ADX Snapshots
http://adxsnapshots.azurewebsites.net/
default

true



oss-snapshots
Open Source Snapshots
https://oss.sonatype.org/content/repositories/snapshots/
default

true
always





com.microsoft.azure
azure-client-authentication
1.0.0-SNAPSHOT

   
com.microsoft.azure
azure-mgmt-datalake-store
1.0.0-SNAPSHOT




From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com]
Sent: Thursday, June 9, 2016 5:11 PM
To: users@nifi.apache.org
Subject: Re: Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl

Yes, it has to be available to your code otherwise you will continue to get 
ClassNotFoundException

Oleg
On Jun 9, 2016, at 8:06 PM, Kumiko Yada 
> wrote:

Does this mean that I have to include this dependency in nar pom file now?



com.squareup.retrofit2

retrofit
2.0.4

Thanks
Kumiko

From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com]
Sent: Thursday, June 9, 2016 4:27 PM
To: users@nifi.apache.org
Subject: Re: Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl

No, it just means that retrofit2.BaseUri is not in the NAR class path (see 
class not found at the bottom of the trace). Is it packaged with your new Nar?

Sent from my iPhone

On Jun 9, 2016, at 19:01, Kumiko Yada 
> wrote:
Hello,

My custom processor is throwing the run-time exception, so I reverted the 
changes that I made and I’m still getting this below error.   Does this mean 
that there is some changes in Microsoft Azure SDK?  My custom processor nar 
file which I compiled yesterday is working fine.  Note:  I’m getting this 
exception in single thread.

2016-06-09 15:50:46,718 WARN [Timer-Driven Process Thread-5] 
o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding 
PutFileAzureDLStore[id=73299986-7f85-4114-8940-b77798757d3a] due to uncaught 
Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl
2016-06-09 15:50:46,719 WARN [Timer-Driven Process Thread-5] 
o.a.n.c.t.ContinuallyRunProcessorTask
java.lang.NoClassDefFoundError: retrofit2/BaseUrl
at 
nifi.azure.dlstore.processors.PutFileAzureDLStore.SetupClients(PutFileAzureDLStore.java:257)
 ~[na:na]
at 
nifi.azure.dlstore.processors.PutFileAzureDLStore.onTrigger(PutFileAzureDLStore.java:210)
 ~[na:na]
at 
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
 ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057)
 ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_77]
at 
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [na:1.8.0_77]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [na:1.8.0_77]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_77]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
Caused by: java.lang.ClassNotFoundException: retrofit2.BaseUrl
at

Re: Funnel

2016-06-09 Thread Igor Kravzov

Ok. Got it. Thanks guys.

On Thu, Jun 9, 2016 at 7:37 PM James Wing  wrote:

> The funnel merges multiple queues into a single queue.  You then have one
> queue to configure prioritization, backpressure, etc.  And one queue to
> monitor and administer.  It can also be helpful visually if you really have
> a lot of input queues, as a self-describing way of asserting they go to one
> conceptual output.
>
> Using the funnel is certainly not required, you can have more than one
> input queue into a processor.  I've found it useful to manage backpressure
> and make sure I don't pile up too many flow files and fill up the disk.
>
> Thanks,
>
> James
>
> On Thu, Jun 9, 2016 at 2:35 PM, Igor Kravzov 
> wrote:
>
>> Guys, what is the actual role of funnel?
>>
>
>

RE: Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl

2016-06-09 Thread Kumiko Yada

Does this mean that I have to include this dependency in nar pom file now?



com.squareup.retrofit2

retrofit
2.0.4

Thanks
Kumiko

From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com]
Sent: Thursday, June 9, 2016 4:27 PM
To: users@nifi.apache.org
Subject: Re: Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl

No, it just means that retrofit2.BaseUri is not in the NAR class path (see 
class not found at the bottom of the trace). Is it packaged with your new Nar?

Sent from my iPhone

On Jun 9, 2016, at 19:01, Kumiko Yada 
> wrote:
Hello,

My custom processor is throwing the run-time exception, so I reverted the 
changes that I made and I'm still getting this below error.   Does this mean 
that there is some changes in Microsoft Azure SDK?  My custom processor nar 
file which I compiled yesterday is working fine.  Note:  I'm getting this 
exception in single thread.

2016-06-09 15:50:46,718 WARN [Timer-Driven Process Thread-5] 
o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding 
PutFileAzureDLStore[id=73299986-7f85-4114-8940-b77798757d3a] due to uncaught 
Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl
2016-06-09 15:50:46,719 WARN [Timer-Driven Process Thread-5] 
o.a.n.c.t.ContinuallyRunProcessorTask
java.lang.NoClassDefFoundError: retrofit2/BaseUrl
at 
nifi.azure.dlstore.processors.PutFileAzureDLStore.SetupClients(PutFileAzureDLStore.java:257)
 ~[na:na]
at 
nifi.azure.dlstore.processors.PutFileAzureDLStore.onTrigger(PutFileAzureDLStore.java:210)
 ~[na:na]
at 
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
 ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057)
 ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_77]
at 
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [na:1.8.0_77]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [na:1.8.0_77]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_77]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
Caused by: java.lang.ClassNotFoundException: retrofit2.BaseUrl
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
~[na:1.8.0_77]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
~[na:1.8.0_77]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 
~[na:1.8.0_77]

Thanks
Kumiko

Re: Failure when running a workflow created from a template from another NiFi version.

2016-06-09 Thread James Wing

Thanks for reporting back, Keith. It sounds like a bug got fixed, and
that's always great news even if we don't exactly where :).

James

On Thu, Jun 9, 2016 at 4:10 PM, Keith Lim  wrote:

> Thanks James,  I built from the latest 0.x branch and the issue does not
> show up.
>
>
> Thanks,
> Keith
>
>
> --
> *From:* James Wing 
> *Sent:* Wednesday, June 08, 2016 9:23:22 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: Failure when running a workflow created from a template
> from another NiFi version.
>
> Do you know how recently your 0.7.0-SNAPSHOT was built from source?  Have
> you tried the latest code in the 0.x branch?
>
> Thanks,
>
> James
>
> On Wed, Jun 8, 2016 at 5:43 PM, Keith Lim  wrote:
>
>> My repro was when I start the workflow, the error message shows up and
>> was on nifi-0.7.0-SNAPSHOT with java 1.8.0_91.
>>
>>
>> Thanks,
>> Keith
>> --
>> *From:* James Wing 
>> *Sent:* Wednesday, June 08, 2016 4:59 PM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: Failure when running a workflow created from a template
>> from another NiFi version.
>>
>> I was able to import the template without errors on Windows 10, running
>> the latest 0.x branch (c4ddb52) and Java 1.8.0_66.  Can you expand on your
>> NiFi version, Java etc?
>>
>> Thanks,
>>
>> James
>>
>> On Wed, Jun 8, 2016 at 4:18 PM, Keith Lim  wrote:
>>
>>> Hi James,
>>>
>>>
>>> Here is the template.  Need to remove the .txt extension.
>>>
>>> Note: I didn't bother to put in the correct flowfile xpath content for
>>> the processor to evaluate to success result.
>>>
>>>
>>> Thanks,
>>> Keith
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> *From:* James Wing 
>>> *Sent:* Wednesday, June 08, 2016 3:59 PM
>>> *To:* users@nifi.apache.org
>>> *Subject:* Re: Failure when running a workflow created from a template
>>> from another NiFi version.
>>>
>>> Keith,
>>>
>>> Would you be able to share your sample template with us as an
>>> attachment, a GitHub Gist, or something similar?
>>>
>>> Thanks,
>>>
>>> James
>>>
>>> On Wed, Jun 8, 2016 at 3:36 PM, Keith Lim  wrote:
>>>
 Hi Joe,

 I created a simple workflow with EvaluateXPath and managed to repro the
 issue.
 Here is the snapshot of the screen.  This is on Windows with 0.7.0
 snapshot.There is no issue on 0.6.1 on Windows and 0.7.0 on Linux.

 We can't roll back to 0.6.1 as we have a dependency on a feature in
 0.7.0

 Thanks,
 Keith

 From: Keith Lim 
 Sent: Wednesday, June 08, 2016 10:28 AM
 To: users@nifi.apache.org
 Subject: Re: Failure when running a workflow created from a template
 from another NiFi version.

 Hi Joe,

 Interesting... I didn't find anything in the nifi-app.log, however, I
 found these logged in the nifi.user.log.

 2016-06-08 10:18:22,089 INFO [NiFi Web Server-20]
 org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous)
 PUT
 http://nifi.dev.azure.ds-iq.corp:1/nifi-api/controller/process-groups/658947de-a030-3832-95a6-1f4d2b346131
 (source ip: 10.200.0.31)
 2016-06-08 10:18:22,089 INFO [NiFi Web Server-20]
 o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
 EvaluateXPath[id=6c73cffa-0d34-3b25-94d4-14d115b8b62b] is not in a valid
 state. Returning Conflict response.
 2016-06-08 10:18:24,636 INFO [NiFi Web Server-18]
 org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous)
 GET http://nifi.dev.azure.ds-iq.corp:1/nifi-api/controller (source
 ip: 10.200.0.31)
 2016-06-08 10:18:25,324 INFO [NiFi Web Server-22]
 org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous)
 GET http://nifi.dev.azure.ds-iq.corp:1/nifi-api/controller (source
 ip: 10.200.0.31)
 2016-06-08 10:18:29,511 INFO [NiFi Web Server-24]
 org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous)
 PUT
 http://nifi.dev.azure.ds-iq.corp:1/nifi-api/controller/process-groups/658947de-a030-3832-95a6-1f4d2b346131
 (source ip: 10.200.0.31)
 2016-06-08 10:18:29,542 INFO [NiFi Web Server-24]
 o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
 EvaluateXPath[id=6c73cffa-0d34-3b25-94d4-14d115b8b62b] is not in a valid
 state. Returning Conflict response.

 This is the message box I got when I tried to run my workflow:
 Node azudevmgmt01.ds-iq.corp:10001 is unable to fulfill this request
 due to: EvaluateXPath[id=6c73cffa-0d34-3b25-94d4-14d115b8b62b] is not in a
 valid state

 Thanks,
 Keith

 From: Joe Witt

Re: Failure when running a workflow created from a template from another NiFi version.

2016-06-09 Thread Keith Lim

Thanks James,  I built from the latest 0.x branch and the issue does not show 
up.

Thanks,
Keith

From: James Wing 
Sent: Wednesday, June 08, 2016 9:23:22 PM
To: users@nifi.apache.org
Subject: Re: Failure when running a workflow created from a template from 
another NiFi version.

Do you know how recently your 0.7.0-SNAPSHOT was built from source?  Have you 
tried the latest code in the 0.x branch?

Thanks,

James

On Wed, Jun 8, 2016 at 5:43 PM, Keith Lim 
> wrote:

My repro was when I start the workflow, the error message shows up and was on 
nifi-0.7.0-SNAPSHOT with java 1.8.0_91.

Thanks,
Keith

From: James Wing >
Sent: Wednesday, June 08, 2016 4:59 PM
To: users@nifi.apache.org
Subject: Re: Failure when running a workflow created from a template from 
another NiFi version.

I was able to import the template without errors on Windows 10, running the 
latest 0.x branch (c4ddb52) and Java 1.8.0_66.  Can you expand on your NiFi 
version, Java etc?

Thanks,

James

On Wed, Jun 8, 2016 at 4:18 PM, Keith Lim 
> wrote:

Hi James,

Here is the template.  Need to remove the .txt extension.

Note: I didn't bother to put in the correct flowfile xpath content for the 
processor to evaluate to success result.

Thanks,
Keith

From: James Wing >
Sent: Wednesday, June 08, 2016 3:59 PM
To: users@nifi.apache.org
Subject: Re: Failure when running a workflow created from a template from 
another NiFi version.

Keith,

Would you be able to share your sample template with us as an attachment, a 
GitHub Gist, or something similar?

Thanks,

James

On Wed, Jun 8, 2016 at 3:36 PM, Keith Lim 
> wrote:
Hi Joe,

I created a simple workflow with EvaluateXPath and managed to repro the issue.
Here is the snapshot of the screen.  This is on Windows with 0.7.0 snapshot.
There is no issue on 0.6.1 on Windows and 0.7.0 on Linux.

We can't roll back to 0.6.1 as we have a dependency on a feature in 0.7.0

Thanks,
Keith

[cid:36ee5712-5a4f-45cb-847e-3a9234eb4887]

From: Keith Lim >
Sent: Wednesday, June 08, 2016 10:28 AM
To: users@nifi.apache.org
Subject: Re: Failure when running a workflow created from a template from 
another NiFi version.

Hi Joe,

Interesting... I didn't find anything in the nifi-app.log, however, I found 
these logged in the nifi.user.log.

2016-06-08 10:18:22,089 INFO [NiFi Web Server-20] 
org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) PUT 
http://nifi.dev.azure.ds-iq.corp:1/nifi-api/controller/process-groups/658947de-a030-3832-95a6-1f4d2b346131
 (source ip: 10.200.0.31)
2016-06-08 10:18:22,089 INFO [NiFi Web Server-20] 
o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: 
EvaluateXPath[id=6c73cffa-0d34-3b25-94d4-14d115b8b62b] is not in a valid state. 
Returning Conflict response.
2016-06-08 10:18:24,636 INFO [NiFi Web Server-18] 
org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) GET 
http://nifi.dev.azure.ds-iq.corp:1/nifi-api/controller (source ip: 
10.200.0.31)
2016-06-08 10:18:25,324 INFO [NiFi Web Server-22] 
org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) GET 
http://nifi.dev.azure.ds-iq.corp:1/nifi-api/controller (source ip: 
10.200.0.31)
2016-06-08 10:18:29,511 INFO [NiFi Web Server-24] 
org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) PUT 
http://nifi.dev.azure.ds-iq.corp:1/nifi-api/controller/process-groups/658947de-a030-3832-95a6-1f4d2b346131
 (source ip: 10.200.0.31)
2016-06-08 10:18:29,542 INFO [NiFi Web Server-24] 
o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: 
EvaluateXPath[id=6c73cffa-0d34-3b25-94d4-14d115b8b62b] is not in a valid state. 
Returning Conflict response.

This is the message box I got when I tried to run my workflow:
Node azudevmgmt01.ds-iq.corp:10001 is unable to fulfill this request due to: 
EvaluateXPath[id=6c73cffa-0d34-3b25-94d4-14d115b8b62b] is not in a valid state

Thanks,
Keith

From: Joe Witt >
Sent: Wednesday, June 08, 2016 10:02 AM
To: users@nifi.apache.org
Subject: Re: Failure when running a workflow created from a template from 
another NiFi version.

Keith,

Can you collect more information on this from the nifi-app.log?  Not
in valid state should be associated with some exceptions and such that
ideally will be in the logs.

Thanks
Joe

On Wed, Jun 8, 2016 at

Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl

2016-06-09 Thread Kumiko Yada

Hello,

My custom processor is throwing the run-time exception, so I reverted the 
changes that I made and I'm still getting this below error.   Does this mean 
that there is some changes in Microsoft Azure SDK?  My custom processor nar 
file which I compiled yesterday is working fine.  Note:  I'm getting this 
exception in single thread.

2016-06-09 15:50:46,718 WARN [Timer-Driven Process Thread-5] 
o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding 
PutFileAzureDLStore[id=73299986-7f85-4114-8940-b77798757d3a] due to uncaught 
Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl
2016-06-09 15:50:46,719 WARN [Timer-Driven Process Thread-5] 
o.a.n.c.t.ContinuallyRunProcessorTask
java.lang.NoClassDefFoundError: retrofit2/BaseUrl
at 
nifi.azure.dlstore.processors.PutFileAzureDLStore.SetupClients(PutFileAzureDLStore.java:257)
 ~[na:na]
at 
nifi.azure.dlstore.processors.PutFileAzureDLStore.onTrigger(PutFileAzureDLStore.java:210)
 ~[na:na]
at 
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
 ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057)
 ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_77]
at 
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [na:1.8.0_77]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [na:1.8.0_77]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_77]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
Caused by: java.lang.ClassNotFoundException: retrofit2.BaseUrl
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
~[na:1.8.0_77]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
~[na:1.8.0_77]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 
~[na:1.8.0_77]

Thanks
Kumiko

Funnel

2016-06-09 Thread Igor Kravzov

Guys, what is the actual role of funnel?

RE: Custom processor is failing for concurrency

2016-06-09 Thread Kumiko Yada

Here is the code:  
https://github.com/kyada1/PutFileAzureDLStore/blob/master/nifi-azure-dlstore-bundle/nifi-azure-dlstore-processors/src/main/java/nifi/azure/dlstore/processors/PutFileAzureDLStore.java

The problems are :
Line 209 - SetupClients(creds);
Line 217 - CreateFile(_path + _filename, value.get(), true);

To wrap the call in synchronized method, are these looked correct?  Or for the 
line 217, does the method have to return something?

For the line 209:
synchronized (this) {
SetupClients(creds);
if (creds!= null) {
return creds;
}

For the line 217:
synchronized {
CreateFile(_path + _filename, value.get(), true);
}

Thanks
Kumiko

From: Matt Foley [mailto:mfo...@hortonworks.com]
Sent: Thursday, June 9, 2016 12:27 PM
To: users@nifi.apache.org
Cc: Kevin Verhoeven ; Ki Kang 
Subject: Re: Custom processor is failing for concurrency

If there are multiple SDK calls that share the same problematic code (that is, 
multiple SDK methods that would interact with each other in a non-thread-safe 
way), then one must synchronize their calls to a shared lock object, which only 
requires a couple more lines of code.

From: Oleg Zhurakousky 
>
Sent: Thursday, June 09, 2016 12:15 PM
To: users@nifi.apache.org
Cc: Kevin Verhoeven; Ki Kang
Subject: Re: Custom processor is failing for concurrency

+1, was just responding with the same.

On Jun 9, 2016, at 3:11 PM, Matt Foley 
> wrote:

Kumiko, would it be sufficient to just wrap your call to the non-thread-safe MS 
SDK routine, in a 'synchronized' method?  You could then use the standard NiFi 
thread management and avoid a lot of complexity.  And the result should be >= 
efficiency of having a dedicated thread to manage the problem action.
--Matt F

From: Kumiko Yada >
Sent: Thursday, June 09, 2016 11:49 AM
To: users@nifi.apache.org
Cc: Kevin Verhoeven; Ki Kang
Subject: Re: Custom processor is failing for concurrency

Hi Bryan,

Does this mean that even I create the multiple threads in onTriger, I will 
still hit the Microsoft SDK issue where it's not a thread safe?  Sounds like 
basically what I am trying to do and creating the  multiple threads via UI 
might be the same thing.

Thanks
Kumiko

From: Bryan Bende >
Sent: Thursday, June 9, 2016 11:26:10 AM
To: users@nifi.apache.org
Cc: Kevin Verhoeven; Ki Kang
Subject: Re: Custom processor is failing for concurrency

Kumiko,

In general you shouldn't have to create threads in your processors, with the 
exception of some special cases.
The framework has a thread pool and it takes one of those threads and calls the 
onTrigger method of your processor.

If you want multiple threads to call onTrigger, then each processor has a 
Concurrent Tasks property in the UI on the scheduling tab,
which equates to the number of threads that will concurrently call onTrigger.

A processor developer needs to only worry about the business logic in the 
onTrigger method, and needs to ensure
thread-safe access to any member variables or state stored in the processor.

Hope that helps.

-Bryan

On Thu, Jun 9, 2016 at 2:11 PM, Kumiko Yada 
> wrote:
Microsoft found this is an issue with the SDK, they are working on a fix, they 
do not have the ETA for the fix.  To workaround this issue, I’m trying to 
create the multiple threads in using AbstractSessionFactoryProcessor and handle 
the Create a file in a single thread.   I’m having a problem that the single 
thread is not working correctly.  The processor is still acting like a single 
thread.

When I create a thread to handle the create a file, do I have to call this 
method using java.util.concurrent.ExecutorService?

Are there any sample processors that I can take a look?

Thanks
Kumiko

From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com]
Sent: Sunday, June 5, 2016 6:28 PM
To: users@nifi.apache.org
Cc: Ki Kang >; Kevin Verhoeven 
>
Subject: RE: Custom processor is failing for concurrency

Thank you, Bryan.  I’m working with Microsoft on this issue.  Will keep you 
guys updated.

Thanks
Kumiko

From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Friday, June 3, 2016 2:32 PM
To: users@nifi.apache.org
Subject: Re: Custom processor is failing for concurrency

It is hard to say for sure, but I think your NiFi processor is generally ok

Re: Not a Valid JSON content

2016-06-09 Thread Pierre Villard

Hi Anuj,

This method supposes your XML input file is not too complex. Have a look on
this thread, this should answer your questions:
https://mail-archives.apache.org/mod_mbox/nifi-users/201606.mbox/%3CCALo_M1_erYWJkWB=D7kJ17gvpmnyeVGmYiQH9=mqqktb-nh...@mail.gmail.com%3E

Hope this helps.

2016-06-09 22:22 GMT+02:00 Anuj Handa :

> Hi ,
>
> I am getting the below error of not a valid JSON content
>
> a text file containing XML was sconverted to JSON.
>
> 2016-06-09 16:10:11,765 ERROR [Timer-Driven Process Thread-10]
> o.a.n.p.standard.EvaluateJsonPath
> EvaluateJsonPath[id=475d43ec-549b-4378-81d5-778041558781] FlowFile
> StandardFlowFileRecord[uuid=f238fd6d-ab1b-4d4b-b85d-77ade6102e18,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1464296313030-621,
> container=default, section=621], offset=116652,
> length=2039],offset=0,name=xml1.txt,size=2039] did not have valid JSON
> content.
> 2016-06-09 16:10:11,771 ERROR [Timer-Driven Process Thread-6]
> o.a.n.p.standard.EvaluateJsonPath
> EvaluateJsonPath[id=475d43ec-549b-4378-81d5-778041558781] FlowFile
> StandardFlowFileRecord[uuid=75ef7876-fb50-48f5-a651-c97212524e8a,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1464296313030-621,
> container=default, section=621], offset=118691,
> length=2214],offset=0,name=xml1.txt,size=2214] did not have valid JSON
> content.
> 2016-06-09 16:10:11,779 ERROR [Timer-Driven Process Thread-8]
> o.a.n.p.standard.EvaluateJsonPath
> EvaluateJsonPath[id=475d43ec-549b-4378-81d5-778041558781] FlowFile
> StandardFlowFileRecord[uuid=bc6855fb-8bc4-44ef-9196-41bab14c7aab,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1465502918815-4632,
> container=default, section=536], offset=531630,
> length=2226],offset=0,name=xml1.txt,size=2226] did not have valid JSON
> content.
>
> this is the properties of evaluate json content
>
> [image: Inline image 1]
>
> and ia m usig folloring XSLT to convert XML to JSON
>
>
> https://community.hortonworks.com/articles/29474/nifi-converting-xml-to-json.html
>
> ANuj
>

Re: Best way to compare dates

2016-06-09 Thread Igor Kravzov

Thanks James. That's exactly what I ended up with :)

On Thu, Jun 9, 2016 at 12:48 PM, James Wing <jvw...@gmail.com> wrote:

> Igor,
>
> One way would be to format both dates as strings (like "20160609" in your
> HDFS paths) first, then compare the two strings for equality.  In a
> RouteOnAttribute expression:
>
> ${now():format("MMdd"):equals(${entryDate:format("MMdd")})}
>
> If your goal is to merge the records into larger files, you might be able
> to do this with MergeContent, if you have the Correlation Attribute set to
> an attribute populated with the MMdd-formatteddate so it bins files by
> day.
>
> Thanks,
>
> James
>
>
> On Thu, Jun 9, 2016 at 7:37 AM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
>
>> Hi,
>>
>> In my workflow data is coming form Kafka Topic in JSON format and saved
>> ot HDFS directory by day.
>>
>> HDFS directory path constructed as /mypath/${now():format("MMdd")}
>> JSON file contains attribute {... "adddate":"2016-06-07 17:06:17" }
>>
>> I want to avoid over-spill of a previous date files into "today's"
>> directory.
>> The thing is if add date is the same as taday's date, a want file merged
>> to a bigger file and save to HDFS. Previous file not merged and saved as is.
>>
>> I am thinking to use EvaluateJsonPath to grab add date and to use
>> RouteOnAttribute to direct to appropriate  path.
>>
>> How comparison would look like?
>>
>> Thanks in advance.
>>
>
>

Not a Valid JSON content

2016-06-09 Thread Anuj Handa

Hi ,

I am getting the below error of not a valid JSON content

a text file containing XML was sconverted to JSON.

2016-06-09 16:10:11,765 ERROR [Timer-Driven Process Thread-10]
o.a.n.p.standard.EvaluateJsonPath
EvaluateJsonPath[id=475d43ec-549b-4378-81d5-778041558781] FlowFile
StandardFlowFileRecord[uuid=f238fd6d-ab1b-4d4b-b85d-77ade6102e18,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1464296313030-621,
container=default, section=621], offset=116652,
length=2039],offset=0,name=xml1.txt,size=2039] did not have valid JSON
content.
2016-06-09 16:10:11,771 ERROR [Timer-Driven Process Thread-6]
o.a.n.p.standard.EvaluateJsonPath
EvaluateJsonPath[id=475d43ec-549b-4378-81d5-778041558781] FlowFile
StandardFlowFileRecord[uuid=75ef7876-fb50-48f5-a651-c97212524e8a,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1464296313030-621,
container=default, section=621], offset=118691,
length=2214],offset=0,name=xml1.txt,size=2214] did not have valid JSON
content.
2016-06-09 16:10:11,779 ERROR [Timer-Driven Process Thread-8]
o.a.n.p.standard.EvaluateJsonPath
EvaluateJsonPath[id=475d43ec-549b-4378-81d5-778041558781] FlowFile
StandardFlowFileRecord[uuid=bc6855fb-8bc4-44ef-9196-41bab14c7aab,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1465502918815-4632,
container=default, section=536], offset=531630,
length=2226],offset=0,name=xml1.txt,size=2226] did not have valid JSON
content.

this is the properties of evaluate json content

[image: Inline image 1]

and ia m usig folloring XSLT to convert XML to JSON

https://community.hortonworks.com/articles/29474/nifi-converting-xml-to-json.html

ANuj

Re: Custom processor is failing for concurrency

2016-06-09 Thread Matt Foley

If there are multiple SDK calls that share the same problematic code (that is, 
multiple SDK methods that would interact with each other in a non-thread-safe 
way), then one must synchronize their calls to a shared lock object, which only 
requires a couple more lines of code.

From: Oleg Zhurakousky 
Sent: Thursday, June 09, 2016 12:15 PM
To: users@nifi.apache.org
Cc: Kevin Verhoeven; Ki Kang
Subject: Re: Custom processor is failing for concurrency

+1, was just responding with the same.

On Jun 9, 2016, at 3:11 PM, Matt Foley 
> wrote:

Kumiko, would it be sufficient to just wrap your call to the non-thread-safe MS 
SDK routine, in a 'synchronized' method?  You could then use the standard NiFi 
thread management and avoid a lot of complexity.  And the result should be >= 
efficiency of having a dedicated thread to manage the problem action.
--Matt F

From: Kumiko Yada >
Sent: Thursday, June 09, 2016 11:49 AM
To: users@nifi.apache.org
Cc: Kevin Verhoeven; Ki Kang
Subject: Re: Custom processor is failing for concurrency

Hi Bryan,

Does this mean that even I create the multiple threads in onTriger, I will 
still hit the Microsoft SDK issue where it's not a thread safe?  Sounds like 
basically what I am trying to do and creating the  multiple threads via UI 
might be the same thing.

Thanks
Kumiko

From: Bryan Bende >
Sent: Thursday, June 9, 2016 11:26:10 AM
To: users@nifi.apache.org
Cc: Kevin Verhoeven; Ki Kang
Subject: Re: Custom processor is failing for concurrency

Kumiko,

In general you shouldn't have to create threads in your processors, with the 
exception of some special cases.
The framework has a thread pool and it takes one of those threads and calls the 
onTrigger method of your processor.

If you want multiple threads to call onTrigger, then each processor has a 
Concurrent Tasks property in the UI on the scheduling tab,
which equates to the number of threads that will concurrently call onTrigger.

A processor developer needs to only worry about the business logic in the 
onTrigger method, and needs to ensure
thread-safe access to any member variables or state stored in the processor.

Hope that helps.

-Bryan

On Thu, Jun 9, 2016 at 2:11 PM, Kumiko Yada 
> wrote:
Microsoft found this is an issue with the SDK, they are working on a fix, they 
do not have the ETA for the fix.  To workaround this issue, I’m trying to 
create the multiple threads in using AbstractSessionFactoryProcessor and handle 
the Create a file in a single thread.   I’m having a problem that the single 
thread is not working correctly.  The processor is still acting like a single 
thread.

When I create a thread to handle the create a file, do I have to call this 
method using java.util.concurrent.ExecutorService?

Are there any sample processors that I can take a look?

Thanks
Kumiko

From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com]
Sent: Sunday, June 5, 2016 6:28 PM
To: users@nifi.apache.org
Cc: Ki Kang >; Kevin Verhoeven 
>
Subject: RE: Custom processor is failing for concurrency

Thank you, Bryan.  I’m working with Microsoft on this issue.  Will keep you 
guys updated.

Thanks
Kumiko

From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Friday, June 3, 2016 2:32 PM
To: users@nifi.apache.org
Subject: Re: Custom processor is failing for concurrency

It is hard to say for sure, but I think your NiFi processor is generally ok 
regarding thread safety, but I think there could be a problem in the Azure SDK 
code...

RequestFactory has an instance of BaseUrl and every time 
RequestFactory.create() is called, it calls BaseUrl.url().

The implementation of BaseUrl is the following (according to my IntelliJ 
attaching the sources...):

public class AutoRestBaseUrl implements BaseUrl {
/** A template based URL with variables wrapped in {}s. */
private String template;
/** a mapping from {} wrapped variables in the template and their actual 
values. */
private Map mappings;

@Override
public HttpUrl url() {
String url = template;
for (Map.Entry entry : mappings.entrySet()) {
url = url.replace(entry.getKey(), entry.getValue());
}
mappings.clear();
return HttpUrl.parse(url);
}

/**
* Creates an instance of a template based URL.
*
* @param url the template based URL to use.
*/
public AutoRestBaseUrl(String url) {
this.template = url;
this.mappings = new HashMap<>();
}

/**
* Sets the value for the {} wrapped variables in

Re: Custom processor is failing for concurrency

2016-06-09 Thread Oleg Zhurakousky

+1, was just responding with the same.

On Jun 9, 2016, at 3:11 PM, Matt Foley 
> wrote:

Kumiko, would it be sufficient to just wrap your call to the non-thread-safe MS 
SDK routine, in a 'synchronized' method?  You could then use the standard NiFi 
thread management and avoid a lot of complexity.  And the result should be >= 
efficiency of having a dedicated thread to manage the problem action.
--Matt F

From: Kumiko Yada >
Sent: Thursday, June 09, 2016 11:49 AM
To: users@nifi.apache.org
Cc: Kevin Verhoeven; Ki Kang
Subject: Re: Custom processor is failing for concurrency

Hi Bryan,

Does this mean that even I create the multiple threads in onTriger, I will 
still hit the Microsoft SDK issue where it's not a thread safe?  Sounds like 
basically what I am trying to do and creating the  multiple threads via UI 
might be the same thing.

Thanks
Kumiko

From: Bryan Bende >
Sent: Thursday, June 9, 2016 11:26:10 AM
To: users@nifi.apache.org
Cc: Kevin Verhoeven; Ki Kang
Subject: Re: Custom processor is failing for concurrency

Kumiko,

In general you shouldn't have to create threads in your processors, with the 
exception of some special cases.
The framework has a thread pool and it takes one of those threads and calls the 
onTrigger method of your processor.

If you want multiple threads to call onTrigger, then each processor has a 
Concurrent Tasks property in the UI on the scheduling tab,
which equates to the number of threads that will concurrently call onTrigger.

A processor developer needs to only worry about the business logic in the 
onTrigger method, and needs to ensure
thread-safe access to any member variables or state stored in the processor.

Hope that helps.

-Bryan

On Thu, Jun 9, 2016 at 2:11 PM, Kumiko Yada 
> wrote:
Microsoft found this is an issue with the SDK, they are working on a fix, they 
do not have the ETA for the fix.  To workaround this issue, I’m trying to 
create the multiple threads in using AbstractSessionFactoryProcessor and handle 
the Create a file in a single thread.   I’m having a problem that the single 
thread is not working correctly.  The processor is still acting like a single 
thread.

When I create a thread to handle the create a file, do I have to call this 
method using java.util.concurrent.ExecutorService?

Are there any sample processors that I can take a look?

Thanks
Kumiko

From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com]
Sent: Sunday, June 5, 2016 6:28 PM
To: users@nifi.apache.org
Cc: Ki Kang >; Kevin Verhoeven 
>
Subject: RE: Custom processor is failing for concurrency

Thank you, Bryan.  I’m working with Microsoft on this issue.  Will keep you 
guys updated.

Thanks
Kumiko

From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Friday, June 3, 2016 2:32 PM
To: users@nifi.apache.org
Subject: Re: Custom processor is failing for concurrency

It is hard to say for sure, but I think your NiFi processor is generally ok 
regarding thread safety, but I think there could be a problem in the Azure SDK 
code...

RequestFactory has an instance of BaseUrl and every time 
RequestFactory.create() is called, it calls BaseUrl.url().

The implementation of BaseUrl is the following (according to my IntelliJ 
attaching the sources...):

public class AutoRestBaseUrl implements BaseUrl {
/** A template based URL with variables wrapped in {}s. */
private String template;
/** a mapping from {} wrapped variables in the template and their actual 
values. */
private Map mappings;

@Override
public HttpUrl url() {
String url = template;
for (Map.Entry entry : mappings.entrySet()) {
url = url.replace(entry.getKey(), entry.getValue());
}
mappings.clear();
return HttpUrl.parse(url);
}

/**
* Creates an instance of a template based URL.
*
* @param url the template based URL to use.
*/
public AutoRestBaseUrl(String url) {
this.template = url;
this.mappings = new HashMap<>();
}

/**
* Sets the value for the {} wrapped variables in the template URL.
* @param matcher the {} wrapped variable to replace.
* @param value the value to set for the variable.
*/
public void set(CharSequence matcher, String value) {
this.mappings.put(matcher, value);
}
}

The exception is coming from the line where it is looping over the entryset:

for (Map.Entry entry : mappings.entrySet()) {

Right after that loop it calls mappings.clear() so if the RequestFactory is 
shared by multiple threads (which I think it is), then one thread could be

Re: Custom processor is failing for concurrency

2016-06-09 Thread Matt Foley

Kumiko, would it be sufficient to just wrap your call to the non-thread-safe MS 
SDK routine, in a 'synchronized' method?  You could then use the standard NiFi 
thread management and avoid a lot of complexity.  And the result should be >= 
efficiency of having a dedicated thread to manage the problem action.

--Matt F

From: Kumiko Yada 
Sent: Thursday, June 09, 2016 11:49 AM
To: users@nifi.apache.org
Cc: Kevin Verhoeven; Ki Kang
Subject: Re: Custom processor is failing for concurrency

Hi Bryan,

Does this mean that even I create the multiple threads in onTriger, I will 
still hit the Microsoft SDK issue where it's not a thread safe?  Sounds like 
basically what I am trying to do and creating the  multiple threads via UI 
might be the same thing.

Thanks
Kumiko

From: Bryan Bende 
Sent: Thursday, June 9, 2016 11:26:10 AM
To: users@nifi.apache.org
Cc: Kevin Verhoeven; Ki Kang
Subject: Re: Custom processor is failing for concurrency

Kumiko,

In general you shouldn't have to create threads in your processors, with the 
exception of some special cases.
The framework has a thread pool and it takes one of those threads and calls the 
onTrigger method of your processor.

If you want multiple threads to call onTrigger, then each processor has a 
Concurrent Tasks property in the UI on the scheduling tab,
which equates to the number of threads that will concurrently call onTrigger.

A processor developer needs to only worry about the business logic in the 
onTrigger method, and needs to ensure
thread-safe access to any member variables or state stored in the processor.

Hope that helps.

-Bryan

On Thu, Jun 9, 2016 at 2:11 PM, Kumiko Yada 
> wrote:
Microsoft found this is an issue with the SDK, they are working on a fix, they 
do not have the ETA for the fix.  To workaround this issue, I’m trying to 
create the multiple threads in using AbstractSessionFactoryProcessor and handle 
the Create a file in a single thread.   I’m having a problem that the single 
thread is not working correctly.  The processor is still acting like a single 
thread.

When I create a thread to handle the create a file, do I have to call this 
method using java.util.concurrent.ExecutorService?

Are there any sample processors that I can take a look?

Thanks
Kumiko

From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com]
Sent: Sunday, June 5, 2016 6:28 PM
To: users@nifi.apache.org
Cc: Ki Kang >; Kevin Verhoeven 
>
Subject: RE: Custom processor is failing for concurrency

Thank you, Bryan.  I’m working with Microsoft on this issue.  Will keep you 
guys updated.

Thanks
Kumiko

From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Friday, June 3, 2016 2:32 PM
To: users@nifi.apache.org
Subject: Re: Custom processor is failing for concurrency

It is hard to say for sure, but I think your NiFi processor is generally ok 
regarding thread safety, but I think there could be a problem in the Azure SDK 
code...

RequestFactory has an instance of BaseUrl and every time 
RequestFactory.create() is called, it calls BaseUrl.url().

The implementation of BaseUrl is the following (according to my IntelliJ 
attaching the sources...):

public class AutoRestBaseUrl implements BaseUrl {
/** A template based URL with variables wrapped in {}s. */
private String template;
/** a mapping from {} wrapped variables in the template and their actual 
values. */
private Map mappings;

@Override
public HttpUrl url() {
String url = template;
for (Map.Entry entry : mappings.entrySet()) {
url = url.replace(entry.getKey(), entry.getValue());
}
mappings.clear();
return HttpUrl.parse(url);
}

/**
* Creates an instance of a template based URL.
*
* @param url the template based URL to use.
*/
public AutoRestBaseUrl(String url) {
this.template = url;
this.mappings = new HashMap<>();
}

/**
* Sets the value for the {} wrapped variables in the template URL.
* @param matcher the {} wrapped variable to replace.
* @param value the value to set for the variable.
*/
public void set(CharSequence matcher, String value) {
this.mappings.put(matcher, value);
}
}

The exception is coming from the line where it is looping over the entryset:

for (Map.Entry entry : mappings.entrySet()) {

Right after that loop it calls mappings.clear() so if the RequestFactory is 
shared by multiple threads (which I think it is), then one thread could be 
iterating over the set, which another calls mappings.clear().

On Fri, Jun 3, 2016 at 5:09 PM, Oleg Zhurakousky 
> wrote:
Kumiko

It appears that the current state of the source you linked in

Re: Custom processor is failing for concurrency

2016-06-09 Thread Bryan Bende

Yes I think it ends up being the same thing.

If you create multiple threads that all use the
same DataLakeStoreFileSystemManagementClient,
or if you increase concurrent threads > 1 in the UI, both will potentially
run into the problem in the Microsoft SDK.

On Thu, Jun 9, 2016 at 2:49 PM, Kumiko Yada  wrote:

> Hi Bryan,
>
> Does this mean that even I create the multiple threads in onTriger, I will
> still hit the Microsoft SDK issue where it's not a thread safe?  Sounds
> like basically what I am trying to do and creating the  multiple threads
> via UI might be the same thing.
>
> Thanks
> Kumiko
> --
> *From:* Bryan Bende 
> *Sent:* Thursday, June 9, 2016 11:26:10 AM
> *To:* users@nifi.apache.org
> *Cc:* Kevin Verhoeven; Ki Kang
>
> *Subject:* Re: Custom processor is failing for concurrency
>
> Kumiko,
>
> In general you shouldn't have to create threads in your processors, with
> the exception of some special cases.
> The framework has a thread pool and it takes one of those threads and
> calls the onTrigger method of your processor.
>
> If you want multiple threads to call onTrigger, then each processor has a
> Concurrent Tasks property in the UI on the scheduling tab,
> which equates to the number of threads that will concurrently call
> onTrigger.
>
> A processor developer needs to only worry about the business logic in the
> onTrigger method, and needs to ensure
> thread-safe access to any member variables or state stored in the
> processor.
>
> Hope that helps.
>
> -Bryan
>
>
> On Thu, Jun 9, 2016 at 2:11 PM, Kumiko Yada  wrote:
>
>> Microsoft found this is an issue with the SDK, they are working on a fix,
>> they do not have the ETA for the fix.  To workaround this issue, I’m trying
>> to create the multiple threads in using AbstractSessionFactoryProcessor and
>> handle the Create a file in a single thread.   I’m having a problem that
>> the single thread is not working correctly.  The processor is still acting
>> like a single thread.
>>
>>
>>
>> When I create a thread to handle the create a file, do I have to call
>> this method using java.util.concurrent.ExecutorService?
>>
>>
>>
>> Are there any sample processors that I can take a look?
>>
>>
>>
>> Thanks
>>
>> Kumiko
>>
>>
>>
>> *From:* Kumiko Yada [mailto:kumiko.y...@ds-iq.com]
>> *Sent:* Sunday, June 5, 2016 6:28 PM
>> *To:* users@nifi.apache.org
>> *Cc:* Ki Kang ; Kevin Verhoeven <
>> kevin.verhoe...@ds-iq.com>
>> *Subject:* RE: Custom processor is failing for concurrency
>>
>>
>>
>> Thank you, Bryan.  I’m working with Microsoft on this issue.  Will keep
>> you guys updated.
>>
>>
>>
>> Thanks
>>
>> Kumiko
>>
>>
>>
>> *From:* Bryan Bende [mailto:bbe...@gmail.com ]
>> *Sent:* Friday, June 3, 2016 2:32 PM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: Custom processor is failing for concurrency
>>
>>
>>
>> It is hard to say for sure, but I think your NiFi processor is generally
>> ok regarding thread safety, but I think there could be a problem in the
>> Azure SDK code...
>>
>>
>>
>> RequestFactory has an instance of BaseUrl and every time
>> RequestFactory.create() is called, it calls BaseUrl.url().
>>
>>
>>
>> The implementation of BaseUrl is the following (according to my IntelliJ
>> attaching the sources...):
>>
>>
>>
>> public class AutoRestBaseUrl implements BaseUrl {
>> /** A template based URL with variables wrapped in {}s. */
>> private String template;
>> /** a mapping from {} wrapped variables in the template and their actual
>> values. */
>> private Map mappings;
>>
>> @Override
>> public HttpUrl url() {
>> String url = template;
>> for (Map.Entry entry : mappings.entrySet()) {
>> url = url.replace(entry.getKey(), entry.getValue());
>> }
>> mappings.clear();
>> return HttpUrl.parse(url);
>> }
>>
>> /**
>> * Creates an instance of a template based URL.
>> *
>> * @param url the template based URL to use.
>> */
>> public AutoRestBaseUrl(String url) {
>> this.template = url;
>> this.mappings = new HashMap<>();
>> }
>>
>> /**
>> * Sets the value for the {} wrapped variables in the template URL.
>> * @param matcher the {} wrapped variable to replace.
>> * @param value the value to set for the variable.
>> */
>> public void set(CharSequence matcher, String value) {
>> this.mappings.put(matcher, value);
>> }
>> }
>>
>>
>>
>> The exception is coming from the line where it is looping over the
>> entryset:
>>
>>
>>
>> for (Map.Entry entry : mappings.entrySet()) {
>>
>>
>>
>> Right after that loop it calls mappings.clear() so if the RequestFactory
>> is shared by multiple threads (which I think it is), then one thread could
>> be iterating over the set, which another calls mappings.clear().
>>
>>
>>
>>
>>
>> On Fri, Jun 3, 2016 at 5:09 PM, Oleg Zhurakousky <
>> ozhurakou...@hortonworks.com> wrote:
>>
>> Kumiko
>>
>>
>>
>> It appears that the

Re: Custom processor is failing for concurrency

2016-06-09 Thread Kumiko Yada

Hi Bryan,

Does this mean that even I create the multiple threads in onTriger, I will 
still hit the Microsoft SDK issue where it's not a thread safe?  Sounds like 
basically what I am trying to do and creating the  multiple threads via UI 
might be the same thing.

Thanks
Kumiko

From: Bryan Bende 
Sent: Thursday, June 9, 2016 11:26:10 AM
To: users@nifi.apache.org
Cc: Kevin Verhoeven; Ki Kang
Subject: Re: Custom processor is failing for concurrency

Kumiko,

In general you shouldn't have to create threads in your processors, with the 
exception of some special cases.
The framework has a thread pool and it takes one of those threads and calls the 
onTrigger method of your processor.

If you want multiple threads to call onTrigger, then each processor has a 
Concurrent Tasks property in the UI on the scheduling tab,
which equates to the number of threads that will concurrently call onTrigger.

A processor developer needs to only worry about the business logic in the 
onTrigger method, and needs to ensure
thread-safe access to any member variables or state stored in the processor.

Hope that helps.

-Bryan

On Thu, Jun 9, 2016 at 2:11 PM, Kumiko Yada 
> wrote:
Microsoft found this is an issue with the SDK, they are working on a fix, they 
do not have the ETA for the fix.  To workaround this issue, I’m trying to 
create the multiple threads in using AbstractSessionFactoryProcessor and handle 
the Create a file in a single thread.   I’m having a problem that the single 
thread is not working correctly.  The processor is still acting like a single 
thread.

When I create a thread to handle the create a file, do I have to call this 
method using java.util.concurrent.ExecutorService?

Are there any sample processors that I can take a look?

Thanks
Kumiko

From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com]
Sent: Sunday, June 5, 2016 6:28 PM
To: users@nifi.apache.org
Cc: Ki Kang >; Kevin Verhoeven 
>
Subject: RE: Custom processor is failing for concurrency

Thank you, Bryan.  I’m working with Microsoft on this issue.  Will keep you 
guys updated.

Thanks
Kumiko

From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Friday, June 3, 2016 2:32 PM
To: users@nifi.apache.org
Subject: Re: Custom processor is failing for concurrency

It is hard to say for sure, but I think your NiFi processor is generally ok 
regarding thread safety, but I think there could be a problem in the Azure SDK 
code...

RequestFactory has an instance of BaseUrl and every time 
RequestFactory.create() is called, it calls BaseUrl.url().

The implementation of BaseUrl is the following (according to my IntelliJ 
attaching the sources...):

public class AutoRestBaseUrl implements BaseUrl {
/** A template based URL with variables wrapped in {}s. */
private String template;
/** a mapping from {} wrapped variables in the template and their actual 
values. */
private Map mappings;

@Override
public HttpUrl url() {
String url = template;
for (Map.Entry entry : mappings.entrySet()) {
url = url.replace(entry.getKey(), entry.getValue());
}
mappings.clear();
return HttpUrl.parse(url);
}

/**
* Creates an instance of a template based URL.
*
* @param url the template based URL to use.
*/
public AutoRestBaseUrl(String url) {
this.template = url;
this.mappings = new HashMap<>();
}

/**
* Sets the value for the {} wrapped variables in the template URL.
* @param matcher the {} wrapped variable to replace.
* @param value the value to set for the variable.
*/
public void set(CharSequence matcher, String value) {
this.mappings.put(matcher, value);
}
}

The exception is coming from the line where it is looping over the entryset:

for (Map.Entry entry : mappings.entrySet()) {

Right after that loop it calls mappings.clear() so if the RequestFactory is 
shared by multiple threads (which I think it is), then one thread could be 
iterating over the set, which another calls mappings.clear().

On Fri, Jun 3, 2016 at 5:09 PM, Oleg Zhurakousky 
> wrote:
Kumiko

It appears that the current state of the source you linked in is not in sync 
with what is in the stack trace. Perhaps you have made some code modifications 
(e.g., line 218 is an empty line in code while it has a pointer in the star 
trace).
In any event, from what I can see the error is coming from Azure libraries (not 
NiFi). Specifically ‘com.microsoft.rest.AutoRestBaseUrl.url(..)’ seems to be 
doing some iteration where I presume the remove is called. Perhaps it is not a 
thread safe class after all. What does Microsoft documentation says? Have you 
looked at the source to see what’s going on there?

Re: Custom processor is failing for concurrency

2016-06-09 Thread Bryan Bende

Kumiko,

In general you shouldn't have to create threads in your processors, with
the exception of some special cases.
The framework has a thread pool and it takes one of those threads and calls
the onTrigger method of your processor.

If you want multiple threads to call onTrigger, then each processor has a
Concurrent Tasks property in the UI on the scheduling tab,
which equates to the number of threads that will concurrently call
onTrigger.

A processor developer needs to only worry about the business logic in the
onTrigger method, and needs to ensure
thread-safe access to any member variables or state stored in the processor.

Hope that helps.

-Bryan


On Thu, Jun 9, 2016 at 2:11 PM, Kumiko Yada  wrote:

> Microsoft found this is an issue with the SDK, they are working on a fix,
> they do not have the ETA for the fix.  To workaround this issue, I’m trying
> to create the multiple threads in using AbstractSessionFactoryProcessor and
> handle the Create a file in a single thread.   I’m having a problem that
> the single thread is not working correctly.  The processor is still acting
> like a single thread.
>
>
>
> When I create a thread to handle the create a file, do I have to call this
> method using java.util.concurrent.ExecutorService?
>
>
>
> Are there any sample processors that I can take a look?
>
>
>
> Thanks
>
> Kumiko
>
>
>
> *From:* Kumiko Yada [mailto:kumiko.y...@ds-iq.com]
> *Sent:* Sunday, June 5, 2016 6:28 PM
> *To:* users@nifi.apache.org
> *Cc:* Ki Kang ; Kevin Verhoeven <
> kevin.verhoe...@ds-iq.com>
> *Subject:* RE: Custom processor is failing for concurrency
>
>
>
> Thank you, Bryan.  I’m working with Microsoft on this issue.  Will keep
> you guys updated.
>
>
>
> Thanks
>
> Kumiko
>
>
>
> *From:* Bryan Bende [mailto:bbe...@gmail.com ]
> *Sent:* Friday, June 3, 2016 2:32 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: Custom processor is failing for concurrency
>
>
>
> It is hard to say for sure, but I think your NiFi processor is generally
> ok regarding thread safety, but I think there could be a problem in the
> Azure SDK code...
>
>
>
> RequestFactory has an instance of BaseUrl and every time
> RequestFactory.create() is called, it calls BaseUrl.url().
>
>
>
> The implementation of BaseUrl is the following (according to my IntelliJ
> attaching the sources...):
>
>
>
> public class AutoRestBaseUrl implements BaseUrl {
> /** A template based URL with variables wrapped in {}s. */
> private String template;
> /** a mapping from {} wrapped variables in the template and their actual
> values. */
> private Map mappings;
>
> @Override
> public HttpUrl url() {
> String url = template;
> for (Map.Entry entry : mappings.entrySet()) {
> url = url.replace(entry.getKey(), entry.getValue());
> }
> mappings.clear();
> return HttpUrl.parse(url);
> }
>
> /**
> * Creates an instance of a template based URL.
> *
> * @param url the template based URL to use.
> */
> public AutoRestBaseUrl(String url) {
> this.template = url;
> this.mappings = new HashMap<>();
> }
>
> /**
> * Sets the value for the {} wrapped variables in the template URL.
> * @param matcher the {} wrapped variable to replace.
> * @param value the value to set for the variable.
> */
> public void set(CharSequence matcher, String value) {
> this.mappings.put(matcher, value);
> }
> }
>
>
>
> The exception is coming from the line where it is looping over the
> entryset:
>
>
>
> for (Map.Entry entry : mappings.entrySet()) {
>
>
>
> Right after that loop it calls mappings.clear() so if the RequestFactory
> is shared by multiple threads (which I think it is), then one thread could
> be iterating over the set, which another calls mappings.clear().
>
>
>
>
>
> On Fri, Jun 3, 2016 at 5:09 PM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com> wrote:
>
> Kumiko
>
>
>
> It appears that the current state of the source you linked in is not in
> sync with what is in the stack trace. Perhaps you have made some code
> modifications (e.g., line 218 is an empty line in code while it has a
> pointer in the star trace).
>
> In any event, from what I can see the error is coming from Azure libraries
> (not NiFi). Specifically ‘com.microsoft.rest.AutoRestBaseUrl.url(..)’ seems
> to be doing some iteration where I presume the remove is called. Perhaps it
> is not a thread safe class after all. What does Microsoft documentation
> says? Have you looked at the source to see what’s going on there? If its
> open please link and we can tale a look.
>
>
>
> Cheers
>
> Oleg
>
>
>
> On Jun 3, 2016, at 4:58 PM, Kumiko Yada  wrote:
>
>
>
> Here is the code, https://github.com/kyada1/PutFileAzureDLStore.
>
>
>
> Thanks
>
> Kumiko
>
>
>
> *From:* Bryan Bende [mailto:bbe...@gmail.com ]
> *Sent:* Friday, June 3, 2016 12:57 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: Custom processor is failing for the

RE: Custom processor is failing for concurrency

2016-06-09 Thread Kumiko Yada

Microsoft found this is an issue with the SDK, they are working on a fix, they 
do not have the ETA for the fix.  To workaround this issue, I’m trying to 
create the multiple threads in using AbstractSessionFactoryProcessor and handle 
the Create a file in a single thread.   I’m having a problem that the single 
thread is not working correctly.  The processor is still acting like a single 
thread.

When I create a thread to handle the create a file, do I have to call this 
method using java.util.concurrent.ExecutorService?

Are there any sample processors that I can take a look?

Thanks
Kumiko

From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com]
Sent: Sunday, June 5, 2016 6:28 PM
To: users@nifi.apache.org
Cc: Ki Kang ; Kevin Verhoeven 
Subject: RE: Custom processor is failing for concurrency

Thank you, Bryan.  I’m working with Microsoft on this issue.  Will keep you 
guys updated.

Thanks
Kumiko

From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Friday, June 3, 2016 2:32 PM
To: users@nifi.apache.org
Subject: Re: Custom processor is failing for concurrency

It is hard to say for sure, but I think your NiFi processor is generally ok 
regarding thread safety, but I think there could be a problem in the Azure SDK 
code...

RequestFactory has an instance of BaseUrl and every time 
RequestFactory.create() is called, it calls BaseUrl.url().

The implementation of BaseUrl is the following (according to my IntelliJ 
attaching the sources...):

public class AutoRestBaseUrl implements BaseUrl {
/** A template based URL with variables wrapped in {}s. */
private String template;
/** a mapping from {} wrapped variables in the template and their actual 
values. */
private Map mappings;

@Override
public HttpUrl url() {
String url = template;
for (Map.Entry entry : mappings.entrySet()) {
url = url.replace(entry.getKey(), entry.getValue());
}
mappings.clear();
return HttpUrl.parse(url);
}

/**
* Creates an instance of a template based URL.
*
* @param url the template based URL to use.
*/
public AutoRestBaseUrl(String url) {
this.template = url;
this.mappings = new HashMap<>();
}

/**
* Sets the value for the {} wrapped variables in the template URL.
* @param matcher the {} wrapped variable to replace.
* @param value the value to set for the variable.
*/
public void set(CharSequence matcher, String value) {
this.mappings.put(matcher, value);
}
}

The exception is coming from the line where it is looping over the entryset:

for (Map.Entry entry : mappings.entrySet()) {

Right after that loop it calls mappings.clear() so if the RequestFactory is 
shared by multiple threads (which I think it is), then one thread could be 
iterating over the set, which another calls mappings.clear().

On Fri, Jun 3, 2016 at 5:09 PM, Oleg Zhurakousky 
> wrote:
Kumiko

It appears that the current state of the source you linked in is not in sync 
with what is in the stack trace. Perhaps you have made some code modifications 
(e.g., line 218 is an empty line in code while it has a pointer in the star 
trace).
In any event, from what I can see the error is coming from Azure libraries (not 
NiFi). Specifically ‘com.microsoft.rest.AutoRestBaseUrl.url(..)’ seems to be 
doing some iteration where I presume the remove is called. Perhaps it is not a 
thread safe class after all. What does Microsoft documentation says? Have you 
looked at the source to see what’s going on there? If its open please link and 
we can tale a look.

Cheers
Oleg

On Jun 3, 2016, at 4:58 PM, Kumiko Yada 
> wrote:

Here is the code, https://github.com/kyada1/PutFileAzureDLStore.

Thanks
Kumiko

From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Friday, June 3, 2016 12:57 PM
To: users@nifi.apache.org
Subject: Re: Custom processor is failing for the custom processor

Hello,

Would you be able to share your code for PutFileAzureDLStore so we can help 
identify if there is a concurrency problem?

-Bryan

On Fri, Jun 3, 2016 at 3:39 PM, Kumiko Yada 
> wrote:
Hello,

I wrote the following custom service control and processor.  When the custom 
processor is running concurrently, it’s failing often with several different 
errors.  Are there any special handlings for concurrently that I need to add in 
the custom processor?  I wrote the sample Java program which does the same 
thing as the custom processor (authenticate every time the file is 
created/create a file, create 2 threads and run concurrently), it’s working 
fine.  The custom processor also fine when this is not running concurrently.

Custom service control – set the properties for the Microsoft Azure Datalake 
Store
Custom processor – authenticate, then create a file in Microsoft Azure

Re: Kerberos / NiFi 0.5.1 / ListHDFS

2016-06-09 Thread Michael Dyer

Brian,

Thank you - that was the problem.

I updated my xml files from the cluster in question and all is working fine
now.

Michael

Re: Best way to compare dates

2016-06-09 Thread James Wing

Igor,

One way would be to format both dates as strings (like "20160609" in your
HDFS paths) first, then compare the two strings for equality.  In a
RouteOnAttribute expression:

${now():format("MMdd"):equals(${entryDate:format("MMdd")})}

If your goal is to merge the records into larger files, you might be able
to do this with MergeContent, if you have the Correlation Attribute set to
an attribute populated with the MMdd-formatteddate so it bins files by
day.

Thanks,

James

On Thu, Jun 9, 2016 at 7:37 AM, Igor Kravzov <igork.ine...@gmail.com> wrote:

> Hi,
>
> In my workflow data is coming form Kafka Topic in JSON format and saved ot
> HDFS directory by day.
>
> HDFS directory path constructed as /mypath/${now():format("MMdd")}
> JSON file contains attribute {... "adddate":"2016-06-07 17:06:17" }
>
> I want to avoid over-spill of a previous date files into "today's"
> directory.
> The thing is if add date is the same as taday's date, a want file merged
> to a bigger file and save to HDFS. Previous file not merged and saved as is.
>
> I am thinking to use EvaluateJsonPath to grab add date and to use
> RouteOnAttribute to direct to appropriate  path.
>
> How comparison would look like?
>
> Thanks in advance.
>

Best way to compare dates

2016-06-09 Thread Igor Kravzov

Hi,

In my workflow data is coming form Kafka Topic in JSON format and saved ot
HDFS directory by day.

HDFS directory path constructed as /mypath/${now():format("MMdd")}
JSON file contains attribute {... "adddate":"2016-06-07 17:06:17" }

I want to avoid over-spill of a previous date files into "today's"
directory.
The thing is if add date is the same as taday's date, a want file merged to
a bigger file and save to HDFS. Previous file not merged and saved as is.

I am thinking to use EvaluateJsonPath to grab add date and to use
RouteOnAttribute to direct to appropriate  path.

How comparison would look like?

Thanks in advance.

RE: Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl

Re: Funnel

RE: Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl

Re: Failure when running a workflow created from a template from another NiFi version.

Re: Failure when running a workflow created from a template from another NiFi version.

Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl

Funnel

RE: Custom processor is failing for concurrency

Re: Not a Valid JSON content

Re: Best way to compare dates

Not a Valid JSON content

Re: Custom processor is failing for concurrency

Re: Custom processor is failing for concurrency

Re: Custom processor is failing for concurrency

Re: Custom processor is failing for concurrency

Re: Custom processor is failing for concurrency

Re: Custom processor is failing for concurrency

RE: Custom processor is failing for concurrency

Re: Kerberos / NiFi 0.5.1 / ListHDFS

Re: Best way to compare dates

Best way to compare dates

21 matches

Site Navigation

Mail list logo

Footer information