RE: Upgrade 0.7.0 to 1.0.0
These are two specific questions that we have.
1) In Nifi Summary/Remove Process Group, what value need to be used for
the Target URI. We were able to use the specific URI in Nifi 0.7.0 because we
had to specified the master node; however, Nifi 1.0.0 uses zero master
clustering and automatically assign the master.
2) We are getting the error "2016-09-08 16:00:01,246 ERROR [NiFi Web
Server-6153] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Cannot replicate request
GET /nifi-api/site-to-site because there are 100 outstanding HTTP Requests
already. Request Counts Per URI = {/nifi-api/site-to-site=99,
/nifi-api/flow/current-user=1}.
In
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/ThreadPoolRequestReplicator.java,
the limit is hardcoded (MAX_CONCURRENT_REQUESTS = 100) in the
ThreadPoolRequestReplicator class, which is completely new with version 1.0.
Are we getting this error because we hit this hard coded value 100? If so,
what is the best workaround, would the increase this value a good idea?
Thanks
Kumiko
From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com]
Sent: Friday, September 9, 2016 10:06 AM
To: users@nifi.apache.org
Cc: Chien Le <chien...@ds-iq.com>; Kevin Verhoeven <kevin.verhoe...@ds-iq.com>;
Wei Zhang <wei.zh...@ds-iq.com>
Subject: Upgrade 0.7.0 to 1.0.0
Hello,
We upgraded from 0.7.0 to 1.0.0, and we encountered some issues. We think that
this upgrade brought a change to how the site-to-site calls are made for the
remote process groups and is causing us to hit a max requests limit that was
also introduced with v1.0. Is this a known issue?
The upgrade changes the serialization version of the provenance files that is
not backwards compatibility (nifi-1.0 -> 1-9, nifi-0.7 -> 1-8). Is there any
workaround this?
Thanks
Kumiko
RE: Issue writing file (~50mb) to azure data lake with Nifi
Lorenzo was not clear how we tested. I wrote the sample java program using the Azure SDK, then uploaded the 50 MB file and it’s worked without error. Nifi custom processor used the same SDK code; however, it’s failing when the custom processor is tried to uploaded 50 MB file. Thanks Kumiko From: Tony Kurc [mailto:trk...@gmail.com] Sent: Tuesday, September 6, 2016 3:02 PM To: Kumiko Yada <kumiko.y...@ds-iq.com> Cc: users@nifi.apache.org; Joe Witt <joe.w...@gmail.com>; #Operations Automation and Tools <#opsto...@ds-iq.com>; Kevin Verhoeven <kevin.verhoe...@ds-iq.com> Subject: RE: Issue writing file (~50mb) to azure data lake with Nifi I apologize if I'm missing something, as I'm trying to read the code on my phone, but it looks like the test script is using a different api call to perform the upload - did you already test using the same call in your script? On Sep 6, 2016 5:48 PM, "Kumiko Yada" <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: Here is the code: https://github.com/kyada1/ADL_UploadFile. I removed the following values for the security reason. final static String ADLS_ACCOUNT_NAME = ""; final static String RESOURCE_GROUP_NAME = ""; final static String LOCATION = ""; final static String TENANT_ID = ""; final static String SUBSCRIPTION_ID = ""; final static String CLIENT_ID = ""; final static String CLIENT_SECRET = ""; / Thanks Kumiko From: Tony Kurc [mailto:trk...@gmail.com<mailto:trk...@gmail.com>] Sent: Tuesday, September 6, 2016 2:41 PM To: Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> Cc: Joe Witt <joe.w...@gmail.com<mailto:joe.w...@gmail.com>>; users@nifi.apache.org<mailto:users@nifi.apache.org>; #Operations Automation and Tools <#opsto...@ds-iq.com<mailto:opsto...@ds-iq.com>>; Kevin Verhoeven <kevin.verhoe...@ds-iq.com<mailto:kevin.verhoe...@ds-iq.com>> Subject: RE: Issue writing file (~50mb) to azure data lake with Nifi I was referring to this: "Then we wrote a java sdk script to upload this same file without Nifi into the data lake and it worked successfully." Is that code somewhere? On Sep 6, 2016 5:38 PM, "Kumiko Yada" <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: I didn’t add any test code. This custom controller and processor is working for a small size file. Thanks Kumiko From: Tony Kurc [mailto:trk...@gmail.com<mailto:trk...@gmail.com>] Sent: Tuesday, September 6, 2016 2:32 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Cc: Joe Witt <joe.w...@gmail.com<mailto:joe.w...@gmail.com>>; #Operations Automation and Tools <#opsto...@ds-iq.com<mailto:opsto...@ds-iq.com>>; Kevin Verhoeven <kevin.verhoe...@ds-iq.com<mailto:kevin.verhoe...@ds-iq.com>> Subject: RE: Issue writing file (~50mb) to azure data lake with Nifi I didn't see the test script that worked in the source code - did I miss it, or is it not in the tree? On Sep 6, 2016 3:17 PM, "Kumiko Yada" <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: Joe, Here is the log (there was no callstack related to this error) and code, https://github.com/kyada1/dl_sdkworkaround/tree/master/nifi-azure-dlstore-bundle. 2016-09-06 12:06:50,508 INFO [NiFi Web Server-19] c.s.j.s.i.application.WebApplicationImpl Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25 AM' 2016-09-06 12:07:00,991 INFO [StandardProcessScheduler Thread-1] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled PutFileAzureDLStore[id=00dd95dc-0157-1000-8ab1-2de88c159b55] to run with 1 threads 2016-09-06 12:07:01,545 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@28f414a1<mailto:org.apache.nifi.controller.FlowController@28f414a1> // Another save pending = false 2016-09-06 12:07:01,904 INFO [pool-27-thread-1] c.m.aad.adal4j.AuthenticationAuthority [Correlation ID: 564fb5ec-643b-43e6-ab68-59f259a4843a] Instance discovery was successful 2016-09-06 12:08:05,988 ERROR [Timer-Driven Process Thread-1] n.a.d.processors.PutFileAzureDLStore PutFileAzureDLStore[id=00dd95dc-0157-1000-8ab1-2de88c159b55] File was not created: /kumiko/test/20160906120701022.txt com.microsoft.azure.management.datalake.store.models.AdlsErrorException: Invalid status code 404 2016-09-06 12:08:12,541 INFO [Provenance Maintenance Thread-1] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 618 2016-09-06 12:08:12,838 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (1 records) into single Provenance Log File .\provenance_repository\617.prov in 305 milliseconds 2016-09-06
RE: Issue writing file (~50mb) to azure data lake with Nifi
Here is the code: https://github.com/kyada1/ADL_UploadFile. I removed the following values for the security reason. final static String ADLS_ACCOUNT_NAME = ""; final static String RESOURCE_GROUP_NAME = ""; final static String LOCATION = ""; final static String TENANT_ID = ""; final static String SUBSCRIPTION_ID = ""; final static String CLIENT_ID = ""; final static String CLIENT_SECRET = ""; / Thanks Kumiko From: Tony Kurc [mailto:trk...@gmail.com] Sent: Tuesday, September 6, 2016 2:41 PM To: Kumiko Yada <kumiko.y...@ds-iq.com> Cc: Joe Witt <joe.w...@gmail.com>; users@nifi.apache.org; #Operations Automation and Tools <#opsto...@ds-iq.com>; Kevin Verhoeven <kevin.verhoe...@ds-iq.com> Subject: RE: Issue writing file (~50mb) to azure data lake with Nifi I was referring to this: "Then we wrote a java sdk script to upload this same file without Nifi into the data lake and it worked successfully." Is that code somewhere? On Sep 6, 2016 5:38 PM, "Kumiko Yada" <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: I didn’t add any test code. This custom controller and processor is working for a small size file. Thanks Kumiko From: Tony Kurc [mailto:trk...@gmail.com<mailto:trk...@gmail.com>] Sent: Tuesday, September 6, 2016 2:32 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Cc: Joe Witt <joe.w...@gmail.com<mailto:joe.w...@gmail.com>>; #Operations Automation and Tools <#opsto...@ds-iq.com<mailto:opsto...@ds-iq.com>>; Kevin Verhoeven <kevin.verhoe...@ds-iq.com<mailto:kevin.verhoe...@ds-iq.com>> Subject: RE: Issue writing file (~50mb) to azure data lake with Nifi I didn't see the test script that worked in the source code - did I miss it, or is it not in the tree? On Sep 6, 2016 3:17 PM, "Kumiko Yada" <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: Joe, Here is the log (there was no callstack related to this error) and code, https://github.com/kyada1/dl_sdkworkaround/tree/master/nifi-azure-dlstore-bundle. 2016-09-06 12:06:50,508 INFO [NiFi Web Server-19] c.s.j.s.i.application.WebApplicationImpl Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25 AM' 2016-09-06 12:07:00,991 INFO [StandardProcessScheduler Thread-1] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled PutFileAzureDLStore[id=00dd95dc-0157-1000-8ab1-2de88c159b55] to run with 1 threads 2016-09-06 12:07:01,545 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@28f414a1<mailto:org.apache.nifi.controller.FlowController@28f414a1> // Another save pending = false 2016-09-06 12:07:01,904 INFO [pool-27-thread-1] c.m.aad.adal4j.AuthenticationAuthority [Correlation ID: 564fb5ec-643b-43e6-ab68-59f259a4843a] Instance discovery was successful 2016-09-06 12:08:05,988 ERROR [Timer-Driven Process Thread-1] n.a.d.processors.PutFileAzureDLStore PutFileAzureDLStore[id=00dd95dc-0157-1000-8ab1-2de88c159b55] File was not created: /kumiko/test/20160906120701022.txt com.microsoft.azure.management.datalake.store.models.AdlsErrorException: Invalid status code 404 2016-09-06 12:08:12,541 INFO [Provenance Maintenance Thread-1] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 618 2016-09-06 12:08:12,838 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (1 records) into single Provenance Log File .\provenance_repository\617.prov in 305 milliseconds 2016-09-06 12:08:12,838 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 1 records 2016-09-06 12:08:21,148 INFO [NiFi Web Server-28] o.a.n.controller.StandardProcessorNode Stopping processor: class nifi.azure.dlstore.processors.PutFileAzureDLStore Thanks Kumiko From: Joe Witt [mailto:joe.w...@gmail.com<mailto:joe.w...@gmail.com>] Sent: Friday, September 2, 2016 2:56 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Cc: #Operations Automation and Tools <#opsto...@ds-iq.com<mailto:opsto...@ds-iq.com>>; Kevin Verhoeven <kevin.verhoe...@ds-iq.com<mailto:kevin.verhoe...@ds-iq.com>> Subject: Re: Issue writing file (~50mb) to azure data lake with Nifi Lorenzo Without seeing the code and logs it would be very difficult to help. nifi has no trouble by design writing large files (GBs) to many things including hdfs so the issue is probably in how this client library interacts with the data stream. On Sep 2, 2016 4:19 PM, "Lorenzo Peder" <lorenzo.pe...@ds-iq.com<mailto:lorenzo.pe...@ds-iq.com>> wrote: Hi All, We’ve run into an issue uploading a larger file (~50Mb) into
RE: IDE-specific setup
IntelliJ works perfectly, thank you Bryan and Matt! -Kumiko -Original Message- From: Matt Burgess [mailto:mattyb...@gmail.com] Sent: Tuesday, June 21, 2016 2:31 PM To: users@nifi.apache.org Subject: Re: IDE-specific setup Same as Bryan, I choose New (Project or Module) from Existing Sources and point at the POM for that directory/project/module, IntelliJ does a good job of getting everything set up. On Tue, Jun 21, 2016 at 5:29 PM, Bryan Bende <bbe...@gmail.com> wrote: > I personally use IntelliJ and it generally does pretty well at > automatically importing everything. > > The process to import an existing Maven project is described here: > https://www.jetbrains.com/help/idea/2016.1/importing-project-from-mave > n-model.html > > In Step #2 you would select the root directory of the nifi source code > that you cloned from git. After that I think all of the defaults should work. > > -Bryan > > > On Tue, Jun 21, 2016 at 4:47 PM, Kumiko Yada <kumiko.y...@ds-iq.com> wrote: >> >> I downloaded the source code by git clone >> http://git-wip-us.apache.org/repos/asf/nifi.git. Is it possible >> import all aggravate projects in Eclipse? When I imported the Maven >> proejcts, I have been getting many errors and How everyone is setting >> up the Nifi source code? >> >> >> >> I also imported my custom processor project in Eclipse, I can’t get >> the Intellisense to work that I will get the error “This compilation >> unit is not on the build path on a Java project). I’m a beginner >> level on Java, so I have a limited knowledge to any Java IDEs. >> >> >> >> Thanks >> >> Kumiko >> >> >> >> From: Andy LoPresto [mailto:alopre...@apache.org] >> Sent: Tuesday, June 21, 2016 10:10 AM >> >> >> To: users@nifi.apache.org >> Subject: Re: IDE-specific setup >> >> >> >> For more developer-focused information, you should also consider >> joining the dev mailing list [1]. >> >> >> >> [1] https://nifi.apache.org/mailing_lists.html >> >> >> >> Andy LoPresto >> >> alopre...@apache.org >> >> alopresto.apa...@gmail.com >> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> >> >> On Jun 21, 2016, at 10:06 AM, Kumiko Yada <kumiko.y...@ds-iq.com> wrote: >> >> >> >> Hi Andy, >> >> >> >> I read several emails from the user mailing list archive that someone >> was going to write up the how to setup the IDE. However, I didn’t >> find any documents, so I was just wondering. >> >> >> >> Thanks >> >> Kumiko >> >> >> >> From: Andy LoPresto [mailto:alopre...@apache.org] >> Sent: Tuesday, June 21, 2016 10:03 AM >> To: users@nifi.apache.org >> Subject: Re: IDE-specific setup >> >> >> >> Kumiko, >> >> >> >> The Contributor Guide [1] has some information about configuring the >> NiFi checkstyle rules for IntelliJ IDEA and Eclipse. Are you looking >> for something specific? >> >> >> >> [1] >> https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#Co >> ntributorGuide-CodeStyle >> >> >> >> Andy LoPresto >> >> alopre...@apache.org >> >> alopresto.apa...@gmail.com >> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> >> >> On Jun 21, 2016, at 9:57 AM, Kumiko Yada <kumiko.y...@ds-iq.com> wrote: >> >> >> >> Hello, >> >> >> >> Is there any documentations on how to setup the Nifi with >> IDE-specific setup? >> >> >> >> Thanks >> >> Kumiko >> >> > >
RE: IDE-specific setup
I downloaded the source code by git clone http://git-wip-us.apache.org/repos/asf/nifi.git. Is it possible import all aggravate projects in Eclipse? When I imported the Maven proejcts, I have been getting many errors and How everyone is setting up the Nifi source code? I also imported my custom processor project in Eclipse, I can’t get the Intellisense to work that I will get the error “This compilation unit is not on the build path on a Java project). I’m a beginner level on Java, so I have a limited knowledge to any Java IDEs. Thanks Kumiko From: Andy LoPresto [mailto:alopre...@apache.org] Sent: Tuesday, June 21, 2016 10:10 AM To: users@nifi.apache.org Subject: Re: IDE-specific setup For more developer-focused information, you should also consider joining the dev mailing list [1]. [1] https://nifi.apache.org/mailing_lists.html Andy LoPresto alopre...@apache.org<mailto:alopre...@apache.org> alopresto.apa...@gmail.com<mailto:alopresto.apa...@gmail.com> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 On Jun 21, 2016, at 10:06 AM, Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: Hi Andy, I read several emails from the user mailing list archive that someone was going to write up the how to setup the IDE. However, I didn’t find any documents, so I was just wondering. Thanks Kumiko From: Andy LoPresto [mailto:alopre...@apache.org] Sent: Tuesday, June 21, 2016 10:03 AM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: Re: IDE-specific setup Kumiko, The Contributor Guide [1] has some information about configuring the NiFi checkstyle rules for IntelliJ IDEA and Eclipse. Are you looking for something specific? [1] https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-CodeStyle Andy LoPresto alopre...@apache.org<mailto:alopre...@apache.org> alopresto.apa...@gmail.com<mailto:alopresto.apa...@gmail.com> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 On Jun 21, 2016, at 9:57 AM, Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: Hello, Is there any documentations on how to setup the Nifi with IDE-specific setup? Thanks Kumiko
RE: IDE-specific setup
Hi Andy, I read several emails from the user mailing list archive that someone was going to write up the how to setup the IDE. However, I didn't find any documents, so I was just wondering. Thanks Kumiko From: Andy LoPresto [mailto:alopre...@apache.org] Sent: Tuesday, June 21, 2016 10:03 AM To: users@nifi.apache.org Subject: Re: IDE-specific setup Kumiko, The Contributor Guide [1] has some information about configuring the NiFi checkstyle rules for IntelliJ IDEA and Eclipse. Are you looking for something specific? [1] https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-CodeStyle Andy LoPresto alopre...@apache.org<mailto:alopre...@apache.org> alopresto.apa...@gmail.com<mailto:alopresto.apa...@gmail.com> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 On Jun 21, 2016, at 9:57 AM, Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: Hello, Is there any documentations on how to setup the Nifi with IDE-specific setup? Thanks Kumiko
IDE-specific setup
Hello, Is there any documentations on how to setup the Nifi with IDE-specific setup? Thanks Kumiko
RE: Custom Controller Service
Thank you, Mark! It worked as great per your suggestion.
Thanks
Kumiko
From: Mark Payne [mailto:marka...@hotmail.com]
Sent: Friday, June 17, 2016 12:26 PM
To: users@nifi.apache.org
Cc: Ki Kang <ki.k...@ds-iq.com>; Kevin Verhoeven <kevin.verhoe...@ds-iq.com>
Subject: Re: Custom Controller Service
Hi Kumiko,
I would recommend that in your OnEnabled method that you just create a
ScheduledExecutorService and
schedule the task to occur every 24 hours or however you'd like. Then, in your
OnDisabled method call
shutdown on that ScheduledExecutorService.
A word to the wise, though - NiFi tends to have quite a few threads running, so
for debugging purposes
it is easier to see what's going on if you provide a ThreadFactory to your
executor so that you can name
your threads appropriately. Something like:
Executors.newScheduledThreadPool(3, new ThreadFactory() {
@Override
public Thread newThread(Runnable r) {
final Thread t = Executors.defaultThreadFactory().newThread(r);
t.setDaemon(true);
t.setName("Refresh Azure DataLake Token");
return t;
}
});
I hope this helps!
Thanks
-Mark
On Jun 17, 2016, at 3:00 PM, Kumiko Yada
<kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote:
Hello,
I wrote the Custom Controller Service to authenticate the Azure Datalake Store,
and I’d like to refresh a token every 24 hours. What is the best way to
authenticate every 24 hours? Should I loop the authenticate code every 24
hours in OnEnabled?
Thanks
Kumiko
Custom Controller Service
Hello, I wrote the Custom Controller Service to authenticate the Azure Datalake Store, and I'd like to refresh a token every 24 hours. What is the best way to authenticate every 24 hours? Should I loop the authenticate code every 24 hours in OnEnabled? Thanks Kumiko
RE: Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl
I found out that I didn’t have specify this dependency in pom file that it would be packaged as part of below by finding the jar file in …\nifi-azure-dlstore-nar\target\classes\META-INF\bundled-dependencies. I verified that it is packaged with a new Nar, and I even copied this jar file in …\nifi-0.7.0-SNAPSHOT\lib folder, but I’m still getting the exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl. adx-snapshots Azure ADX Snapshots http://adxsnapshots.azurewebsites.net/ default true oss-snapshots Open Source Snapshots https://oss.sonatype.org/content/repositories/snapshots/ default true always com.microsoft.azure azure-client-authentication 1.0.0-SNAPSHOT com.microsoft.azure azure-mgmt-datalake-store 1.0.0-SNAPSHOT From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com] Sent: Thursday, June 9, 2016 5:11 PM To: users@nifi.apache.org Subject: Re: Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl Yes, it has to be available to your code otherwise you will continue to get ClassNotFoundException Oleg On Jun 9, 2016, at 8:06 PM, Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: Does this mean that I have to include this dependency in nar pom file now? com.squareup.retrofit2 retrofit 2.0.4 Thanks Kumiko From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com] Sent: Thursday, June 9, 2016 4:27 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: Re: Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl No, it just means that retrofit2.BaseUri is not in the NAR class path (see class not found at the bottom of the trace). Is it packaged with your new Nar? Sent from my iPhone On Jun 9, 2016, at 19:01, Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: Hello, My custom processor is throwing the run-time exception, so I reverted the changes that I made and I’m still getting this below error. Does this mean that there is some changes in Microsoft Azure SDK? My custom processor nar file which I compiled yesterday is working fine. Note: I’m getting this exception in single thread. 2016-06-09 15:50:46,718 WARN [Timer-Driven Process Thread-5] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding PutFileAzureDLStore[id=73299986-7f85-4114-8940-b77798757d3a] due to uncaught Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl 2016-06-09 15:50:46,719 WARN [Timer-Driven Process Thread-5] o.a.n.c.t.ContinuallyRunProcessorTask java.lang.NoClassDefFoundError: retrofit2/BaseUrl at nifi.azure.dlstore.processors.PutFileAzureDLStore.SetupClients(PutFileAzureDLStore.java:257) ~[na:na] at nifi.azure.dlstore.processors.PutFileAzureDLStore.onTrigger(PutFileAzureDLStore.java:210) ~[na:na] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057) ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] Caused by: java.lang.ClassNotFou
RE: Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl
Does this mean that I have to include this dependency in nar pom file now? com.squareup.retrofit2 retrofit 2.0.4 Thanks Kumiko From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com] Sent: Thursday, June 9, 2016 4:27 PM To: users@nifi.apache.org Subject: Re: Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl No, it just means that retrofit2.BaseUri is not in the NAR class path (see class not found at the bottom of the trace). Is it packaged with your new Nar? Sent from my iPhone On Jun 9, 2016, at 19:01, Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: Hello, My custom processor is throwing the run-time exception, so I reverted the changes that I made and I'm still getting this below error. Does this mean that there is some changes in Microsoft Azure SDK? My custom processor nar file which I compiled yesterday is working fine. Note: I'm getting this exception in single thread. 2016-06-09 15:50:46,718 WARN [Timer-Driven Process Thread-5] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding PutFileAzureDLStore[id=73299986-7f85-4114-8940-b77798757d3a] due to uncaught Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl 2016-06-09 15:50:46,719 WARN [Timer-Driven Process Thread-5] o.a.n.c.t.ContinuallyRunProcessorTask java.lang.NoClassDefFoundError: retrofit2/BaseUrl at nifi.azure.dlstore.processors.PutFileAzureDLStore.SetupClients(PutFileAzureDLStore.java:257) ~[na:na] at nifi.azure.dlstore.processors.PutFileAzureDLStore.onTrigger(PutFileAzureDLStore.java:210) ~[na:na] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057) ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] Caused by: java.lang.ClassNotFoundException: retrofit2.BaseUrl at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_77] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_77] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_77] Thanks Kumiko
Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl
Hello, My custom processor is throwing the run-time exception, so I reverted the changes that I made and I'm still getting this below error. Does this mean that there is some changes in Microsoft Azure SDK? My custom processor nar file which I compiled yesterday is working fine. Note: I'm getting this exception in single thread. 2016-06-09 15:50:46,718 WARN [Timer-Driven Process Thread-5] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding PutFileAzureDLStore[id=73299986-7f85-4114-8940-b77798757d3a] due to uncaught Exception: java.lang.NoClassDefFoundError: retrofit2/BaseUrl 2016-06-09 15:50:46,719 WARN [Timer-Driven Process Thread-5] o.a.n.c.t.ContinuallyRunProcessorTask java.lang.NoClassDefFoundError: retrofit2/BaseUrl at nifi.azure.dlstore.processors.PutFileAzureDLStore.SetupClients(PutFileAzureDLStore.java:257) ~[na:na] at nifi.azure.dlstore.processors.PutFileAzureDLStore.onTrigger(PutFileAzureDLStore.java:210) ~[na:na] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057) ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] Caused by: java.lang.ClassNotFoundException: retrofit2.BaseUrl at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_77] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_77] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_77] Thanks Kumiko
RE: Custom processor is failing for concurrency
Here is the code:
https://github.com/kyada1/PutFileAzureDLStore/blob/master/nifi-azure-dlstore-bundle/nifi-azure-dlstore-processors/src/main/java/nifi/azure/dlstore/processors/PutFileAzureDLStore.java
The problems are :
Line 209 - SetupClients(creds);
Line 217 - CreateFile(_path + _filename, value.get(), true);
To wrap the call in synchronized method, are these looked correct? Or for the
line 217, does the method have to return something?
For the line 209:
synchronized (this) {
SetupClients(creds);
if (creds!= null) {
return creds;
}
For the line 217:
synchronized {
CreateFile(_path + _filename, value.get(), true);
}
Thanks
Kumiko
From: Matt Foley [mailto:mfo...@hortonworks.com]
Sent: Thursday, June 9, 2016 12:27 PM
To: users@nifi.apache.org
Cc: Kevin Verhoeven <kevin.verhoe...@ds-iq.com>; Ki Kang <ki.k...@ds-iq.com>
Subject: Re: Custom processor is failing for concurrency
If there are multiple SDK calls that share the same problematic code (that is,
multiple SDK methods that would interact with each other in a non-thread-safe
way), then one must synchronize their calls to a shared lock object, which only
requires a couple more lines of code.
From: Oleg Zhurakousky
<ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>>
Sent: Thursday, June 09, 2016 12:15 PM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Cc: Kevin Verhoeven; Ki Kang
Subject: Re: Custom processor is failing for concurrency
+1, was just responding with the same.
On Jun 9, 2016, at 3:11 PM, Matt Foley
<mfo...@hortonworks.com<mailto:mfo...@hortonworks.com>> wrote:
Kumiko, would it be sufficient to just wrap your call to the non-thread-safe MS
SDK routine, in a 'synchronized' method? You could then use the standard NiFi
thread management and avoid a lot of complexity. And the result should be >=
efficiency of having a dedicated thread to manage the problem action.
--Matt F
____
From: Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>>
Sent: Thursday, June 09, 2016 11:49 AM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Cc: Kevin Verhoeven; Ki Kang
Subject: Re: Custom processor is failing for concurrency
Hi Bryan,
Does this mean that even I create the multiple threads in onTriger, I will
still hit the Microsoft SDK issue where it's not a thread safe? Sounds like
basically what I am trying to do and creating the multiple threads via UI
might be the same thing.
Thanks
Kumiko
From: Bryan Bende <bbe...@gmail.com<mailto:bbe...@gmail.com>>
Sent: Thursday, June 9, 2016 11:26:10 AM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Cc: Kevin Verhoeven; Ki Kang
Subject: Re: Custom processor is failing for concurrency
Kumiko,
In general you shouldn't have to create threads in your processors, with the
exception of some special cases.
The framework has a thread pool and it takes one of those threads and calls the
onTrigger method of your processor.
If you want multiple threads to call onTrigger, then each processor has a
Concurrent Tasks property in the UI on the scheduling tab,
which equates to the number of threads that will concurrently call onTrigger.
A processor developer needs to only worry about the business logic in the
onTrigger method, and needs to ensure
thread-safe access to any member variables or state stored in the processor.
Hope that helps.
-Bryan
On Thu, Jun 9, 2016 at 2:11 PM, Kumiko Yada
<kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote:
Microsoft found this is an issue with the SDK, they are working on a fix, they
do not have the ETA for the fix. To workaround this issue, I’m trying to
create the multiple threads in using AbstractSessionFactoryProcessor and handle
the Create a file in a single thread. I’m having a problem that the single
thread is not working correctly. The processor is still acting like a single
thread.
When I create a thread to handle the create a file, do I have to call this
method using java.util.concurrent.ExecutorService?
Are there any sample processors that I can take a look?
Thanks
Kumiko
From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>]
Sent: Sunday, June 5, 2016 6:28 PM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Cc: Ki Kang <ki.k...@ds-iq.com<mailto:ki.k...@ds-iq.com>>; Kevin Verhoeven
<kevin.verhoe...@ds-iq.com<mailto:kevin.verhoe...@ds-iq.com>>
Subject: RE: Custom processor is failing for concurrency
Thank you, Bryan. I’m working with Microsoft on this issue. Will keep you
guys updated.
Thanks
Kumiko
From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Friday, June 3, 2016 2:32 PM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Re: Custom processor is failing for concurrency
Hi Bryan,
Does this mean that even I create the multiple threads in onTriger, I will
still hit the Microsoft SDK issue where it's not a thread safe? Sounds like
basically what I am trying to do and creating the multiple threads via UI
might be the same thing.
Thanks
Kumiko
From: Bryan Bende <bbe...@gmail.com>
Sent: Thursday, June 9, 2016 11:26:10 AM
To: users@nifi.apache.org
Cc: Kevin Verhoeven; Ki Kang
Subject: Re: Custom processor is failing for concurrency
Kumiko,
In general you shouldn't have to create threads in your processors, with the
exception of some special cases.
The framework has a thread pool and it takes one of those threads and calls the
onTrigger method of your processor.
If you want multiple threads to call onTrigger, then each processor has a
Concurrent Tasks property in the UI on the scheduling tab,
which equates to the number of threads that will concurrently call onTrigger.
A processor developer needs to only worry about the business logic in the
onTrigger method, and needs to ensure
thread-safe access to any member variables or state stored in the processor.
Hope that helps.
-Bryan
On Thu, Jun 9, 2016 at 2:11 PM, Kumiko Yada
<kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote:
Microsoft found this is an issue with the SDK, they are working on a fix, they
do not have the ETA for the fix. To workaround this issue, I’m trying to
create the multiple threads in using AbstractSessionFactoryProcessor and handle
the Create a file in a single thread. I’m having a problem that the single
thread is not working correctly. The processor is still acting like a single
thread.
When I create a thread to handle the create a file, do I have to call this
method using java.util.concurrent.ExecutorService?
Are there any sample processors that I can take a look?
Thanks
Kumiko
From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>]
Sent: Sunday, June 5, 2016 6:28 PM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Cc: Ki Kang <ki.k...@ds-iq.com<mailto:ki.k...@ds-iq.com>>; Kevin Verhoeven
<kevin.verhoe...@ds-iq.com<mailto:kevin.verhoe...@ds-iq.com>>
Subject: RE: Custom processor is failing for concurrency
Thank you, Bryan. I’m working with Microsoft on this issue. Will keep you
guys updated.
Thanks
Kumiko
From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Friday, June 3, 2016 2:32 PM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: Custom processor is failing for concurrency
It is hard to say for sure, but I think your NiFi processor is generally ok
regarding thread safety, but I think there could be a problem in the Azure SDK
code...
RequestFactory has an instance of BaseUrl and every time
RequestFactory.create() is called, it calls BaseUrl.url().
The implementation of BaseUrl is the following (according to my IntelliJ
attaching the sources...):
public class AutoRestBaseUrl implements BaseUrl {
/** A template based URL with variables wrapped in {}s. */
private String template;
/** a mapping from {} wrapped variables in the template and their actual
values. */
private Map<CharSequence, String> mappings;
@Override
public HttpUrl url() {
String url = template;
for (Map.Entry<CharSequence, String> entry : mappings.entrySet()) {
url = url.replace(entry.getKey(), entry.getValue());
}
mappings.clear();
return HttpUrl.parse(url);
}
/**
* Creates an instance of a template based URL.
*
* @param url the template based URL to use.
*/
public AutoRestBaseUrl(String url) {
this.template = url;
this.mappings = new HashMap<>();
}
/**
* Sets the value for the {} wrapped variables in the template URL.
* @param matcher the {} wrapped variable to replace.
* @param value the value to set for the variable.
*/
public void set(CharSequence matcher, String value) {
this.mappings.put(matcher, value);
}
}
The exception is coming from the line where it is looping over the entryset:
for (Map.Entry<CharSequence, String> entry : mappings.entrySet()) {
Right after that loop it calls mappings.clear() so if the RequestFactory is
shared by multiple threads (which I think it is), then one thread could be
iterating over the set, which another calls mappings.clear().
On Fri, Jun 3, 2016 at 5:09 PM, Oleg Zhurakousky
<ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:
Kumiko
It appears that the current state of the source you linked in is not in sync
with what is in the stack trace. Perhaps you have made some code modifications
(e.g., line 218 is an empty line in code while it has a pointer in the star
trace).
In any event, from what I can see the error is coming from Azure libraries (not
NiFi). Specifically ‘com.microsoft.rest.AutoRestBaseUrl.url(..)’ seems to be
doing some iteration where I presume the remove is called. Perhaps it is not a
thread safe class after
RE: Custom processor is failing for concurrency
Microsoft found this is an issue with the SDK, they are working on a fix, they
do not have the ETA for the fix. To workaround this issue, I’m trying to
create the multiple threads in using AbstractSessionFactoryProcessor and handle
the Create a file in a single thread. I’m having a problem that the single
thread is not working correctly. The processor is still acting like a single
thread.
When I create a thread to handle the create a file, do I have to call this
method using java.util.concurrent.ExecutorService?
Are there any sample processors that I can take a look?
Thanks
Kumiko
From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com]
Sent: Sunday, June 5, 2016 6:28 PM
To: users@nifi.apache.org
Cc: Ki Kang <ki.k...@ds-iq.com>; Kevin Verhoeven <kevin.verhoe...@ds-iq.com>
Subject: RE: Custom processor is failing for concurrency
Thank you, Bryan. I’m working with Microsoft on this issue. Will keep you
guys updated.
Thanks
Kumiko
From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Friday, June 3, 2016 2:32 PM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: Custom processor is failing for concurrency
It is hard to say for sure, but I think your NiFi processor is generally ok
regarding thread safety, but I think there could be a problem in the Azure SDK
code...
RequestFactory has an instance of BaseUrl and every time
RequestFactory.create() is called, it calls BaseUrl.url().
The implementation of BaseUrl is the following (according to my IntelliJ
attaching the sources...):
public class AutoRestBaseUrl implements BaseUrl {
/** A template based URL with variables wrapped in {}s. */
private String template;
/** a mapping from {} wrapped variables in the template and their actual
values. */
private Map<CharSequence, String> mappings;
@Override
public HttpUrl url() {
String url = template;
for (Map.Entry<CharSequence, String> entry : mappings.entrySet()) {
url = url.replace(entry.getKey(), entry.getValue());
}
mappings.clear();
return HttpUrl.parse(url);
}
/**
* Creates an instance of a template based URL.
*
* @param url the template based URL to use.
*/
public AutoRestBaseUrl(String url) {
this.template = url;
this.mappings = new HashMap<>();
}
/**
* Sets the value for the {} wrapped variables in the template URL.
* @param matcher the {} wrapped variable to replace.
* @param value the value to set for the variable.
*/
public void set(CharSequence matcher, String value) {
this.mappings.put(matcher, value);
}
}
The exception is coming from the line where it is looping over the entryset:
for (Map.Entry<CharSequence, String> entry : mappings.entrySet()) {
Right after that loop it calls mappings.clear() so if the RequestFactory is
shared by multiple threads (which I think it is), then one thread could be
iterating over the set, which another calls mappings.clear().
On Fri, Jun 3, 2016 at 5:09 PM, Oleg Zhurakousky
<ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:
Kumiko
It appears that the current state of the source you linked in is not in sync
with what is in the stack trace. Perhaps you have made some code modifications
(e.g., line 218 is an empty line in code while it has a pointer in the star
trace).
In any event, from what I can see the error is coming from Azure libraries (not
NiFi). Specifically ‘com.microsoft.rest.AutoRestBaseUrl.url(..)’ seems to be
doing some iteration where I presume the remove is called. Perhaps it is not a
thread safe class after all. What does Microsoft documentation says? Have you
looked at the source to see what’s going on there? If its open please link and
we can tale a look.
Cheers
Oleg
On Jun 3, 2016, at 4:58 PM, Kumiko Yada
<kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote:
Here is the code, https://github.com/kyada1/PutFileAzureDLStore.
Thanks
Kumiko
From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Friday, June 3, 2016 12:57 PM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: Custom processor is failing for the custom processor
Hello,
Would you be able to share your code for PutFileAzureDLStore so we can help
identify if there is a concurrency problem?
-Bryan
On Fri, Jun 3, 2016 at 3:39 PM, Kumiko Yada
<kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote:
Hello,
I wrote the following custom service control and processor. When the custom
processor is running concurrently, it’s failing often with several different
errors. Are there any special handlings for concurrently that I need to add in
the custom processor? I wrote the sample Java program which does the same
thing as the custom processor (authenticate every time the file is
created/create a file, create 2 threads and run concurrently), it’s working
fine. The custom processor also fine when this is not running concurrently.
Custom service control – set the properties for the Microsoft Azure Da
RE: Custom processor is failing for concurrency
Thank you, Bryan. I’m working with Microsoft on this issue. Will keep you
guys updated.
Thanks
Kumiko
From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Friday, June 3, 2016 2:32 PM
To: users@nifi.apache.org
Subject: Re: Custom processor is failing for concurrency
It is hard to say for sure, but I think your NiFi processor is generally ok
regarding thread safety, but I think there could be a problem in the Azure SDK
code...
RequestFactory has an instance of BaseUrl and every time
RequestFactory.create() is called, it calls BaseUrl.url().
The implementation of BaseUrl is the following (according to my IntelliJ
attaching the sources...):
public class AutoRestBaseUrl implements BaseUrl {
/** A template based URL with variables wrapped in {}s. */
private String template;
/** a mapping from {} wrapped variables in the template and their actual
values. */
private Map<CharSequence, String> mappings;
@Override
public HttpUrl url() {
String url = template;
for (Map.Entry<CharSequence, String> entry : mappings.entrySet()) {
url = url.replace(entry.getKey(), entry.getValue());
}
mappings.clear();
return HttpUrl.parse(url);
}
/**
* Creates an instance of a template based URL.
*
* @param url the template based URL to use.
*/
public AutoRestBaseUrl(String url) {
this.template = url;
this.mappings = new HashMap<>();
}
/**
* Sets the value for the {} wrapped variables in the template URL.
* @param matcher the {} wrapped variable to replace.
* @param value the value to set for the variable.
*/
public void set(CharSequence matcher, String value) {
this.mappings.put(matcher, value);
}
}
The exception is coming from the line where it is looping over the entryset:
for (Map.Entry<CharSequence, String> entry : mappings.entrySet()) {
Right after that loop it calls mappings.clear() so if the RequestFactory is
shared by multiple threads (which I think it is), then one thread could be
iterating over the set, which another calls mappings.clear().
On Fri, Jun 3, 2016 at 5:09 PM, Oleg Zhurakousky
<ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:
Kumiko
It appears that the current state of the source you linked in is not in sync
with what is in the stack trace. Perhaps you have made some code modifications
(e.g., line 218 is an empty line in code while it has a pointer in the star
trace).
In any event, from what I can see the error is coming from Azure libraries (not
NiFi). Specifically ‘com.microsoft.rest.AutoRestBaseUrl.url(..)’ seems to be
doing some iteration where I presume the remove is called. Perhaps it is not a
thread safe class after all. What does Microsoft documentation says? Have you
looked at the source to see what’s going on there? If its open please link and
we can tale a look.
Cheers
Oleg
On Jun 3, 2016, at 4:58 PM, Kumiko Yada
<kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote:
Here is the code, https://github.com/kyada1/PutFileAzureDLStore.
Thanks
Kumiko
From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Friday, June 3, 2016 12:57 PM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: Custom processor is failing for the custom processor
Hello,
Would you be able to share your code for PutFileAzureDLStore so we can help
identify if there is a concurrency problem?
-Bryan
On Fri, Jun 3, 2016 at 3:39 PM, Kumiko Yada
<kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote:
Hello,
I wrote the following custom service control and processor. When the custom
processor is running concurrently, it’s failing often with several different
errors. Are there any special handlings for concurrently that I need to add in
the custom processor? I wrote the sample Java program which does the same
thing as the custom processor (authenticate every time the file is
created/create a file, create 2 threads and run concurrently), it’s working
fine. The custom processor also fine when this is not running concurrently.
Custom service control – set the properties for the Microsoft Azure Datalake
Store
Custom processor – authenticate, then create a file in Microsoft Azure Datalake
Store
Error1:
2016-06-03 12:29:31,942 INFO [pool-2815-thread-1]
c.m.aad.adal4j.AuthenticationAuthority [Correlation ID:
64c89876-2f7b-4fbb-8e6f-fb395a47aa87] Instance discovery was successful
2016-06-03 12:29:31,946 ERROR [Timer-Driven Process Thread-10]
n.a.d.processors.PutFileAzureDLStore
java.util.ConcurrentModificationException: null
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
~[na:1.8.0_77]
at java.util.HashMap$EntryIterator.next(HashMap.java:1463)
~[na:1.8.0_77]
at java.util.HashMap$EntryIterator.next(HashMap.java:1461)
~[na:1.8.0_77]
at
com.microsoft.rest.AutoRestBaseUrl.url(AutoRestBaseUrl.java:28) ~[na:na]
at retrofit2.RequestFactory.create(RequestFactory.java:50)
~[na:na]
RE: Custom processor is failing for concurrency
Here is the callstacks with the latest changes. 2016-06-03 14:32:32,766 WARN [Timer-Driven Process Thread-4] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding PutFileAzureDLStore[id=2133fdad-54b7-46a5-9bcf-958ddd8cfac0] due to uncaught Exception: com.microsoft.rest.ServiceException: Invalid status code 403 2016-06-03 14:32:32,768 WARN [Timer-Driven Process Thread-4] o.a.n.c.t.ContinuallyRunProcessorTask com.microsoft.rest.ServiceException: Invalid status code 403 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_77] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_77] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_77] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_77] at com.microsoft.rest.ServiceResponseBuilder.build(ServiceResponseBuilder.java:147) ~[na:na] at com.microsoft.azure.management.datalake.store.FileSystemOperationsImpl.createDelegate(FileSystemOperationsImpl.java:1491) ~[na:na] at com.microsoft.azure.management.datalake.store.FileSystemOperationsImpl.create(FileSystemOperationsImpl.java:1432) ~[na:na] at nifi.azure.dlstore.processors.PutFileAzureDLStore.CreateFile(PutFileAzureDLStore.java:252) ~[na:na] at nifi.azure.dlstore.processors.PutFileAzureDLStore.onTrigger(PutFileAzureDLStore.java:217) ~[na:na] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057) ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] Here is the library codes: https://github.com/Azure/autorest-clientruntime-for-java/blob/master/client-runtime/src/main/java/com/microsoft/rest/AutoRestBaseUrl.java https://github.com/Azure https://github.com/Azure/azure-sdk-for-java Thanks Kumiko From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com] Sent: Friday, June 3, 2016 2:10 PM To: users@nifi.apache.org Subject: Re: Custom processor is failing for concurrency Kumiko It appears that the current state of the source you linked in is not in sync with what is in the stack trace. Perhaps you have made some code modifications (e.g., line 218 is an empty line in code while it has a pointer in the star trace). In any event, from what I can see the error is coming from Azure libraries (not NiFi). Specifically ‘com.microsoft.rest.AutoRestBaseUrl.url(..)’ seems to be doing some iteration where I presume the remove is called. Perhaps it is not a thread safe class after all. What does Microsoft documentation says? Have you looked at the source to see what’s going on there? If its open please link and we can tale a look. Cheers Oleg On Jun 3, 2016, at 4:58 PM, Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: Here is the code, https://github.com/kyada1/PutFileAzureDLStore. Thanks Kumiko From: Bryan Bende [mailto:bbe...@gmail.com] Sent: Friday, June 3, 2016 12:57 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: Re: Custom processor is failing for the custom processor Hello, Would you be able to share your code for PutFileAzureDLStore so we can help identify if there is a concurrency problem? -Bryan On Fri, Jun 3, 2016 at 3:39 PM, Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: Hello, I wrot
Re: How to access the counter and provenance Info
Is there a way to query all the rows in provenance? This does't have to be in processors. Thanks Kumiko From: Kumiko Yada <kumiko.y...@ds-iq.com> Sent: Thursday, June 2, 2016 9:27:46 AM To: users@nifi.apache.org Subject: Re: How to access the counter and provenance Info Hi Brian, Would the AtomicInteger counter reset when the custom processor is stopped and restarted? Is there a plan to add the ability to query the counter or provenance from processors? Thanks Kumiko From: Bryan Bende <bbe...@gmail.com> Sent: Thursday, June 2, 2016 7:55:15 AM To: users@nifi.apache.org Subject: Re: How to access the counter and provenance Info Hi Kumiko, I think you can only increment counters and report provenance events from processors, but not query counters or query provenance. If you are in a custom processor, could have an AtomicInteger counter that increments on every onTrigger and reset when 24 hours has passed? Thanks, Bryan On Thu, Jun 2, 2016 at 10:50 AM, Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: Could anyone help me on this? Thanks Kumiko ________ From: Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> Sent: Wednesday, June 1, 2016 4:22:25 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: RE: How to access the counter and provenance Info When I clicked the Data Provenance from context menu of processor, the item history are showed the NiFi Flow Data Provenance UI. How can I get this item list from the custom processor? Thanks Kumiko From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>] Sent: Wednesday, June 1, 2016 12:56 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: How to access the counter and provenance Info Hello, I’d like to get the how many times custom processor run past 24 hours in onTrigger method. How can I get this using counter or provenance Info? Thanks Kumiko
Re: How to access the counter and provenance Info
Hi Brian, Would the AtomicInteger counter reset when the custom processor is stopped and restarted? Is there a plan to add the ability to query the counter or provenance from processors? Thanks Kumiko From: Bryan Bende <bbe...@gmail.com> Sent: Thursday, June 2, 2016 7:55:15 AM To: users@nifi.apache.org Subject: Re: How to access the counter and provenance Info Hi Kumiko, I think you can only increment counters and report provenance events from processors, but not query counters or query provenance. If you are in a custom processor, could have an AtomicInteger counter that increments on every onTrigger and reset when 24 hours has passed? Thanks, Bryan On Thu, Jun 2, 2016 at 10:50 AM, Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: Could anyone help me on this? Thanks Kumiko ____ From: Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> Sent: Wednesday, June 1, 2016 4:22:25 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: RE: How to access the counter and provenance Info When I clicked the Data Provenance from context menu of processor, the item history are showed the NiFi Flow Data Provenance UI. How can I get this item list from the custom processor? Thanks Kumiko From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>] Sent: Wednesday, June 1, 2016 12:56 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: How to access the counter and provenance Info Hello, I'd like to get the how many times custom processor run past 24 hours in onTrigger method. How can I get this using counter or provenance Info? Thanks Kumiko
Re: How to access the counter and provenance Info
Thank you, Joe.
Could you let me know how I can get the lists of provenance data?
Thanks
Kumiko
From: Joe Witt <joe.w...@gmail.com>
Sent: Thursday, June 2, 2016 7:55:29 AM
To: users@nifi.apache.org
Subject: Re: How to access the counter and provenance Info
Hello Kumiko
The ProcessSession API exposes to you the ability to adjust counters [1].
You would simply call that in your on trigger method with something like
session.adjustCounter("mycounter",1L,true);
[1]
https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/processor/ProcessSession.java#L123
Thanks
On Thu, Jun 2, 2016 at 7:50 AM, Kumiko Yada <kumiko.y...@ds-iq.com> wrote:
> Could anyone help me on this?
>
> Thanks
> Kumiko
> ____
> From: Kumiko Yada <kumiko.y...@ds-iq.com>
> Sent: Wednesday, June 1, 2016 4:22:25 PM
> To: users@nifi.apache.org
> Subject: RE: How to access the counter and provenance Info
>
>
> When I clicked the Data Provenance from context menu of processor, the item
> history are showed the NiFi Flow Data Provenance UI. How can I get this
> item list from the custom processor?
>
>
>
> Thanks
>
> Kumiko
>
>
>
> From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com]
> Sent: Wednesday, June 1, 2016 12:56 PM
> To: users@nifi.apache.org
> Subject: How to access the counter and provenance Info
>
>
>
> Hello,
>
>
>
> I’d like to get the how many times custom processor run past 24 hours in
> onTrigger method. How can I get this using counter or provenance Info?
>
>
>
> Thanks
>
> Kumiko
Re: How to access the counter and provenance Info
Could anyone help me on this? Thanks Kumiko From: Kumiko Yada <kumiko.y...@ds-iq.com> Sent: Wednesday, June 1, 2016 4:22:25 PM To: users@nifi.apache.org Subject: RE: How to access the counter and provenance Info When I clicked the Data Provenance from context menu of processor, the item history are showed the NiFi Flow Data Provenance UI. How can I get this item list from the custom processor? Thanks Kumiko From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com] Sent: Wednesday, June 1, 2016 12:56 PM To: users@nifi.apache.org Subject: How to access the counter and provenance Info Hello, I'd like to get the how many times custom processor run past 24 hours in onTrigger method. How can I get this using counter or provenance Info? Thanks Kumiko
RE: How to access the counter and provenance Info
When I clicked the Data Provenance from context menu of processor, the item history are showed the NiFi Flow Data Provenance UI. How can I get this item list from the custom processor? Thanks Kumiko From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com] Sent: Wednesday, June 1, 2016 12:56 PM To: users@nifi.apache.org Subject: How to access the counter and provenance Info Hello, I'd like to get the how many times custom processor run past 24 hours in onTrigger method. How can I get this using counter or provenance Info? Thanks Kumiko
Best way to process the process the requests in batch
Hello,
We implemented the custom process that are similar to the InvokeHTTP that the
part of URL can be replace with the Context Data List, then write the weather
to the flowfile. For example, URL to get the weather feed have to include the
zip code in URL, and the ZIP code is {0} in the URL and replaced the zip code
from the Context Data List property.
URL
http://example{0}/weather
Context Data List:
0
1
2
Processor with make the following requests:
http://example{0}/weather
http://example0/weather
http://example1/weather
http://example2/weather
This processor is processed in one request at a time and have a perf issue.
I'd like to modify to process in batches. What are the best way to process in
batches? And also, would the Nifi keep track how many requests the processor
is processed? If so, how the Nifi keep track this and how long the Nifi keep
track of data? I'd like to add the quota priorities in this processor to keep
track of quota. For example, if the weather feeds can be requested only 100
requests a day, I don't want to processor to executed once the quota is reached.
Thanks
Kumiko
RE: Dependency for SSL
That did it, thank you Matt! From: Matt Burgess [mailto:mattyb...@gmail.com] Sent: Wednesday, May 25, 2016 3:28 PM To: users@nifi.apache.org Subject: Re: Dependency for SSL If that library has dependencies, you may need to remove the jar so that it brings in the POM, which should get the JAR and its dependencies. Regards, Matt On May 25, 2016, at 5:38 PM, Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: Thank you Bryan and Matt. I added the dependency in the pom of processor jar & nar project, and I’m getting the ClassNotFoundException: com.squareup.okhttp.Authenticator error when the Nifi is started. I added the below dependency in pom of processor jar. Do I need to add something else to correctly handle this dependency? Pom of the processors jar project: com.burgstaller okhttp-digest 0.6 jar Thanks Kumiko From: Bryan Bende [mailto:bbe...@gmail.com] Sent: Wednesday, May 25, 2016 1:16 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: Re: Dependency for SSL Hello, This Wiki page shows how to setup the dependencies to use the SSLContextService from a custom processor bundle: https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices Thanks, Bryan On Wed, May 25, 2016 at 3:44 PM, Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: Hello, I’d like to use the following in the custom processor in Nifi 0.6.1, and I added the dependency. However, I’m getting the following errors. Do I need to add something else to correctly handle this dependency? import org.apache.nifi.ssl.SSLContextService; import org.apache.nifi.ssl.SSLContextService.ClientAuth; org.apache.nifi nifi-ssl-context-service-api Error: at java.util.ServiceLoader.fail(ServiceLoader.java:232) ~[na:1.8.0_77] at java.util.ServiceLoader.access$100(ServiceLoader.java:185) ~[na:1.8.0_77] at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) ~[na:1.8.0_77] at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[na:1.8.0_77] at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[na:1.8.0_77] at org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:107) ~[nifi-nar-utils-0.6.1.jar:0.6.1] at org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:88) ~[nifi-nar-utils-0.6.1.jar:0.6.1] at org.apache.nifi.NiFi.(NiFi.java:120) ~[nifi-runtime-0.6.1.jar:0.6.1] at org.apache.nifi.NiFi.main(NiFi.java:227) ~[nifi-runtime-0.6.1.jar:0.6.1] Caused by: java.lang.NoClassDefFoundError: org/apache/nifi/ssl/SSLContextService at nifi.processors.http.looper.InvokeHTTPLooper.(InvokeHTTPLooper.java:200) ~[nifi-http.looper-processors-1.0.jar:1.0] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_77] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_77] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_77] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_77] at java.lang.Class.newInstance(Class.java:442) ~[na:1.8.0_77] at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ~[na:1.8.0_77] ... 6 common frames omitted Caused by: java.lang.ClassNotFoundException: org.apache.nifi.ssl.SSLContextService at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_77] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_77] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_77] Thanks Kumiko
Dependencies for StandardValidators & SoftLimitBoundedByteArrayOutputStream
Hello, I'd like to use the followings in my custom processor, and I'm still getting the build errors that they do not exist. Am I able to use them by additional dependencies, if so what are the missing dependencies or do I need to port the codes? import org.apache.nifi.processor.util.StandardValidators; import org.apache.nifi.processors.standard.util.SoftLimitBoundedByteArrayOutputStream; org.apache.nifi nifi-processor-utils org.apache.nifi nifi-standard-processors Thanks Kumiko
Dependency for SSL
Hello, I'd like to use the following in the custom processor in Nifi 0.6.1, and I added the dependency. However, I'm getting the following errors. Do I need to add something else to correctly handle this dependency? import org.apache.nifi.ssl.SSLContextService; import org.apache.nifi.ssl.SSLContextService.ClientAuth; org.apache.nifi nifi-ssl-context-service-api Error: at java.util.ServiceLoader.fail(ServiceLoader.java:232) ~[na:1.8.0_77] at java.util.ServiceLoader.access$100(ServiceLoader.java:185) ~[na:1.8.0_77] at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) ~[na:1.8.0_77] at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[na:1.8.0_77] at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[na:1.8.0_77] at org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:107) ~[nifi-nar-utils-0.6.1.jar:0.6.1] at org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:88) ~[nifi-nar-utils-0.6.1.jar:0.6.1] at org.apache.nifi.NiFi.(NiFi.java:120) ~[nifi-runtime-0.6.1.jar:0.6.1] at org.apache.nifi.NiFi.main(NiFi.java:227) ~[nifi-runtime-0.6.1.jar:0.6.1] Caused by: java.lang.NoClassDefFoundError: org/apache/nifi/ssl/SSLContextService at nifi.processors.http.looper.InvokeHTTPLooper.(InvokeHTTPLooper.java:200) ~[nifi-http.looper-processors-1.0.jar:1.0] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_77] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_77] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_77] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_77] at java.lang.Class.newInstance(Class.java:442) ~[na:1.8.0_77] at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ~[na:1.8.0_77] ... 6 common frames omitted Caused by: java.lang.ClassNotFoundException: org.apache.nifi.ssl.SSLContextService at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_77] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_77] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_77] Thanks Kumiko
RE: Read the custom attributes
Thank you James, it’s working now ☺
-Kumiko
From: James Wing [mailto:jvw...@gmail.com]
Sent: Tuesday, May 24, 2016 4:04 PM
To: users@nifi.apache.org
Subject: Re: Read the custom attributes
Expression language support must be configured in the PropertyDescriptor for
all properties, including dynamic properties. Your processor should override
getSupportedDynamicPropertyDescriptor() to provide dynamic properties. Take a
look at the UpdateAttribute processor for an example:
@Override
protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String
propertyDescriptorName) {
return new PropertyDescriptor.Builder()
.name(propertyDescriptorName)
.required(false)
.addValidator(StandardValidators.createAttributeExpressionLanguageValidator(AttributeExpression.ResultType.STRING,
true))
.addValidator(StandardValidators.ATTRIBUTE_KEY_PROPERTY_NAME_VALIDATOR)
.expressionLanguageSupported(true)
.dynamic(true)
.build();
}
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-processor/src/main/java/org/apache/nifi/processors/attributes/UpdateAttribute.java#L194
On Tue, May 24, 2016 at 3:43 PM, Kumiko Yada
<kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote:
Hello,
When the Nifi expression language is used in the custom processor property,
it’s working. However, I’m not able get from expression language for the
custom attributes. How can I enable to read the custom attributes?
Thanks
Kumiko
Read the custom attributes
Hello, When the Nifi expression language is used in the custom processor property, it's working. However, I'm not able get from expression language for the custom attributes. How can I enable to read the custom attributes? Thanks Kumiko
RE: Exception: .java.sql.SQLException: No suitable driver found for jdbc:sqlserver:
The driver jar is located in the program folder, and I added the path; however, it was not picking up correctly as you suggested. I added the driver jar in the lib dir, and it’s working now. Thank you, Pierre! Thanks Kumiko From: Pierre Villard [mailto:pierre.villard...@gmail.com] Sent: Tuesday, May 24, 2016 9:47 AM To: users@nifi.apache.org Subject: Re: Exception: .java.sql.SQLException: No suitable driver found for jdbc:sqlserver: Hi, Where did you place your driver jar? I think the issue is here. You could try to add the driver jar in the /lib directory of NiFi to check if it helps. Pierre 2016-05-24 18:18 GMT+02:00 Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>>: Hello, I’m writing the custom processor to execute the store procedure against Azure SQL Data Warehouse, and I wrote the test code and it’s working. However, when I executed this code as the custom processor, I’m getting an exception that java.sql.SQLException: No suitable driver found for jdbc:sqlserver://xxx, database.windows.net:1433... Since the connection string is working in the test program, I don’t think this is the connection string, and I’m using the sqljdbc42.jar. Do you have any idea why this jar is not used when the custom processor is executed? Thanks Kumiko
Exception: .java.sql.SQLException: No suitable driver found for jdbc:sqlserver:
Hello, I’m writing the custom processor to execute the store procedure against Azure SQL Data Warehouse, and I wrote the test code and it’s working. However, when I executed this code as the custom processor, I’m getting an exception that java.sql.SQLException: No suitable driver found for jdbc:sqlserver://xxx, database.windows.net:1433... Since the connection string is working in the test program, I don’t think this is the connection string, and I’m using the sqljdbc42.jar. Do you have any idea why this jar is not used when the custom processor is executed? Thanks Kumiko
RE: Build error for the custom processor against 0.7.0-SNAPSHOT
Hi Andrew, I used the cmd “mvn archetype:generate” to generate the archetype. Yes, I locally build the 0.7.0-SNAPSHOT. Was the 0.7.0-SNAPSHOT deleted recently from Maven repository? This was working fine for me until last Friday. Thanks Kumiko From: Andrew Psaltis [mailto:psaltis.and...@gmail.com] Sent: Tuesday, May 10, 2016 12:04 PM To: users@nifi.apache.org Cc: Ki Kang <ki.k...@ds-iq.com> Subject: Re: Build error for the custom processor against 0.7.0-SNAPSHOT Hi Kumiko, I am not sure of the exact maven command you were using to generate the archetype. However when looking at the Maven repository here (https://repo1.maven.org/maven2/org/apache/nifi/nifi-maven-archetypes/) I do not see the 0.7.0-SNAPSHOT. I may be mistaken, but I think the most recent version is actually 0.6.1. Did you by chance build the maven archetypes locally? Thanks, Andrew On Tue, May 10, 2016 at 2:48 PM, Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: Hello, I’m getting the following build error against 0.7.0-SNAPSHOT. I also tried to create a new custom processor again for 0.7.0-SNAPSHOT and getting the same error. Is this a known issue? Build error: [INFO] Scanning for projects... [ERROR] [ERROR] Some problems were encountered while processing the POMs: [FATAL] Non-resolvable parent POM for :processors:1.0: Could not find artifact org.apache.nifi:nifi-nar-bundles:pom:0.7.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 19, column 13 @ Creating new processor error: [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 15.235 s [INFO] Finished at: 2016-05-10T11:42:03-07:00 [INFO] Final Memory: 13M/219M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:2.4:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.nifi:nifi-processor-bundle-archetype:0.7.0-SNAPSHOT) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException Thanks Kumiko -- Thanks, Andrew Subscribe to my book: Streaming Data<http://manning.com/psaltis> [https://static.licdn.com/scds/common/u/img/webpromo/btn_viewmy_160x25.png]<https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> twiiter: @itmdata<http://twitter.com/intent/user?screen_name=itmdata>
Build error for the custom processor against 0.7.0-SNAPSHOT
Hello, I'm getting the following build error against 0.7.0-SNAPSHOT. I also tried to create a new custom processor again for 0.7.0-SNAPSHOT and getting the same error. Is this a known issue? Build error: [INFO] Scanning for projects... [ERROR] [ERROR] Some problems were encountered while processing the POMs: [FATAL] Non-resolvable parent POM for :processors:1.0: Could not find artifact org.apache.nifi:nifi-nar-bundles:pom:0.7.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 19, column 13 @ Creating new processor error: [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 15.235 s [INFO] Finished at: 2016-05-10T11:42:03-07:00 [INFO] Final Memory: 13M/219M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:2.4:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.nifi:nifi-processor-bundle-archetype:0.7.0-SNAPSHOT) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException Thanks Kumiko
RE: Cannot Authenticate the Azure Datalake Store
HI Bryan, Here are the dependencies: org.apache.nifi nifi-api org.apache.nifi nifi-processor-utils org.apache.nifi nifi-mock test org.slf4j slf4j-simple test junit junit 4.11 test com.microsoft.azure azure-client-authentication 1.0.0-SNAPSHOT com.microsoft.azure azure-mgmt-datalake-store 1.0.0-SNAPSHOT org.slf4j slf4j-api 1.7.5 org.slf4j slf4j-simple 1.6.4 Thanks Kumiko From: Bryan Bende [mailto:bbe...@gmail.com] Sent: Friday, May 6, 2016 7:04 AM To: users@nifi.apache.org Cc: Ki Kang <ki.k...@ds-iq.com> Subject: Re: Cannot Authenticate the Azure Datalake Store Hello, It seems like maybe a wrong version of a library is being used, or maybe a JAR is missing that needs to be included in your NAR. Can you share what dependencies you have in the pom.xml of your processors project? You can check under NIFI_HOME/work/nar/extensions//META-INF/bundled-dependencies/ to see the JARs that are being included in your NAR. -Bryan On Fri, May 6, 2016 at 12:52 AM, Kumiko Yada <kumiko.y...@ds-iq.com<mailto:kumiko.y...@ds-iq.com>> wrote: Hello, I’m writing the custom processor to create a file in the Azure Datalake Store. I wrote the sample java program to do using this sample code, https://azure.microsoft.com/en-us/documentation/articles/data-lake-store-get-started-java-sdk/. This is working fine. However, when I wrote the code in the custom processor, it’s failing if the Authenticate code with this this error when this custom processor is running. I didn’t get any build error when I built this custom processor. Do you have any ideas why I’m getting this error? Is there any file I need to copy to the nifi folder other than nar file for the custom processor? 2016-05-05 20:37:37,338 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] failed to process due to java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method ()V not found; rolling back session: java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method ()V not found 2016-05-05 20:37:37,342 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@3dcbc96c<mailto:org.apache.nifi.controller.FlowController@3dcbc96c> // Another save pending = false 2016-05-05 20:37:37,342 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method ()V not found at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.(DataLakeStoreAccountManagementClientImpl.java:176) ~[na:na] at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.(DataLakeStoreAccountManagementClientImpl.java:166) ~[na:na] at dsiq.processors.custom.CreateFileAzureDatalakeStore.SetupClients(CreateFileAzureDatalakeStore.java:286) ~[na:na] at dsiq.processors.custom.CreateFileAzureDatalakeStore.onTrigger(CreateFileAzureDatalakeStore.java:269) ~[na:na] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$Schedu
Cannot Authenticate the Azure Datalake Store
Hello, I'm writing the custom processor to create a file in the Azure Datalake Store. I wrote the sample java program to do using this sample code, https://azure.microsoft.com/en-us/documentation/articles/data-lake-store-get-started-java-sdk/. This is working fine. However, when I wrote the code in the custom processor, it's failing if the Authenticate code with this this error when this custom processor is running. I didn't get any build error when I built this custom processor. Do you have any ideas why I'm getting this error? Is there any file I need to copy to the nifi folder other than nar file for the custom processor? 2016-05-05 20:37:37,338 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] failed to process due to java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method ()V not found; rolling back session: java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method ()V not found 2016-05-05 20:37:37,342 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@3dcbc96c // Another save pending = false 2016-05-05 20:37:37,342 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method ()V not found at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.(DataLakeStoreAccountManagementClientImpl.java:176) ~[na:na] at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.(DataLakeStoreAccountManagementClientImpl.java:166) ~[na:na] at dsiq.processors.custom.CreateFileAzureDatalakeStore.SetupClients(CreateFileAzureDatalakeStore.java:286) ~[na:na] at dsiq.processors.custom.CreateFileAzureDatalakeStore.onTrigger(CreateFileAzureDatalakeStore.java:269) ~[na:na] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1057) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] 2016-05-05 20:37:37,346 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] CreateFileAzureDatalakeStore[id=f89e5860-1d99-4d20-be71-769f6bb775c7] failed to process session due to java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method ()V not found: java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method ()V not found 2016-05-05 20:37:37,347 ERROR [Timer-Driven Process Thread-4] d.p.custom.CreateFileAzureDatalakeStore java.lang.NoSuchMethodError: com.microsoft.azure.AzureServiceClient: method ()V not found at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.(DataLakeStoreAccountManagementClientImpl.java:176) ~[na:na] at com.microsoft.azure.management.datalake.store.DataLakeStoreAccountManagementClientImpl.(DataLakeStoreAccountManagementClientImpl.java:166) ~[na:na] at dsiq.processors.custom.CreateFileAzureDatalakeStore.SetupClients(CreateFileAzureDatalakeStore.java:286)
RE: Apache NiFi - WebHDFS
We would need the WebHDFS processor for Microsoft Azure Data Lake store. Thanks Kumiko -Original Message- From: Jeremy Dyer [mailto:jdy...@gmail.com] Sent: Thursday, April 21, 2016 4:28 PM To: users@nifi.apache.org Subject: Re: Apache NiFi - WebHDFS Kumiko, Just curious what makes you want to use WebHDFS over the existing HDFS processors? Are you looking for a feature they don't contain or just looking for something a little lighter without all of the configuration files? Sent from my iPhone > On Apr 21, 2016, at 6:11 PM, Tom Stewart <stewartthom...@yahoo.com> wrote: > > What about Knox Gateway? > >> On Apr 21, 2016, at 3:21 PM, Kumiko Yada <kumiko.y...@ds-iq.com> wrote: >> >> Will do. >> >> Thanks >> Kumiko >> >> -Original Message- >> From: Joe Witt [mailto:joe.w...@gmail.com] >> Sent: Thursday, April 21, 2016 12:45 PM >> To: users@nifi.apache.org >> Subject: Re: Apache NiFi - WebHDFS >> >> Kumiko, >> >> Not that I am aware of. If you do end up doing so and are interested in >> contributing please let us know. >> >> Thanks >> Joe >> >>> On Thu, Apr 21, 2016 at 3:43 PM, Kumiko Yada <kumiko.y...@ds-iq.com> wrote: >>> Hello, >>> >>> >>> >>> Has anyone written the custom process for WebHDFS? >>> >>> >>> >>> Thanks >>> >>> Kumiko >
RE: Apache NiFi - WebHDFS
Will do. Thanks Kumiko -Original Message- From: Joe Witt [mailto:joe.w...@gmail.com] Sent: Thursday, April 21, 2016 12:45 PM To: users@nifi.apache.org Subject: Re: Apache NiFi - WebHDFS Kumiko, Not that I am aware of. If you do end up doing so and are interested in contributing please let us know. Thanks Joe On Thu, Apr 21, 2016 at 3:43 PM, Kumiko Yada <kumiko.y...@ds-iq.com> wrote: > Hello, > > > > Has anyone written the custom process for WebHDFS? > > > > Thanks > > Kumiko
Apache NiFi - WebHDFS
Hello, Has anyone written the custom process for WebHDFS? Thanks Kumiko
