Re: Executing Stored Procedure only once

[Anil Rai]

Thanks Matt. The reason to split is due to the file size.
We will try removing split record and try. If not follow the other options
you have mentioned.

Thanks Again
Anil


On Wed, Jun 6, 2018 at 5:07 PM, Matt Burgess  wrote:

> Anil,
>
> Are you using SplitRecord because something times out in
> PutDatabaseRecord otherwise? If not, you shouldn't need the
> SplitRecord, and without it, things should work the way you expect.
>
> If you do need to split into smaller flow files, you should be able to
> use Wait/Notify with the parent flow file UUID as the identifier and
> the signal count equal to the number of rows divided by the number of
> Records Per Split in the SplitRecord processor. If you won't know the
> value of signal count (10 in your example), you might be able to use
> MergeRecord (with a Defragment strategy) after PutDatabaseRecord to
> put them back together, then you route that to ExecuteSQL. That
> involves unnecessary merging of 1 GB of content only to get a single
> flow file as a trigger, so that would be the least attractive of the
> options.
>
> Regards,
> Matt
>
>
> On Wed, Jun 6, 2018 at 3:54 PM, Anil Rai  wrote:
> > Hi All,
> >
> >
> > Usecase - Fetch very very large csv file (1 GB). Insert into a DB and
> then
> > execute a stored procedure
> >
> > Implementation -> ListSFTP -> FetchSFTP -> SplitRecord (splitting into
> > smaller files)-> PutDatabaseRecord -> ExecuteSQL
> >
> > Problem -> SplitRecord does not tell me how many files it is generating.
> > Assuming it generates 10 files, ExecuteSQL executes 10 times since
> > PutDatabaseRecords executes 10 time
> >
> > We would like to execute ExecuteSQL only once at the end of the flow ones
> > all of PutDatabaseRecord operations are complete.
> >
> > How do we implement this?
> >
> > Thanks
> > Anil
>

Executing Stored Procedure only once

[Anil Rai]

Hi All,


Usecase - Fetch very very large csv file (1 GB). Insert into a DB and then
execute a stored procedure

Implementation -> ListSFTP -> FetchSFTP -> SplitRecord (splitting into
smaller files)-> PutDatabaseRecord -> ExecuteSQL

Problem -> SplitRecord does not tell me how many files it is generating.
Assuming it generates 10 files, ExecuteSQL executes 10 times since
PutDatabaseRecords executes 10 time

We would like to execute ExecuteSQL only once at the end of the flow ones
all of PutDatabaseRecord operations are complete.

How do we implement this?

Thanks
Anil

Re: oAuth for API stood up on nifi

[Anil Rai]

Thanks Bryan.

On Mon, Jun 4, 2018 at 2:34 PM, Bryan Bende  wrote:

> Hello,
>
> Processors can't leverage NiFi's internal authentication and
> authorization mechanisms.
>
> For HandleHttpRequest it supports two-way TLS for client authentication.
>
> -Bryan
>
>
> On Mon, Jun 4, 2018 at 2:11 PM, Anil Rai  wrote:
> > Team, for invoking the nifi API's on a secured cluster, we have to get
> the
> > token and then invoke.
> > Now if I stand up a new API via HandleHttpRequest processor, is there a
> way
> > to use the inbuilt oAuth to access this API?
> >
> > Regards
> > Anil
>

oAuth for API stood up on nifi

[Anil Rai]

Team, for invoking the nifi API's on a secured cluster, we have to get the
token and then invoke.
Now if I stand up a new API via HandleHttpRequest processor, is there a way
to use the inbuilt oAuth to access this API?

Regards
Anil

Re: Nifi Docker cluster issue

[Anil Rai]

Team, I did not hear back on this.
Here are some additional scenario's.
We have a 1 node cluster. We have a flow. When we run that flow, we get the
below error. After which we see the following message in canvas - Action
cannot be performed because there is currently no Cluster Coordinator
elected. The request should be tried again after a moment, after a Cluster
Coordinator has been automatically elected.

Error in the log : 2018-05-22 11:03:46,272 ERROR [Timer-Driven Process
Thread-1] o.a.n.p.standard.HandleHttpResponse
HandleHttpResponse[id=11c43ac0-b502-11f7-82ec-caedaada5452] Failed to
respond to HTTP request for
StandardFlowFileRecord[uuid=be5f42c4-496b-4086-ad7d-bdedb8c87313,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1526985786635-1032,
container=default, section=8], offset=505044,
length=8],offset=0,name=9546340083575618,size=8] because FlowFile had an
'http.context.identifier' attribute of f2df150e-5c72-4560-a78d-15d8e20f316f
but could not find an HTTP Response Object for this identifier

For the below scenario, the problem was that we imported a template that
had reference to other nifi registery (I know this was fixed recently).
We have a 3 node nifi docker cluster running on a VM. Frequently we see one
of the nodes in the cluster stops sending heartbeat. We do not see any
errors in the logs for that container. But when we check the status of
Nifi, it shows it is running. Since the heart beat stops, the node is
disconnected from the cluster.The canvas is not usable at this point till
we go to the cluster menu and explicitly remove the offending node.
Then if we restart the nifi instance, it comes up and joins the cluster.
We are not sure if this is a docker issue or nifi 1.5 issue. On top, we do
not see any error when the heart beat stops from one of the nodes. Any
suggestions on how to trouble shoot this further?

Sounds like the moment flow file gets corrupted, the cluster becomes
unusable?

Regards
Anil


On Tue, May 15, 2018 at 3:47 PM, Anil Rai <anilrain...@gmail.com> wrote:

> Team,
>
> We have a 3 node nifi docker cluster running on a VM. Frequently we see
> one of the nodes in the cluster stops sending heartbeat. We do not see any
> errors in the logs for that container. But when we check the status of
> Nifi, it shows it is running. Since the heart beat stops, the node is
> disconnected from the cluster.The canvas is not usable at this point till
> we go to the cluster menu and explicitly remove the offending node.
> Then if we restart the nifi instance, it comes up and joins the cluster.
> We are not sure if this is a docker issue or nifi 1.5 issue. On top, we do
> not see any error when the heart beat stops from one of the nodes. Any
> suggestions on how to trouble shoot this further?
>
> Thanks
> Anil
>
>

Re: User and Policies

[Anil Rai]

Thanks Bryan. It's just that the canvas looks very cluttered when we have a
lot of process groups. The fact that I cannot do anything with other
process groups, I was wondering if we can not shown them.


On Mon, May 14, 2018 at 4:31 PM, Bryan Bende <bbe...@gmail.com> wrote:

> There intentionally isn't a way to hide components from the canvas.
> You can use the analogy of being in an apartment building and seeing
> all the doors... you can only see whats inside the one you have the
> key to, but you still know all the other locked doors are there.
>
>
> On Mon, May 14, 2018 at 4:10 PM, Anil Rai <anilrain...@gmail.com> wrote:
> > On the same topic, we currently have many teams sharing our common
> > development environment. We have created groups for each team and added
> > users to the groups.
> > Each team is given a process group and this process group is assigned to
> > that specific group. So that they can only work on their process group
> and
> > do not mess with others.
> > It works fine.
> > Only problem on the canvas is that while I cannot mess with other process
> > groups, I still see other process groups in dotted lines and grayed out.
> Is
> > there a way to hide the other process groups from the canvas that I do
> not
> > have permissions to?
> >
> > Thanks
> > Anil
> >
> >
> >
> > On Mon, May 14, 2018 at 3:10 PM, Anil Rai <anilrain...@gmail.com> wrote:
> >
> >> Thanks for the detailed explanation Bryan.
> >>
> >> Cheers
> >> Anil
> >>
> >>
> >> On Mon, May 14, 2018 at 3:01 PM, Bryan Bende <bbe...@gmail.com> wrote:
> >>
> >>> Hello,
> >>>
> >>> When a node joins the clusters, if the node has an empty flow.xml, no
> >>> users, and no authorizations, then the node will inherit all of those
> >>> from the cluster, but if any of those are populated then it won't be
> >>> able to join.
> >>>
> >>> One common issue that prevents this from working, is if you have an
> >>> initial admin populated on the new node, then it starts up and creates
> >>> the initial admin and policies locally, and when it attempts to join
> >>> the cluster it will not inherit because now technically it has a
> >>> user/policies defined which are in conflict with the cluster. So the
> >>> trick for a new node is to not fill in the initial admin in
> >>> authorizers.xml.
> >>>
> >>> Regarding creating users/groups/policies... anything you can do in the
> >>> UI can be done via the REST API, the best way to take a look is to
> >>> open Chrome Dev tools while you use the UI and checkout the calls made
> >>> in the network tab.
> >>>
> >>> You could write a script to perform any of these calls and automate
> >>> the creation.
> >>>
> >>> Thanks,
> >>>
> >>> Bryan
> >>>
> >>>
> >>> On Mon, May 14, 2018 at 2:22 PM, Anil Rai <anilrain...@gmail.com>
> wrote:
> >>> > All,
> >>> >
> >>> > We noticed that we cannot add/modify users and policies when 1 node
> in a
> >>> > cluster is down. So seems like all nodes should have the latest and
> >>> > identical users.xml and auth*.xml. Is this correct? Shouldn't the
> latest
> >>> > and up to date files be copied to other nodes during startup instead?
> >>> (like
> >>> > flow.gz)
> >>> >
> >>> > Secondly, we configures groups, users and policies manually via the
> >>> canvas.
> >>> > Is there any automated way? and how do we propagate the users to
> other
> >>> > environments?
> >>> >
> >>> > Thanks
> >>> > Anil
> >>>
> >>
> >>
>

Re: User and Policies

[Anil Rai]

On the same topic, we currently have many teams sharing our common
development environment. We have created groups for each team and added
users to the groups.
Each team is given a process group and this process group is assigned to
that specific group. So that they can only work on their process group and
do not mess with others.
It works fine.
Only problem on the canvas is that while I cannot mess with other process
groups, I still see other process groups in dotted lines and grayed out. Is
there a way to hide the other process groups from the canvas that I do not
have permissions to?

Thanks
Anil



On Mon, May 14, 2018 at 3:10 PM, Anil Rai <anilrain...@gmail.com> wrote:

> Thanks for the detailed explanation Bryan.
>
> Cheers
> Anil
>
>
> On Mon, May 14, 2018 at 3:01 PM, Bryan Bende <bbe...@gmail.com> wrote:
>
>> Hello,
>>
>> When a node joins the clusters, if the node has an empty flow.xml, no
>> users, and no authorizations, then the node will inherit all of those
>> from the cluster, but if any of those are populated then it won't be
>> able to join.
>>
>> One common issue that prevents this from working, is if you have an
>> initial admin populated on the new node, then it starts up and creates
>> the initial admin and policies locally, and when it attempts to join
>> the cluster it will not inherit because now technically it has a
>> user/policies defined which are in conflict with the cluster. So the
>> trick for a new node is to not fill in the initial admin in
>> authorizers.xml.
>>
>> Regarding creating users/groups/policies... anything you can do in the
>> UI can be done via the REST API, the best way to take a look is to
>> open Chrome Dev tools while you use the UI and checkout the calls made
>> in the network tab.
>>
>> You could write a script to perform any of these calls and automate
>> the creation.
>>
>> Thanks,
>>
>> Bryan
>>
>>
>> On Mon, May 14, 2018 at 2:22 PM, Anil Rai <anilrain...@gmail.com> wrote:
>> > All,
>> >
>> > We noticed that we cannot add/modify users and policies when 1 node in a
>> > cluster is down. So seems like all nodes should have the latest and
>> > identical users.xml and auth*.xml. Is this correct? Shouldn't the latest
>> > and up to date files be copied to other nodes during startup instead?
>> (like
>> > flow.gz)
>> >
>> > Secondly, we configures groups, users and policies manually via the
>> canvas.
>> > Is there any automated way? and how do we propagate the users to other
>> > environments?
>> >
>> > Thanks
>> > Anil
>>
>
>

Re: User and Policies

[Anil Rai]

Thanks for the detailed explanation Bryan.

Cheers
Anil


On Mon, May 14, 2018 at 3:01 PM, Bryan Bende <bbe...@gmail.com> wrote:

> Hello,
>
> When a node joins the clusters, if the node has an empty flow.xml, no
> users, and no authorizations, then the node will inherit all of those
> from the cluster, but if any of those are populated then it won't be
> able to join.
>
> One common issue that prevents this from working, is if you have an
> initial admin populated on the new node, then it starts up and creates
> the initial admin and policies locally, and when it attempts to join
> the cluster it will not inherit because now technically it has a
> user/policies defined which are in conflict with the cluster. So the
> trick for a new node is to not fill in the initial admin in
> authorizers.xml.
>
> Regarding creating users/groups/policies... anything you can do in the
> UI can be done via the REST API, the best way to take a look is to
> open Chrome Dev tools while you use the UI and checkout the calls made
> in the network tab.
>
> You could write a script to perform any of these calls and automate
> the creation.
>
> Thanks,
>
> Bryan
>
>
> On Mon, May 14, 2018 at 2:22 PM, Anil Rai <anilrain...@gmail.com> wrote:
> > All,
> >
> > We noticed that we cannot add/modify users and policies when 1 node in a
> > cluster is down. So seems like all nodes should have the latest and
> > identical users.xml and auth*.xml. Is this correct? Shouldn't the latest
> > and up to date files be copied to other nodes during startup instead?
> (like
> > flow.gz)
> >
> > Secondly, we configures groups, users and policies manually via the
> canvas.
> > Is there any automated way? and how do we propagate the users to other
> > environments?
> >
> > Thanks
> > Anil
>

User and Policies

[Anil Rai]

All,

We noticed that we cannot add/modify users and policies when 1 node in a
cluster is down. So seems like all nodes should have the latest and
identical users.xml and auth*.xml. Is this correct? Shouldn't the latest
and up to date files be copied to other nodes during startup instead? (like
flow.gz)

Secondly, we configures groups, users and policies manually via the canvas.
Is there any automated way? and how do we propagate the users to other
environments?

Thanks
Anil

Re: JSON Path Expression

[Anil Rai]

Thanks Otto.
I did try changing the json path to : $.fields[?(@.name==\"Type\")].maskType
I am still getting [] output.
I am I missing something?


On Fri, Apr 27, 2018 at 4:32 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> For the record, I am not saying that the json path library version nifi
> uses isn’t broken because this doesn’t work.
>
>
>
> On April 27, 2018 at 16:30:57, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> The reason why your path is failing is because of the embedded quotes.
> If you remove the quotes it will work.
> I wrote this test:
>
>
> @Test
> public void testIssue() throws Exception{
> String jsonPathAttrKey = "JsonPath1";
> final TestRunner testRunner = TestRunners.newTestRunner(new
> EvaluateJsonPath());
> testRunner.setProperty(EvaluateJsonPath.DESTINATION,
> EvaluateJsonPath.DESTINATION_CONTENT);
> testRunner.setProperty("JsonPath1", "$.fields[?(@.name==Type)].
> maskType");
>
> testRunner.enqueue(JSON_FAIL_SNIPPET);
> testRunner.run();
>
> Relationship expectedRel = EvaluateJsonPath.REL_MATCH;
>
> testRunner.assertAllFlowFilesTransferred(expectedRel, 1);
> final MockFlowFile out =
> testRunner.getFlowFilesForRelationship(expectedRel).get(0);
> out.assertContentEquals("[\"b\"]");
> }
>
>
> Here, JSON_FAIL_SNIPPET is your content as a file.
>
> I understand that your original path works on the test web sites.  It
> *also* works if you call JsonPath differently:
>
> /**{ "fields":[
>  {
>  "maskType": "a",
>  "name": "Id"
>  },
>  {
>  "maskType": "b",
>  "name": "Type"
>  }
>  ]
>  }*/
>  @Multiline
> static String NIFI_PROB;
>
>  @Test
>  public void testNifi() {
>App app = new App();
>List value = app.justJsonPath(NIFI_PROB.getBytes(),
> "$.fields[?(@.name==\"Type\")].maskType");
>if( value != null ) {
>  value.forEach(new Consumer() {
>@Override
>public void accept(Object o) {
>  System.out.println(o.toString());
>}
>  });
>}
>  }
>
> private final ObjectMapper objectMapper = new ObjectMapper();
> public App() {
>   Configuration.setDefaults(new Configuration.Defaults() {
>
> private final JsonProvider jsonProvider = new JacksonJsonProvider();
> private final MappingProvider mappingProvider = new
> JacksonMappingProvider();
>
> @Override
> public JsonProvider jsonProvider() {
>   return jsonProvider;
> }
>
> @Override
> public MappingProvider mappingProvider() {
>   return mappingProvider;
> }
>
> @Override
> public Set options() {
>   return EnumSet.noneOf(Option.class);
> }
>   });
>   CacheProvider.setCache(new LRUCache(100));
> }
>
> public List justJsonPath(byte[] rawMessage, String jsonPath) {
>   return JsonPath.parse(new String(rawMessage))
>   .read(jsonPath);
> }
>
> b
>
> Process finished with exit code 0
>
> But the way the Nifi uses JsonPath you need to call it as such.
>
> Hope this helps.
>
> ottO
>
>
> On April 27, 2018 at 16:09:32, Mike Thomsen (mikerthom...@gmail.com)
> wrote:
>
> The jayway JSONPath library on GitHub is the library I was referring to.
> I'll check on Jira to see if there's a ticket. Seems like something we
> could do for 1.7
>
> Thanks,
>
> Mike
>
> On Fri, Apr 27, 2018 at 3:46 PM Anil Rai <anilrain...@gmail.com> wrote:
>
> > Hi Mike,
> > Thanks for your quick reply. I am using nifi 1.5.0 release. I am not sure
> > what you mean by "trying against the latest version of java lib?"
> > If you could further elaborate please?
> > Also is there any other way to achieve this outcome (either by changing
> the
> > path or by using a completely different processor other than
> > evaluatejsonpath?)
> >
> > Thanks
> > Anil
> >
> >
> > On Fri, Apr 27, 2018 at 3:37 PM, Mike Thomsen <mikerthom...@gmail.com>
> > wrote:
> >
> > > The JsonPath processor uses an older version of the Java library. I
> think
> > > it's v2.0. Have you tried that against the latest version of the Java
> > lib?
> > > I know there's at least one JsonPath feature the version we use doesn't
> > > support.
> > >
> > > On Fri, Apr 27, 2018 at 3:14 PM Anil Rai <anilrain...@gmail.com>
> wrote:
> > >
> > > > Experts,
> > > >
> > > > Input JSON
> > > > ---
> > > > { "fields":[
> > > > {
> > > > "maskType": "a",
> > > > "name": "Id"
> > > > },
> > > > {
> > > > "maskType": "b",
> > > > "name": "Type"
> > > > }
> > > > ]
> > > > }
> > > > ---
> > > >
> > > > JsonPath : $.fields[?(@.name=="Type")].maskType
> > > >
> > > > --
> > > >
> > > > Expected Output
> > > > ---
> > > > [
> > > > "b"
> > > > ]
> > > > 
> > > >
> > > > But EvaluateJsonPath processor is giving me []
> > > >
> > > > Any idea?
> > > >
> > > > Thanks
> > > > Anil
> > > >
> > >
> >
>

Re: JSON Path Expression

[Anil Rai]

Hi Mike,
Thanks for your quick reply. I am using nifi 1.5.0 release. I am not sure
what you mean by "trying against the latest version of java lib?"
If you could further elaborate please?
Also is there any other way to achieve this outcome (either by changing the
path or by using a completely different processor other than
evaluatejsonpath?)

Thanks
Anil


On Fri, Apr 27, 2018 at 3:37 PM, Mike Thomsen <mikerthom...@gmail.com>
wrote:

> The JsonPath processor uses an older version of the Java library. I think
> it's v2.0. Have you tried that against the latest version of the Java lib?
> I know there's at least one JsonPath feature the version we use doesn't
> support.
>
> On Fri, Apr 27, 2018 at 3:14 PM Anil Rai <anilrain...@gmail.com> wrote:
>
> > Experts,
> >
> > Input JSON
> > ---
> > { "fields":[
> > {
> > "maskType": "a",
> > "name": "Id"
> > },
> > {
> > "maskType": "b",
> > "name": "Type"
> > }
> > ]
> > }
> > ---
> >
> > JsonPath : $.fields[?(@.name=="Type")].maskType
> >
> > --
> >
> > Expected Output
> > ---
> > [
> > "b"
> > ]
> > 
> >
> > But EvaluateJsonPath processor is giving me []
> >
> > Any idea?
> >
> > Thanks
> > Anil
> >
>

JSON Path Expression

[Anil Rai]

Experts,

Input JSON
---
{ "fields":[
{
"maskType": "a",
"name": "Id"
},
{
"maskType": "b",
"name": "Type"
}
]
}
---

JsonPath : $.fields[?(@.name=="Type")].maskType

--

Expected Output
---
[
"b"
]


But EvaluateJsonPath processor is giving me []

Any idea?

Thanks
Anil

Re: Clustering not happening in Kubernetes cluster

[Anil Rai]

Hi Mark,

We are also looking at spinning up nifi on k8s. If you could share more
information on your architecture of Nifi on k8s, that would be very
helpful. Also any best practice that you would suggest.
I am interested in understanding the auto scaling of nifi on k8s. Before I
deep dive, wanted to check if you have explored the best way to scale up or
down and what parameters would qualify in triggering that action.

Thanks
Anil


On Mon, Apr 16, 2018 at 1:36 PM, Mark Payne  wrote:

> Jonathan,
>
> I've spent some time in the last few weeks playing around with getting
> some clusters running on Kubernetes
> as well. I know from the mailing lists that some others have been
> venturing into this also. I'm by no means
> a Kubernetes expert myself, but I've gotten it up & running on a few-node
> cluster in GCE, so I do have some
> familiarity.
>
> One thing to note is that the log file that you shared appears to be from
> an instance that's been running for a while, not
> one that was newly started, so it's possible that there were some
> important details regarding why the clustering is not
> working as expected at the beginning of the logs that were missed.
>
> I also notice that you're setting the "nifi.web.http.host" and
> "nifi.cluster.node.address" properties to "nifi-test".
> I'm not sure that this will work. Perhaps it will, I've never tried that,
> but I don't know if the NiFi process will be able to listen
> on that hostname. What I ended up doing was to update my docker image
> locally to update the "hostname" variable to be
> the fully-qualified hostname:
>
> export hostname=$(hostname -f)
>
> And then set the http.host and node.address properties to that. If setting
> to nifi-test works, though, then that would be great,
> certainly a lot easier. But I suspect we'll need to take a closer look at
> the logs just after startup to see if they reveal anything
> interesting.
>
> Thanks
> -Mark
>
>
> On Apr 16, 2018, at 12:26 PM, Jonathan Kosgei  mailto:jonat...@saharacluster.com>> wrote:
>
> Hi,
>
>
>
> I'm trying to cluster nifi on kubernetes but haven't been able to get the
> pods to connect to each other.
>
>
>
> I have the following statefulset and services;
>
> https://gist.github.com/jonathan-kosgei/cdf7c9ec882948eac12beb6b28ffa748
>
>
>
> I'm running nifi 1.6.0 on Kubernetes v1.8.8-gke.0.
>
>
>
> My nifi-app.log:
>
> https://gist.github.com/jonathan-kosgei/527952976ec18cf3957cec4d1e186f68
>
>
>
> I set every log option to DEBUG in logback.xml
>
>
>
> I've noticed a lot of restarts and several files in the logs folder
>
>
>
> nifi@nifi2-0:/opt/nifi/nifi-1.6.0/logs$ ls
>
> nifi-app.log  nifi-app_2018-04-16_03.0.log
> nifi-app_2018-04-16_06.0.log  nifi-app_2018-04-16_09.0.log
> nifi-app_2018-04-16_12.0.log  nifi-app_2018-04-16_15.0.log
>
> nifi-app_2018-04-16_01.0.log  nifi-app_2018-04-16_04.0.log
> nifi-app_2018-04-16_07.0.log  nifi-app_2018-04-16_10.0.log
> nifi-app_2018-04-16_13.0.log  nifi-bootstrap.log
>
> nifi-app_2018-04-16_02.0.log  nifi-app_2018-04-16_05.0.log
> nifi-app_2018-04-16_08.0.log  nifi-app_2018-04-16_11.0.log
> nifi-app_2018-04-16_14.0.log  nifi-user.log
>
>
>
> Any ideas what I might be missing?
>
>
>
> The cluster configuration seemed very straightforward.
>
>
>
> Thanks!
>
>
>

Re: Data migration in sequence

[Anil Rai]

Thanks Mike and Joe for the quick response.
Hi Mike, Nifi is making api calls to source -> pull data -> Maps -> Post
data to target via API. Generate flow file triggers the flow. Right now i
manually run 1 process group after the previous one is complete.
Hi Joe, The systems are the same. A and B. Just that i will be pulling
different data set and that has to be done in sequence. Example, I will
load customer followed by say order followed by items etc.
The use case is one time data load from system A and B of different data
sets that needs to happen in sequence. Trigger has to be only once to start
the first one.
Hope that helps.

Regards
Anil


On Thu, Apr 12, 2018 at 12:26 PM, Joe Witt <joe.w...@gmail.com> wrote:

> Anil
>
> What is a little hard to tell here is do you mean:
>
> 1) I have to transfer data from system A to system B where there are
> 10 steps I want to execute in NiFi between A and B.
> -or-
> 2) I have to transfer data from systems A,B,C,D,E,F,G,H,I,J to system
> Z where first I want to load data from system A to Z, then B to Z,
> then C to Z and so on.
>
> If it is #1 then of course it is easy.
> If it is #2 then I believe you'd want a series of Wait/Notify
> processors where B waits the notification signal from A, C from B, and
> so on.
>
> Thanks
>
> On Thu, Apr 12, 2018 at 4:16 PM, Mike Thomsen <mikerthom...@gmail.com>
> wrote:
> > It'd be very helpful to know what the input and output processors are for
> > each of the flows because they have different capabilities for triggering
> > them.
> >
> > On Thu, Apr 12, 2018 at 12:03 PM Anil Rai <anilrain...@gmail.com> wrote:
> >
> >> sorry, sent email in between.
> >> This can be achieved by putting few flags in cache and checking them in
> >> subsequent flows. But this is not elegant.
> >> Any other way to achieve this flow runs in a sequence?
> >>
> >> Thanks
> >> Anil
> >>
> >>
> >> On Thu, Apr 12, 2018 at 12:01 PM, Anil Rai <anilrain...@gmail.com>
> wrote:
> >>
> >> > Experts, Below is my requirenment.
> >> >
> >> > I have to do a 1 time data load from system a to b.
> >> > I have to transfer 10 different data sets.
> >> > These transfers have to happen in sequence. 1 followed by 2 followed
> by 3
> >> > so on till 10.
> >> >
> >> > We have created seperate process groups for each of the data set
> >> migration.
> >> > How can we achieve the run sequence once I trigger the first flow?
> >> > I know this can be achieved by putting few flags in cache and checking
> >> > them in subsequen
> >> >
> >>
>

Data migration in sequence

[Anil Rai]

Experts, Below is my requirenment.

I have to do a 1 time data load from system a to b.
I have to transfer 10 different data sets.
These transfers have to happen in sequence. 1 followed by 2 followed by 3
so on till 10.

We have created seperate process groups for each of the data set migration.
How can we achieve the run sequence once I trigger the first flow?
I know this can be achieved by putting few flags in cache and checking them
in subsequen

Re: setting up secure nifi

[Anil Rai]

The below links does not work. Have they moved somewhere else?

https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
https://blog.rosander.ninja/nifi/toolkit/tls/2016/09/19/tls-toolkit-intro.html
https://blog.rosander.ninja/nifi/toolkit/tls/2016/
09/20/tls-toolkit-standalone-multi.html

Thanks
Anil

On Thu, Feb 1, 2018 at 10:35 AM, Anil Rai <anilrain...@gmail.com> wrote:

> Thanks Andy. It did resolve my issue. I got it working.
> Thanks again for all the links. Very helpful.
>
> Cheers
> Anil
>
>
> On Thu, Feb 1, 2018 at 10:14 AM, Andy LoPresto <alopre...@apache.org>
> wrote:
>
>> Hi Anil,
>>
>> In addition to Bryan’s explanation, there are a number of blog posts and
>> articles covering this topic:
>>
>> * Authorization and Multi-Tenancy by Bryan Bende [1]
>> * Secured Cluster Setup by Pierre Villard [2]
>> * TLS Generation Toolkit section of Apache NiFi Admin Guide [3]
>> * Initial Admin Identity section of Apache NiFi Admin Guide [4]
>> * Apache NiFi TLS Toolkit single node standalone by Bryan Rosander [5]
>> * Apache NiFi TLS Toolkit multi-node standalone in Docker by Bryan
>> Rosander [6]
>>
>> The sequence “dc=example,dc=com” in your current user DN (Distinguished
>> Name) is incorrect and not present in the DN of the certificate. I imagine
>> you copied this from an example posted online. “dc=“ is a sequence used in
>> DNS to indicate “Domain Component” [7]. In your case, “CN=TC,OU=NIFI” is
>> the RDN (Relative Distinguished Name) of your user, and “dc=example,dc=com”
>> would be the parent DN. But when you generated the certificate, you did not
>> provide this information, so the DNs do not match, and NiFi correctly
>> asserts that this is not a valid certificate identifying the user DN you
>> specified in your XML files. Removing “dc=example,dc=com” from that
>> definition as Bryan suggested will resolve your issue.
>>
>> [1] https://bryanbende.com/development/2016/08/17/apache-nif
>> i-1-0-0-authorization-and-multi-tenancy
>> [2] https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-s
>> ecured-cluster-setup/
>> [3] https://nifi.apache.org/docs/nifi-docs/html/administrati
>> on-guide.html#tls-generation-toolkit
>> [4] https://nifi.apache.org/docs/nifi-docs/html/administrati
>> on-guide.html#initial-admin-identity
>> [5] https://blog.rosander.ninja/nifi/toolkit/tls/2016/09/19/
>> tls-toolkit-intro.html
>> [6] https://blog.rosander.ninja/nifi/toolkit/tls/2016/09/20/
>> tls-toolkit-standalone-multi.html
>> [7] https://en.wikipedia.org/wiki/Lightweight_Directory_Acce
>> ss_Protocol#Directory_structure
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Jan 31, 2018, at 7:32 PM, Bryan Bende <bbe...@gmail.com> wrote:
>>
>> It’s the same problem, your initial admin should be:
>>
>> CN=TC, OU=NIFI
>>
>> Not
>>
>> CN=TC,OU=NIFI,dc=example,dc=com
>>
>> The first one is the DN of your client cert, the second one is not.
>>
>> On Wed, Jan 31, 2018 at 7:23 PM Anil Rai <anilrain...@gmail.com> wrote:
>>
>> Hi Bryan,
>>
>> Thanks for the quick reply. I did followed your steps. But I am seeing the
>> same error.
>> Now the entry looks like
>>CN=TC,OU=NIFI,dc=example,
>> dc=com
>>
>> Also what does dc stand for after CN and OU. Is that a problem.
>> Is there a blog that talks about installing and making it https using
>> toolkit?. I did not find any good post that talks end to end from
>> installing to making it secure using tls toolkit.
>>
>> Any help is appreciated.
>>
>> Thanks
>> Anil
>>
>>
>>
>> On Wed, Jan 31, 2018 at 6:42 PM, Bryan Bende <bbe...@gmail.com> wrote:
>>
>> Hello,
>>
>> The identity in authorizers.xml for your initial admin does not match the
>> identity of your client cert.
>>
>> You should be putting “CN=TC, OU=NIFI” as the initial admin because that
>>
>> is
>>
>> the DN of your client cert.
>>
>> You’ll need to stop NiFi, edit authorizers.xml, delete users.xml and
>> authorizations.xml, and start back up.
>>
>> Thanks,
>>
>> Bryan
>>
>> On Wed, Jan 31, 2018 at 6:11 PM Anil Rai <anilrain...@gmail.com> wrote:
>>
>> All,
>>
>> I am trying to install nifi 1.5 and making it https. Below is the steps
>> followed and the error

Re: setting up secure nifi

[Anil Rai]

Thanks Andy. It did resolve my issue. I got it working.
Thanks again for all the links. Very helpful.

Cheers
Anil


On Thu, Feb 1, 2018 at 10:14 AM, Andy LoPresto <alopre...@apache.org> wrote:

> Hi Anil,
>
> In addition to Bryan’s explanation, there are a number of blog posts and
> articles covering this topic:
>
> * Authorization and Multi-Tenancy by Bryan Bende [1]
> * Secured Cluster Setup by Pierre Villard [2]
> * TLS Generation Toolkit section of Apache NiFi Admin Guide [3]
> * Initial Admin Identity section of Apache NiFi Admin Guide [4]
> * Apache NiFi TLS Toolkit single node standalone by Bryan Rosander [5]
> * Apache NiFi TLS Toolkit multi-node standalone in Docker by Bryan
> Rosander [6]
>
> The sequence “dc=example,dc=com” in your current user DN (Distinguished
> Name) is incorrect and not present in the DN of the certificate. I imagine
> you copied this from an example posted online. “dc=“ is a sequence used in
> DNS to indicate “Domain Component” [7]. In your case, “CN=TC,OU=NIFI” is
> the RDN (Relative Distinguished Name) of your user, and “dc=example,dc=com”
> would be the parent DN. But when you generated the certificate, you did not
> provide this information, so the DNs do not match, and NiFi correctly
> asserts that this is not a valid certificate identifying the user DN you
> specified in your XML files. Removing “dc=example,dc=com” from that
> definition as Bryan suggested will resolve your issue.
>
> [1] https://bryanbende.com/development/2016/08/17/apache-
> nifi-1-0-0-authorization-and-multi-tenancy
> [2] https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-
> secured-cluster-setup/
> [3] https://nifi.apache.org/docs/nifi-docs/html/
> administration-guide.html#tls-generation-toolkit
> [4] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#
> initial-admin-identity
> [5] https://blog.rosander.ninja/nifi/toolkit/tls/2016/
> 09/19/tls-toolkit-intro.html
> [6] https://blog.rosander.ninja/nifi/toolkit/tls/2016/
> 09/20/tls-toolkit-standalone-multi.html
> [7] https://en.wikipedia.org/wiki/Lightweight_Directory_
> Access_Protocol#Directory_structure
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jan 31, 2018, at 7:32 PM, Bryan Bende <bbe...@gmail.com> wrote:
>
> It’s the same problem, your initial admin should be:
>
> CN=TC, OU=NIFI
>
> Not
>
> CN=TC,OU=NIFI,dc=example,dc=com
>
> The first one is the DN of your client cert, the second one is not.
>
> On Wed, Jan 31, 2018 at 7:23 PM Anil Rai <anilrain...@gmail.com> wrote:
>
> Hi Bryan,
>
> Thanks for the quick reply. I did followed your steps. But I am seeing the
> same error.
> Now the entry looks like
>CN=TC,OU=NIFI,dc=example,
> dc=com
>
> Also what does dc stand for after CN and OU. Is that a problem.
> Is there a blog that talks about installing and making it https using
> toolkit?. I did not find any good post that talks end to end from
> installing to making it secure using tls toolkit.
>
> Any help is appreciated.
>
> Thanks
> Anil
>
>
>
> On Wed, Jan 31, 2018 at 6:42 PM, Bryan Bende <bbe...@gmail.com> wrote:
>
> Hello,
>
> The identity in authorizers.xml for your initial admin does not match the
> identity of your client cert.
>
> You should be putting “CN=TC, OU=NIFI” as the initial admin because that
>
> is
>
> the DN of your client cert.
>
> You’ll need to stop NiFi, edit authorizers.xml, delete users.xml and
> authorizations.xml, and start back up.
>
> Thanks,
>
> Bryan
>
> On Wed, Jan 31, 2018 at 6:11 PM Anil Rai <anilrain...@gmail.com> wrote:
>
> All,
>
> I am trying to install nifi 1.5 and making it https. Below is the steps
> followed and the error i am getting. Below is the config and log files
> content. Please help
>
> 1. Installed nifi 1.5
> 2. Installed nifi toolkit 1.5
> 3. Ran toolkit - ./tls-toolkit.sh standalone -n 'localhost' -C
> 'CN=TC,OU=NIFI' -O -o ../security_output
> 4. Copied generated keystore, truststore and nifi properties to
>
> nifi/config
>
> folder
> 5. Imported the generated certificate to chrome browser
> 6. Modified authorizers.xml as attached.
> 7. With required restarts. Now when i enter the below url in the
>
> browser, I
>
> see the below error.
>
> https://localhost:9443/nifi/
>
> Insufficient Permissions
>
>   - home
>
> Unknown user with identity 'CN=TC, OU=NIFI'. Contact the system
> administrator.
>
>
> authorizers.xml
> 
>
>file-user-group-p

Re: setting up secure nifi

[Anil Rai]

Hi Bryan,

Thanks for the quick reply. I did followed your steps. But I am seeing the
same error.
Now the entry looks like
CN=TC,OU=NIFI,dc=example,
dc=com

Also what does dc stand for after CN and OU. Is that a problem.
Is there a blog that talks about installing and making it https using
toolkit?. I did not find any good post that talks end to end from
installing to making it secure using tls toolkit.

Any help is appreciated.

Thanks
Anil



On Wed, Jan 31, 2018 at 6:42 PM, Bryan Bende <bbe...@gmail.com> wrote:

> Hello,
>
> The identity in authorizers.xml for your initial admin does not match the
> identity of your client cert.
>
> You should be putting “CN=TC, OU=NIFI” as the initial admin because that is
> the DN of your client cert.
>
> You’ll need to stop NiFi, edit authorizers.xml, delete users.xml and
> authorizations.xml, and start back up.
>
> Thanks,
>
> Bryan
>
> On Wed, Jan 31, 2018 at 6:11 PM Anil Rai <anilrain...@gmail.com> wrote:
>
> > All,
> >
> > I am trying to install nifi 1.5 and making it https. Below is the steps
> > followed and the error i am getting. Below is the config and log files
> > content. Please help
> >
> > 1. Installed nifi 1.5
> > 2. Installed nifi toolkit 1.5
> > 3. Ran toolkit - ./tls-toolkit.sh standalone -n 'localhost' -C
> > 'CN=TC,OU=NIFI' -O -o ../security_output
> > 4. Copied generated keystore, truststore and nifi properties to
> nifi/config
> > folder
> > 5. Imported the generated certificate to chrome browser
> > 6. Modified authorizers.xml as attached.
> > 7. With required restarts. Now when i enter the below url in the
> browser, I
> > see the below error.
> >
> > https://localhost:9443/nifi/
> >
> > Insufficient Permissions
> >
> >- home
> >
> > Unknown user with identity 'CN=TC, OU=NIFI'. Contact the system
> > administrator.
> >
> >
> > authorizers.xml
> > 
> > 
> > file-user-group-provider
> > org.apache.nifi.authorization.
> FileUserGroupProvider
> > ./conf/users.xml
> > 
> >
> > cn=TC,ou=NIFI,dc=example,dc=com
> > 
> >
> > 
> > file-access-policy-provider
> >
> > org.apache.nifi.authorization.FileAccessPolicyProvider
> > file-user-group-provider
> > ./conf/authorizations.xml
> > cn=TC,ou=NIFI,dc=example,dc=com
> > 
> >
> > 
> > 
> > 
> >
> > nifi-user.log
> > ---
> > 2018-01-31 17:51:20,220 INFO [main] o.a.n.a.FileUserGroupProvider
> Creating
> > new users file at
> > /Users/anilrai/projects/tc/servicemax/nifi-1.5.0/./conf/users.xml
> > 2018-01-31 17:51:20,234 INFO [main] o.a.n.a.FileUserGroupProvider
> > Users/Groups file loaded at Wed Jan 31 17:51:20 EST 2018
> > 2018-01-31 17:51:20,240 INFO [main] o.a.n.a.FileAccessPolicyProvider
> > Creating new authorizations file at
> > /Users/anilrai/projects/tc/servicemax/nifi-1.5.0/./conf/
> authorizations.xml
> > 2018-01-31 17:51:20,264 INFO [main] o.a.n.a.FileAccessPolicyProvider
> > Populating authorizations for Initial Admin:
> > cn=TC,ou=NIFI,dc=example,dc=com
> > 2018-01-31 17:51:20,271 INFO [main] o.a.n.a.FileAccessPolicyProvider
> > Authorizations file loaded at Wed Jan 31 17:51:20 EST 2018
> > 2018-01-31 17:52:18,192 INFO [NiFi Web Server-28]
> > o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
> > Kerberos ticket login not supported by this NiFi.. Returning Conflict
> > response.
> > 2018-01-31 17:52:18,306 INFO [NiFi Web Server-67]
> > o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
> > OpenId Connect is not configured.. Returning Conflict response.
> > 2018-01-31 17:52:18,350 INFO [NiFi Web Server-27]
> > o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=TC,
> OU=NIFI)
> > GET https://localhost:9443/nifi-api/flow/current-user (source ip:
> > 127.0.0.1)
> > 2018-01-31 17:52:18,354 INFO [NiFi Web Server-27]
> > o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=TC,
> > OU=NIFI
> > 2018-01-31 17:52:18,424 INFO [NiFi Web Server-27]
> > o.a.n.w.a.c.AccessDeniedExceptionMapper identity[CN=TC, OU=NIFI],
> groups[]
> > does not have permission to access the requested resource. Unknown user
> > with identity 'CN=TC, OU=NIFI'. Returning Forbidden response.
> > --
> >
> > Generated users.xml
> > --

setting up secure nifi

[Anil Rai]

All,

I am trying to install nifi 1.5 and making it https. Below is the steps
followed and the error i am getting. Below is the config and log files
content. Please help

1. Installed nifi 1.5
2. Installed nifi toolkit 1.5
3. Ran toolkit - ./tls-toolkit.sh standalone -n 'localhost' -C
'CN=TC,OU=NIFI' -O -o ../security_output
4. Copied generated keystore, truststore and nifi properties to nifi/config
folder
5. Imported the generated certificate to chrome browser
6. Modified authorizers.xml as attached.
7. With required restarts. Now when i enter the below url in the browser, I
see the below error.

https://localhost:9443/nifi/

Insufficient Permissions

   - home

Unknown user with identity 'CN=TC, OU=NIFI'. Contact the system
administrator.


authorizers.xml


file-user-group-provider
org.apache.nifi.authorization.FileUserGroupProvider
./conf/users.xml


cn=TC,ou=NIFI,dc=example,dc=com



file-access-policy-provider

org.apache.nifi.authorization.FileAccessPolicyProvider
file-user-group-provider
./conf/authorizations.xml
cn=TC,ou=NIFI,dc=example,dc=com






nifi-user.log
---
2018-01-31 17:51:20,220 INFO [main] o.a.n.a.FileUserGroupProvider Creating
new users file at
/Users/anilrai/projects/tc/servicemax/nifi-1.5.0/./conf/users.xml
2018-01-31 17:51:20,234 INFO [main] o.a.n.a.FileUserGroupProvider
Users/Groups file loaded at Wed Jan 31 17:51:20 EST 2018
2018-01-31 17:51:20,240 INFO [main] o.a.n.a.FileAccessPolicyProvider
Creating new authorizations file at
/Users/anilrai/projects/tc/servicemax/nifi-1.5.0/./conf/authorizations.xml
2018-01-31 17:51:20,264 INFO [main] o.a.n.a.FileAccessPolicyProvider
Populating authorizations for Initial Admin: cn=TC,ou=NIFI,dc=example,dc=com
2018-01-31 17:51:20,271 INFO [main] o.a.n.a.FileAccessPolicyProvider
Authorizations file loaded at Wed Jan 31 17:51:20 EST 2018
2018-01-31 17:52:18,192 INFO [NiFi Web Server-28]
o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
Kerberos ticket login not supported by this NiFi.. Returning Conflict
response.
2018-01-31 17:52:18,306 INFO [NiFi Web Server-67]
o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
OpenId Connect is not configured.. Returning Conflict response.
2018-01-31 17:52:18,350 INFO [NiFi Web Server-27]
o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=TC, OU=NIFI)
GET https://localhost:9443/nifi-api/flow/current-user (source ip: 127.0.0.1)
2018-01-31 17:52:18,354 INFO [NiFi Web Server-27]
o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=TC, OU=NIFI
2018-01-31 17:52:18,424 INFO [NiFi Web Server-27]
o.a.n.w.a.c.AccessDeniedExceptionMapper identity[CN=TC, OU=NIFI], groups[]
does not have permission to access the requested resource. Unknown user
with identity 'CN=TC, OU=NIFI'. Returning Forbidden response.
--

Generated users.xml










Generated authorizations.xml
--











































nifi.properties

# web properties #
nifi.web.war.directory=./lib
nifi.web.http.host=
nifi.web.http.port=
nifi.web.http.network.interface.default=
nifi.web.https.host=localhost
nifi.web.https.port=9443
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=

# security properties #
nifi.sensitive.props.key=
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=

nifi.security.keystore=./conf/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=dDGiDzvaUrecDVYGbfXq/w6G7z6ldn4oKuQemifG3iI
nifi.security.keyPasswd=dDGiDzvaUrecDVYGbfXq/w6G7z6ldn4oKuQemifG3iI
nifi.security.truststore=./conf/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=Kckcz+CPJduHRzOsdJFaSffmJHLHqJ7noxY3ZHZyqI4
nifi.security.needClientAuth=
nifi.security.user.authorizer=managed-authorizer
nifi.security.user.login.identity.provider=
nifi.security.ocsp.responder.url=
nifi.security.ocsp.responder.certificate=
--



Please help.

Regards
Anil

Nifi on Apache Mesos-DC/OS

[Anil Rai]

Hi,

Has anyone tried to deploy nifi on Mesos or DC/OS and is there a
recommended nifi deployment architecture for mesos or DC/OS?

Regards
Anil

Nifi not starting

[Anil Rai]

Hello,

I have 2 clarifications.

We have a 3 node cluster (3 zookeeper and 3 nodes). When we restart the
cluster (in no particular order) and when the flow are running, the cluster
does not start and we see the attached error in the logs. This has happened
to us couple of time.
Do we have to restart the cluster in any particular order? and also should
we stop all the flows before we restart? please let us know.

Secondly, is there a way to track a transaction flowing through a flow from
start to end out of the box? Basically I would like to know how many
transaction were processed by a flow in an hour/day or week

Thanks
Anil

Re: Mapper

[Anil Rai]

Thank you Aldrin. Will take a look.

Regards
Anil


On Fri, Jun 2, 2017 at 4:44 PM, Aldrin Piri <aldrinp...@gmail.com> wrote:

> Hi Anil,
>
> The Quick Start [1], contributor's guide [2], and developer's guide [3]
> have a lot of good information.  We are of course happy to answer any
> questions across the mailing list should you hit any parts that are unclear
> or just have general questions.
>
> Looking forward to your contributions!
>
> [1] http://nifi.apache.org/quickstart.html
> [2] https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide
> [3] http://nifi.apache.org/developer-guide.html
>
> On Fri, Jun 2, 2017 at 4:39 PM, Anil Rai <anilrain...@gmail.com> wrote:
>
> > To clarify...to start contributing...is there a guide for naming
> > convention, github, unit testing and things like that..
> >
> >
> > On Fri, Jun 2, 2017 at 4:38 PM, Anil Rai <anilrain...@gmail.com> wrote:
> >
> > > Thanks Mark for the detailed explanation and suggestions. Will take a
> > > look. Yes, I was referring to ExecuteSQL.
> > > To start contributing, is there any documentation or guide for
> starters?
> > >
> > > Cheers
> > > Anil
> > >
> > >
> > > On Fri, Jun 2, 2017 at 8:24 AM, Mark Payne <marka...@hotmail.com>
> wrote:
> > >
> > >> Hi Anil,
> > >>
> > >> Sorry that it took so long to respond. This email got lost in my
> inbox,
> > >> it seems.
> > >>
> > >> Regarding a mapper UI: there is not currently anything like that in
> > NiFi.
> > >> I do not
> > >> know of anyone building such a thing either. However, it would be very
> > >> helpful,
> > >> for sure, if it is something that you are interested in contributing.
> > >>
> > >> Regarding ExecuteSQL (I'm assuming that this is the processor that
> > you're
> > >> referring to),
> > >> I agree that this can be better. In the latest version, 1.2.0, we
> > >> introduced the concept of Record
> > >> Readers and Record Writers, and I think that this Processor needs to
> be
> > >> updated to make use of
> > >> those. This would allow you to choose JSON, Avro, etc. as the output
> > >> format. At the moment, there
> > >> is no Record Reader or Writer for XML but I think that we will need to
> > be
> > >> building one in the near
> > >> future (again, another great place for you to jump in if you are
> > >> interested in contributing). I have
> > >> seen someone make use of the scripted Record Writer in order to write
> > >> XML, as well, as Groovy
> > >> has a lot of built-in niceties for reading and writing XML, so that
> > would
> > >> be another option.
> > >>
> > >> Thanks!
> > >> -Mark
> > >>
> > >>
> > >> > On May 19, 2017, at 12:14 PM, Anil Rai <anilrain...@gmail.com>
> wrote:
> > >> >
> > >> > Hi,
> > >> > I was wondering if there is an UI for XML mapping that is already
> > >> available
> > >> > or some is working on?
> > >> > We have transformXML. But the problem is that we need to generate
> the
> > >> XSLT
> > >> > outside and then feed it in. It would be nice if we have a mapper
> > that a
> > >> > developer/business user can use to map the source to the target
> > >> graphically.
> > >> > Also today the DB connector out comes out in avero format. We are
> > >> forced to
> > >> > add additional transforms like avero to json or avero to xml to
> > >> proceed. It
> > >> > would be nice if we could configure the DB processor to output the
> > file
> > >> in
> > >> > the format that we need/
> > >> > Any thoughts?
> > >> >
> > >> > Thanks
> > >> > Anil
> > >>
> > >>
> > >
> >
>

Re: Mapper

[Anil Rai]

To clarify...to start contributing...is there a guide for naming
convention, github, unit testing and things like that..


On Fri, Jun 2, 2017 at 4:38 PM, Anil Rai <anilrain...@gmail.com> wrote:

> Thanks Mark for the detailed explanation and suggestions. Will take a
> look. Yes, I was referring to ExecuteSQL.
> To start contributing, is there any documentation or guide for starters?
>
> Cheers
> Anil
>
>
> On Fri, Jun 2, 2017 at 8:24 AM, Mark Payne <marka...@hotmail.com> wrote:
>
>> Hi Anil,
>>
>> Sorry that it took so long to respond. This email got lost in my inbox,
>> it seems.
>>
>> Regarding a mapper UI: there is not currently anything like that in NiFi.
>> I do not
>> know of anyone building such a thing either. However, it would be very
>> helpful,
>> for sure, if it is something that you are interested in contributing.
>>
>> Regarding ExecuteSQL (I'm assuming that this is the processor that you're
>> referring to),
>> I agree that this can be better. In the latest version, 1.2.0, we
>> introduced the concept of Record
>> Readers and Record Writers, and I think that this Processor needs to be
>> updated to make use of
>> those. This would allow you to choose JSON, Avro, etc. as the output
>> format. At the moment, there
>> is no Record Reader or Writer for XML but I think that we will need to be
>> building one in the near
>> future (again, another great place for you to jump in if you are
>> interested in contributing). I have
>> seen someone make use of the scripted Record Writer in order to write
>> XML, as well, as Groovy
>> has a lot of built-in niceties for reading and writing XML, so that would
>> be another option.
>>
>> Thanks!
>> -Mark
>>
>>
>> > On May 19, 2017, at 12:14 PM, Anil Rai <anilrain...@gmail.com> wrote:
>> >
>> > Hi,
>> > I was wondering if there is an UI for XML mapping that is already
>> available
>> > or some is working on?
>> > We have transformXML. But the problem is that we need to generate the
>> XSLT
>> > outside and then feed it in. It would be nice if we have a mapper that a
>> > developer/business user can use to map the source to the target
>> graphically.
>> > Also today the DB connector out comes out in avero format. We are
>> forced to
>> > add additional transforms like avero to json or avero to xml to
>> proceed. It
>> > would be nice if we could configure the DB processor to output the file
>> in
>> > the format that we need/
>> > Any thoughts?
>> >
>> > Thanks
>> > Anil
>>
>>
>

Re: Mapper

[Anil Rai]

Thanks Mark for the detailed explanation and suggestions. Will take a look.
Yes, I was referring to ExecuteSQL.
To start contributing, is there any documentation or guide for starters?

Cheers
Anil


On Fri, Jun 2, 2017 at 8:24 AM, Mark Payne <marka...@hotmail.com> wrote:

> Hi Anil,
>
> Sorry that it took so long to respond. This email got lost in my inbox, it
> seems.
>
> Regarding a mapper UI: there is not currently anything like that in NiFi.
> I do not
> know of anyone building such a thing either. However, it would be very
> helpful,
> for sure, if it is something that you are interested in contributing.
>
> Regarding ExecuteSQL (I'm assuming that this is the processor that you're
> referring to),
> I agree that this can be better. In the latest version, 1.2.0, we
> introduced the concept of Record
> Readers and Record Writers, and I think that this Processor needs to be
> updated to make use of
> those. This would allow you to choose JSON, Avro, etc. as the output
> format. At the moment, there
> is no Record Reader or Writer for XML but I think that we will need to be
> building one in the near
> future (again, another great place for you to jump in if you are
> interested in contributing). I have
> seen someone make use of the scripted Record Writer in order to write XML,
> as well, as Groovy
> has a lot of built-in niceties for reading and writing XML, so that would
> be another option.
>
> Thanks!
> -Mark
>
>
> > On May 19, 2017, at 12:14 PM, Anil Rai <anilrain...@gmail.com> wrote:
> >
> > Hi,
> > I was wondering if there is an UI for XML mapping that is already
> available
> > or some is working on?
> > We have transformXML. But the problem is that we need to generate the
> XSLT
> > outside and then feed it in. It would be nice if we have a mapper that a
> > developer/business user can use to map the source to the target
> graphically.
> > Also today the DB connector out comes out in avero format. We are forced
> to
> > add additional transforms like avero to json or avero to xml to proceed.
> It
> > would be nice if we could configure the DB processor to output the file
> in
> > the format that we need/
> > Any thoughts?
> >
> > Thanks
> > Anil
>
>

Re: Data Load

[Anil Rai]

Hi Matt,

I quickly developed this and this is how i could do this

DataLake<-ExecuteSQL->ConvertAveroToJson->SplitJson->EvaluateJsonPath->ReplaceText->PutSQL->Postgres(onCloud)

The problem is, this will not scale for huge volumes. Any thoughts?

Regards
Anil


On Tue, May 2, 2017 at 12:07 PM, Matt Burgess <mattyb...@apache.org> wrote:

> Yes that sounds like your best bet, assuming you have the "Maximum
> Value Column" present in the table you want to migrate.  Then a flow
> might look like:
>
> QueryDatabaseTable -> ConvertAvroToJSON -> ConvertJSONToSQL -> PutSQL
>
> In this flow the target tables would need to be created beforehand.
> You might be able to do that with pg_dump or with some fancy SQL that
> you could send to PutSQL in a separate (do-ahead) flow [1].  For
> multiple tables, you will need one QueryDatabaseTable for each table;
> depending on the number of tables and the latency for getting/putting
> rows, you may be able to share the downstream processing. If that
> creates a bottleneck, you may want a copy of the above flow for each
> table.  This is drastically improved in NiFi 1.2.0, as you can use
> ListDatabaseTables -> GenerateTableFetch -> RPG -> Input Port ->
> ExecuteSQL to perform the migration in parallel across a NiFi cluster.
>
> Regards,
> Matt
>
> [1] https://serverfault.com/questions/231952/is-there-a-
> mysql-equivalent-of-show-create-table-in-postgres
>
>
> On Tue, May 2, 2017 at 11:18 AM, Anil Rai <anilrain...@gmail.com> wrote:
> > Thanks Matt for the quick reply. We are using nifi 1.0 release as of now.
> > It's a postgres DB on both sides (on prem and on cloud)
> > and yes incremental load is what i am looking for.
> > so with that, you recommend # 2 option?
> >
> > On Tue, May 2, 2017 at 11:00 AM, Matt Burgess <mattyb...@apache.org>
> wrote:
> >
> >> Anil,
> >>
> >> Is this a "one-time" migration, meaning you would take the on-prem
> >> tables and put them on the cloud DB just once? Or would this be an
> >> incremental operation, where you do the initial move and then take any
> >> "new" rows from the source and apply them to the target?  For the
> >> latter, there are a couple of options:
> >>
> >> 1) Rebuild the cloud DB periodically. You can use ExecuteSQL ->
> >> [processors] -> PutSQL after perhaps deleting your target
> >> DB/tables/etc.  This could be time-consuming and expensive. The
> >> processors in-between probably include ConvertAvroToJSON and
> >> ConvertJSONToSQL.
> >> 2) Use QueryDatabaseTable or (GenerateTableFetch -> ExecuteSQL) to get
> >> the source data. For this your table would need a column whose values
> >> always increase, that column would comprise the value of the "Maximum
> >> Value Column" property in the aforementioned processors' configuration
> >> dialogs. You would need one QueryDatabaseTable or GenerateTableFetch
> >> for each table in your DB.
> >>
> >> In addition to these current solutions, as of the upcoming NiFi 1.2.0
> >> release, you have the following options:
> >> 3) If the source database is MySQL, you can use the CaptureChangeMySQL
> >> processor to get binary log events flowing through various processors
> >> into PutDatabaseRecord to place them at the source. This pattern is
> >> true Change Data Capture (CDC) versus the other two options above.
> >> 4) Option #2 will be improved such that GenerateTableFetch will accept
> >> incoming flow files, so you can use ListDatabaseTables ->
> >> GenerateTableFetch -> ExecuteSQL to handle multiple tables with one
> >> flow.
> >>
> >> If this is a one-time migration, a data flow tool might not be the
> >> best choice, you could consider something like Flyway [1] instead.
> >>
> >> Regards,
> >> Matt
> >>
> >> [1] https://flywaydb.org/documentation/command/migrate
> >>
> >> On Tue, May 2, 2017 at 10:41 AM, Anil Rai <anilrain...@gmail.com>
> wrote:
> >> > I have a simple use case.
> >> >
> >> > DB (On Premise) and DB (On Cloud).
> >> >
> >> > I want to use nifi to extract data from on prem DB (huge volumes) and
> >> > insert into the same table structure that is hosted on cloud.
> >> >
> >> > I could use ExecuteSQL on both sides of the fence (to extract from on
> >> prem
> >> > and insert onto cloud). What processors are needed in between (if at
> >> all)?
> >> > As i am not doing any transformations at allit is just extract and
> >> load
> >> > use case
> >>
>

Re: Data Load

[Anil Rai]

Thanks Matt for the quick reply. We are using nifi 1.0 release as of now.
It's a postgres DB on both sides (on prem and on cloud)
and yes incremental load is what i am looking for.
so with that, you recommend # 2 option?

On Tue, May 2, 2017 at 11:00 AM, Matt Burgess <mattyb...@apache.org> wrote:

> Anil,
>
> Is this a "one-time" migration, meaning you would take the on-prem
> tables and put them on the cloud DB just once? Or would this be an
> incremental operation, where you do the initial move and then take any
> "new" rows from the source and apply them to the target?  For the
> latter, there are a couple of options:
>
> 1) Rebuild the cloud DB periodically. You can use ExecuteSQL ->
> [processors] -> PutSQL after perhaps deleting your target
> DB/tables/etc.  This could be time-consuming and expensive. The
> processors in-between probably include ConvertAvroToJSON and
> ConvertJSONToSQL.
> 2) Use QueryDatabaseTable or (GenerateTableFetch -> ExecuteSQL) to get
> the source data. For this your table would need a column whose values
> always increase, that column would comprise the value of the "Maximum
> Value Column" property in the aforementioned processors' configuration
> dialogs. You would need one QueryDatabaseTable or GenerateTableFetch
> for each table in your DB.
>
> In addition to these current solutions, as of the upcoming NiFi 1.2.0
> release, you have the following options:
> 3) If the source database is MySQL, you can use the CaptureChangeMySQL
> processor to get binary log events flowing through various processors
> into PutDatabaseRecord to place them at the source. This pattern is
> true Change Data Capture (CDC) versus the other two options above.
> 4) Option #2 will be improved such that GenerateTableFetch will accept
> incoming flow files, so you can use ListDatabaseTables ->
> GenerateTableFetch -> ExecuteSQL to handle multiple tables with one
> flow.
>
> If this is a one-time migration, a data flow tool might not be the
> best choice, you could consider something like Flyway [1] instead.
>
> Regards,
> Matt
>
> [1] https://flywaydb.org/documentation/command/migrate
>
> On Tue, May 2, 2017 at 10:41 AM, Anil Rai <anilrain...@gmail.com> wrote:
> > I have a simple use case.
> >
> > DB (On Premise) and DB (On Cloud).
> >
> > I want to use nifi to extract data from on prem DB (huge volumes) and
> > insert into the same table structure that is hosted on cloud.
> >
> > I could use ExecuteSQL on both sides of the fence (to extract from on
> prem
> > and insert onto cloud). What processors are needed in between (if at
> all)?
> > As i am not doing any transformations at allit is just extract and
> load
> > use case
>

Data Load

[Anil Rai]

I have a simple use case.

DB (On Premise) and DB (On Cloud).

I want to use nifi to extract data from on prem DB (huge volumes) and
insert into the same table structure that is hosted on cloud.

I could use ExecuteSQL on both sides of the fence (to extract from on prem
and insert onto cloud). What processors are needed in between (if at all)?
As i am not doing any transformations at allit is just extract and load
use case

Re: Connection Issue

[Anil Rai]

Hi Oleg, if you look at the java code and logs i sent in the previous
email, it does have the code to unwrap. But it is not allowing to cast that
connection object back to OracleConnection. It fails.

On Wed, Mar 15, 2017 at 2:14 PM, Oleg Zhurakousky <
ozhurakou...@hortonworks.com> wrote:

> Ok, so it appears that DBCP does not like you to get access to the
> underlying connection without additional configuration, but Java does since
> java.sql.Connection extends java.sql.Wrapper from which you can do
> something like this:
>
> Connection dbConnection= d.getConnection();
> OracleConnection orcConnection = null;
> if (dbConnection.isWrapperFor(OracleConnection.class)) {
> orcConnection = dbConnection.unwrap(OracleConnection.class);
> }
> . . .
>
> Let me know
>
> Cheers
> Oleg
>
> On Mar 15, 2017, at 1:33 PM, Anil Rai <anilrain...@gmail.com anilrain...@gmail.com>> wrote:
>
> Hi Oleg, we tried and no luck. This is another example where we are seeing
> similar issue. Attached txt has the java code as well as the log.
>
> Thanks in advance
> Anil
>
>
> On Wed, Mar 15, 2017 at 10:02 AM, Anil Rai <anilrain...@gmail.com anilrain...@gmail.com>> wrote:
> Thanks Oleg. Makes sense. Will try and keep you posted.
>
> Regards
> Anil
>
>
> On Wed, Mar 15, 2017 at 9:56 AM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:
> Anil
>
> Thank you for details. That does help a lot.
>
> First, I want to make sure that it is clear that this is not a NiFi issue,
> since the problem is specific to the combination of DBCP and Oracle and the
> expectations between the two.
>
> Seems like Oracle JDBC connection is wrapped in an implementation specific
> class (DBCP in this case I assume).
> It is my believe that you need to obtain reference to native Oracle
> connection to avoid "JMS-112: Connection is invalid".
> So, I think you need to try to cast your Connection object to DBCPs
> DelegatingConnection and then do something like this:
>
> DelegatingConnection wrappedConn = (DelegatingConnection)con;
> OracleConnection ocon =  null ;
> if (wrappedConn != null)
>  ocon = (OracleConnection) wrappedConn.getDelegate();
>  . . .
>
> Let me know how it goes
>
> Cheers
> Oleg
>
> > On Mar 15, 2017, at 9:20 AM, Anil Rai <anilrain...@gmail.com anilrain...@gmail.com>> wrote:
> >
> > 2017-03-14 16:50:42,312 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at
> > java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at
> > java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at java.lang.Thread.run(Thread.java:745)
> > 2017-03-14 16:50:43,567 INFO [NiFi logging handler]
> org.apache.nifi.StdOut
> > Databse Connection :- jdbc:oracle:thin:@xxxog.yy.com:1521/DB1<
> http://jdbc:oracle:thin:@xxxog.yy.com:1521/DB1>,
> > UserName=user, Oracle JDBC driver
> > 2017-03-14 16:50:43,567 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> > oracle.AQ.AQException: JMS-112: Connection is invalid
> > 2017-03-14 16:50:43,567 INFO [NiFi logging handler]
> org.apache.nifi.StdOut
> > AQ Driver Class ---> oracle.AQ.AQOracleDriver
> > 2017-03-14 16:50:43,567 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> > 2017-03-14 16:50:43,568 INFO [NiFi logging handler]
> org.apache.nifi.StdOut
> > Aq Sesssion ---> null
> > 2017-03-14 16:50:43,568 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> > 2017-03-14 16:50:43,568 INFO [NiFi logging handler]
> org.apache.nifi.StdOut
> > Queue Owner ---> 

Re: Connection Issue

[Anil Rai]

Hi Oleg, we tried and no luck. This is another example where we are seeing
similar issue. Attached txt has the java code as well as the log.

Thanks in advance
Anil


On Wed, Mar 15, 2017 at 10:02 AM, Anil Rai <anilrain...@gmail.com> wrote:

> Thanks Oleg. Makes sense. Will try and keep you posted.
>
> Regards
> Anil
>
>
> On Wed, Mar 15, 2017 at 9:56 AM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com> wrote:
>
>> Anil
>>
>> Thank you for details. That does help a lot.
>>
>> First, I want to make sure that it is clear that this is not a NiFi
>> issue, since the problem is specific to the combination of DBCP and Oracle
>> and the expectations between the two.
>>
>> Seems like Oracle JDBC connection is wrapped in an implementation
>> specific class (DBCP in this case I assume).
>> It is my believe that you need to obtain reference to native Oracle
>> connection to avoid "JMS-112: Connection is invalid".
>> So, I think you need to try to cast your Connection object to DBCPs
>> DelegatingConnection and then do something like this:
>>
>> DelegatingConnection wrappedConn = (DelegatingConnection)con;
>> OracleConnection ocon =  null ;
>> if (wrappedConn != null)
>>      ocon = (OracleConnection) wrappedConn.getDelegate();
>>  . . .
>>
>> Let me know how it goes
>>
>> Cheers
>> Oleg
>>
>> > On Mar 15, 2017, at 9:20 AM, Anil Rai <anilrain...@gmail.com> wrote:
>> >
>> > 2017-03-14 16:50:42,312 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>> >  at
>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>> >  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:
>> 308)
>> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>> >  at
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>> >  at
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(ScheduledThreadPoolExecutor.java:294)
>> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>> >  at
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1142)
>> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>> >  at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:617)
>> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>> >  at java.lang.Thread.run(Thread.java:745)
>> > 2017-03-14 16:50:43,567 INFO [NiFi logging handler]
>> org.apache.nifi.StdOut
>> > Databse Connection :- jdbc:oracle:thin:@xxxog.yy.com:1521/DB1,
>> > UserName=user, Oracle JDBC driver
>> > 2017-03-14 16:50:43,567 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>> > oracle.AQ.AQException: JMS-112: Connection is invalid
>> > 2017-03-14 16:50:43,567 INFO [NiFi logging handler]
>> org.apache.nifi.StdOut
>> > AQ Driver Class ---> oracle.AQ.AQOracleDriver
>> > 2017-03-14 16:50:43,567 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>> > 2017-03-14 16:50:43,568 INFO [NiFi logging handler]
>> org.apache.nifi.StdOut
>> > Aq Sesssion ---> null
>> > 2017-03-14 16:50:43,568 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>> > 2017-03-14 16:50:43,568 INFO [NiFi logging handler]
>> org.apache.nifi.StdOut
>> > Queue Owner ---> APPS
>> > 2017-03-14 16:50:43,568 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>> >  at oracle.AQ.AQDriverManager.createAQSession(AQDriverManager.
>> java:193)
>> > 2017-03-14 16:50:43,569 INFO [NiFi logging handler]
>> org.apache.nifi.StdOut
>> > QueueName ---> WF_BPEL_Q
>> > 2017-03-14 16:50:43,569 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>> >  at
>> > com.oracle.xx.connector.processors.xx_SCon_ConsumeAQ.xx_
>> Scon_ConsumeAQ.createSession(xx_Scon_ConsumeAQ.java:183)
>> > 2017-03-14 16:50:43,569 INFO [NiFi logging handler]
>> org.apache.nifi.StdOut
>> > EventName ---> oracle.apps.ar.hz.CustAcctSite.update
>> > 2017-03-14 16:50:43,569 ERROR [NiFi logging 

Re: Connection Issue

[Anil Rai]

Thanks Oleg. Makes sense. Will try and keep you posted.

Regards
Anil


On Wed, Mar 15, 2017 at 9:56 AM, Oleg Zhurakousky <
ozhurakou...@hortonworks.com> wrote:

> Anil
>
> Thank you for details. That does help a lot.
>
> First, I want to make sure that it is clear that this is not a NiFi issue,
> since the problem is specific to the combination of DBCP and Oracle and the
> expectations between the two.
>
> Seems like Oracle JDBC connection is wrapped in an implementation specific
> class (DBCP in this case I assume).
> It is my believe that you need to obtain reference to native Oracle
> connection to avoid "JMS-112: Connection is invalid".
> So, I think you need to try to cast your Connection object to DBCPs
> DelegatingConnection and then do something like this:
>
> DelegatingConnection wrappedConn = (DelegatingConnection)con;
> OracleConnection ocon =  null ;
> if (wrappedConn != null)
>  ocon = (OracleConnection) wrappedConn.getDelegate();
>  . . .
>
> Let me know how it goes
>
> Cheers
> Oleg
>
> > On Mar 15, 2017, at 9:20 AM, Anil Rai <anilrain...@gmail.com> wrote:
> >
> > 2017-03-14 16:50:42,312 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at
> > java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at
> > java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> > 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at java.lang.Thread.run(Thread.java:745)
> > 2017-03-14 16:50:43,567 INFO [NiFi logging handler]
> org.apache.nifi.StdOut
> > Databse Connection :- jdbc:oracle:thin:@xxxog.yy.com:1521/DB1,
> > UserName=user, Oracle JDBC driver
> > 2017-03-14 16:50:43,567 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> > oracle.AQ.AQException: JMS-112: Connection is invalid
> > 2017-03-14 16:50:43,567 INFO [NiFi logging handler]
> org.apache.nifi.StdOut
> > AQ Driver Class ---> oracle.AQ.AQOracleDriver
> > 2017-03-14 16:50:43,567 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> > 2017-03-14 16:50:43,568 INFO [NiFi logging handler]
> org.apache.nifi.StdOut
> > Aq Sesssion ---> null
> > 2017-03-14 16:50:43,568 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> > 2017-03-14 16:50:43,568 INFO [NiFi logging handler]
> org.apache.nifi.StdOut
> > Queue Owner ---> APPS
> > 2017-03-14 16:50:43,568 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at oracle.AQ.AQDriverManager.createAQSession(
> AQDriverManager.java:193)
> > 2017-03-14 16:50:43,569 INFO [NiFi logging handler]
> org.apache.nifi.StdOut
> > QueueName ---> WF_BPEL_Q
> > 2017-03-14 16:50:43,569 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at
> > com.oracle.xx.connector.processors.xx_SCon_ConsumeAQ.xx_Scon_ConsumeAQ.
> createSession(xx_Scon_ConsumeAQ.java:183)
> > 2017-03-14 16:50:43,569 INFO [NiFi logging handler]
> org.apache.nifi.StdOut
> > EventName ---> oracle.apps.ar.hz.CustAcctSite.update
> > 2017-03-14 16:50:43,569 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at
> > com.oracle.xx.connector.processors.XX_SCon_ConsumeAQ.
> XX_Scon_ConsumeAQ.onTrigger(XX_Scon_ConsumeAQ.java:254)
> > 2017-03-14 16:50:43,569 INFO [NiFi logging handler]
> org.apache.nifi.StdOut
> > Consumer Name ---> ORA_69242HHG6SR4CGPI6KSJ8DI36T
> > 2017-03-14 16:50:43,569 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at
> > org.apache.nifi.processor.AbstractProcessor.onTrigger(
> AbstractProcessor.java:27)
> > 2017-03-14 16:50:43,569 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr
> >  at
> > org.apache.nifi.con

Re: Connection Issue

[Anil Rai]

apache.nifi.StdErr
java.lang.NullPointerException


On Wed, Mar 15, 2017 at 9:09 AM, Oleg Zhurakousky <
ozhurakou...@hortonworks.com> wrote:

> Anil
>
> Unfortunately the attachment didn’t come thru. Perhaps you can just paste
> the relevant part of the exception.
>
> Cheers
> Oleg
>
> On Mar 15, 2017, at 8:58 AM, Anil Rai <anilrain...@gmail.com anilrain...@gmail.com>> wrote:
>
> Hi Oleg, Thanks. attached is the log. Let me know if you want us to change
> the log levels and re-run and send you additional logs?
>
>
> On Tue, Mar 14, 2017 at 5:12 PM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:
> Anil
>
> I understand that you are having an issue and we are here to help, but we
> can only do this if you help us just a little more, so it would be very
> helpful if you provided a stack trace (I understand if you have to mask
> sensitive information).
> The “. . .fails saying cannot create AQSession. . .” could be due to
> various reasons and until we see the stack trace everything here would be
> speculation.
> I hope you understand
>
> Cheers
> Oleg
>
> > On Mar 14, 2017, at 4:59 PM, Anil Rai <anilrain...@gmail.com anilrain...@gmail.com>> wrote:
> >
> > Here is the behaviour that we have seen so forhope this helps
> >
> >   1. When we run the java code in eclipse, it works and this is the
> >   connection object that is printed ->
> >   oracle.jdbc.driver.T4CConnection@6f75e721
> >   2. When we hard code all the values as mentioned in my first email in a
> >   custom processor, deploy that. It works as well. The above connection
> >   object gets printed.
> >   3. When we change the code in the custom processor to use the DBCP
> >   connection service, deploy that. The connection object that gets
> printed is
> >   jdbc:oracle:thin:@oged-scan.og.ge.com:1521/ORPOGPB1<http:/
> /jdbc:oracle:thin:@oged-scan.og.ge.com:1521/ORPOGPB1> and this does not
> >   work. aqSession = AQDriverManager.createAQSession(connection) fails
> >   saying cannot create AQSession.
> >
> > Thanks
> > Anil
> >
> >
> > On Tue, Mar 14, 2017 at 4:13 PM, Oleg Zhurakousky <
> > ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>>
> wrote:
> >
> >> Anil
> >>
> >> When you say "it does not like the connection object. . .” what do you
> >> mean by that?
> >> Can you please provide stack trace or some other details?
> >>
> >> Cheers
> >> Oleg
> >>
> >>> On Mar 14, 2017, at 4:06 PM, Anil Rai <anilrain...@gmail.com anilrain...@gmail.com>> wrote:
> >>>
> >>> Thanks Russ. Yes, we are doing exactly the same thing.
> >>>
> >>>   driverClass = context.getProperty(DRIVER_CLASS).getValue();
> >>>   queueName = context.getProperty(QUEUE_NAME).getValue();
> >>>   databaseSchema = context.getProperty(DATABASE_
> SCHEMA).getValue();
> >>>   consumerName = context.getProperty(CONSUMER_NAME).getValue();
> >>>   eventName = context.getProperty(EVENT_NAME).getValue();
> >>>   DBCPService connection =
> >>> context.getProperty(JDBC_CONNECTION_POOL).asControllerService(
> >> DBCPService.class);
> >>>   Connection conn = connection.getConnection();
> >>> *aqSession = AQDriverManager.createAQSession(connection);*
> >>>
> >>> The underlined code above fails as it does not like the connection
> object
> >>> that is returned by the DBCPService.
> >>>
> >>> Regards
> >>> Anil
> >>>
> >>>
> >>> On Tue, Mar 14, 2017 at 2:43 PM, Russell Bateman <
> r...@windofkeltia.com<mailto:r...@windofkeltia.com>>
> >>> wrote:
> >>>
> >>>> Anil,
> >>>>
> >>>> Typically, your custom processor should have a property, something
> like
> >>>>
> >>>>   public static final PropertyDescriptor DBCP_SERVICE = new
> >>>> PropertyDescriptor.Builder()
> >>>>   .name("Database Connection Pooling Service")
> >>>>   .description("The Controller Service that is used to obtain
> >>>> connection to database")
> >>>>   .required(true)
> >>>>   .identifiesControllerService(DBCPService.class)
> >>>>   .build();
> >>>>
> >>>> When your NiFi user sets up the flow, he or

Re: Connection Issue

[Anil Rai]

Hi Oleg, Thanks. attached is the log. Let me know if you want us to change
the log levels and re-run and send you additional logs?


On Tue, Mar 14, 2017 at 5:12 PM, Oleg Zhurakousky <
ozhurakou...@hortonworks.com> wrote:

> Anil
>
> I understand that you are having an issue and we are here to help, but we
> can only do this if you help us just a little more, so it would be very
> helpful if you provided a stack trace (I understand if you have to mask
> sensitive information).
> The “. . .fails saying cannot create AQSession. . .” could be due to
> various reasons and until we see the stack trace everything here would be
> speculation.
> I hope you understand
>
> Cheers
> Oleg
>
> > On Mar 14, 2017, at 4:59 PM, Anil Rai <anilrain...@gmail.com> wrote:
> >
> > Here is the behaviour that we have seen so forhope this helps
> >
> >   1. When we run the java code in eclipse, it works and this is the
> >   connection object that is printed ->
> >   oracle.jdbc.driver.T4CConnection@6f75e721
> >   2. When we hard code all the values as mentioned in my first email in a
> >   custom processor, deploy that. It works as well. The above connection
> >   object gets printed.
> >   3. When we change the code in the custom processor to use the DBCP
> >   connection service, deploy that. The connection object that gets
> printed is
> >   jdbc:oracle:thin:@oged-scan.og.ge.com:1521/ORPOGPB1 and this does not
> >   work. aqSession = AQDriverManager.createAQSession(connection) fails
> >   saying cannot create AQSession.
> >
> > Thanks
> > Anil
> >
> >
> > On Tue, Mar 14, 2017 at 4:13 PM, Oleg Zhurakousky <
> > ozhurakou...@hortonworks.com> wrote:
> >
> >> Anil
> >>
> >> When you say "it does not like the connection object. . .” what do you
> >> mean by that?
> >> Can you please provide stack trace or some other details?
> >>
> >> Cheers
> >> Oleg
> >>
> >>> On Mar 14, 2017, at 4:06 PM, Anil Rai <anilrain...@gmail.com> wrote:
> >>>
> >>> Thanks Russ. Yes, we are doing exactly the same thing.
> >>>
> >>>   driverClass = context.getProperty(DRIVER_CLASS).getValue();
> >>>   queueName = context.getProperty(QUEUE_NAME).getValue();
> >>>   databaseSchema = context.getProperty(DATABASE_
> SCHEMA).getValue();
> >>>   consumerName = context.getProperty(CONSUMER_NAME).getValue();
> >>>   eventName = context.getProperty(EVENT_NAME).getValue();
> >>>   DBCPService connection =
> >>> context.getProperty(JDBC_CONNECTION_POOL).asControllerService(
> >> DBCPService.class);
> >>>   Connection conn = connection.getConnection();
> >>> *aqSession = AQDriverManager.createAQSession(connection);*
> >>>
> >>> The underlined code above fails as it does not like the connection
> object
> >>> that is returned by the DBCPService.
> >>>
> >>> Regards
> >>> Anil
> >>>
> >>>
> >>> On Tue, Mar 14, 2017 at 2:43 PM, Russell Bateman <
> r...@windofkeltia.com>
> >>> wrote:
> >>>
> >>>> Anil,
> >>>>
> >>>> Typically, your custom processor should have a property, something
> like
> >>>>
> >>>>   public static final PropertyDescriptor DBCP_SERVICE = new
> >>>> PropertyDescriptor.Builder()
> >>>>   .name("Database Connection Pooling Service")
> >>>>   .description("The Controller Service that is used to obtain
> >>>> connection to database")
> >>>>   .required(true)
> >>>>   .identifiesControllerService(DBCPService.class)
> >>>>   .build();
> >>>>
> >>>> When your NiFi user sets up the flow, he or she puts a reference to
> >> NiFi's
> >>>> DBCPConnectionPool in it. In configuring that (a ControllerService,
> you
> >>>> tell it that it's Oracle, location, etc.)
> >>>>
> >>>> Then, your onTrigger() code would do something like this:
> >>>>
> >>>>   final DBCPService dbcpService = context.getProperty(DBCP_SERVI
> >>>> CE).asControllerService(DBCPService.class);
> >>>>
> >>>>
> >>>> and later...
> >>>>
> >>>>
> >>>>   try (final Connection connection = db

Re: Connection Issue

[Anil Rai]

Here is the behaviour that we have seen so forhope this helps

   1. When we run the java code in eclipse, it works and this is the
   connection object that is printed ->
   oracle.jdbc.driver.T4CConnection@6f75e721
   2. When we hard code all the values as mentioned in my first email in a
   custom processor, deploy that. It works as well. The above connection
   object gets printed.
   3. When we change the code in the custom processor to use the DBCP
   connection service, deploy that. The connection object that gets printed is
   jdbc:oracle:thin:@oged-scan.og.ge.com:1521/ORPOGPB1 and this does not
   work. aqSession = AQDriverManager.createAQSession(connection) fails
   saying cannot create AQSession.

Thanks
Anil


On Tue, Mar 14, 2017 at 4:13 PM, Oleg Zhurakousky <
ozhurakou...@hortonworks.com> wrote:

> Anil
>
> When you say "it does not like the connection object. . .” what do you
> mean by that?
> Can you please provide stack trace or some other details?
>
> Cheers
> Oleg
>
> > On Mar 14, 2017, at 4:06 PM, Anil Rai <anilrain...@gmail.com> wrote:
> >
> > Thanks Russ. Yes, we are doing exactly the same thing.
> >
> >driverClass = context.getProperty(DRIVER_CLASS).getValue();
> >queueName = context.getProperty(QUEUE_NAME).getValue();
> >databaseSchema = context.getProperty(DATABASE_SCHEMA).getValue();
> >consumerName = context.getProperty(CONSUMER_NAME).getValue();
> >eventName = context.getProperty(EVENT_NAME).getValue();
> >DBCPService connection =
> > context.getProperty(JDBC_CONNECTION_POOL).asControllerService(
> DBCPService.class);
> >Connection conn = connection.getConnection();
> > *aqSession = AQDriverManager.createAQSession(connection);*
> >
> > The underlined code above fails as it does not like the connection object
> > that is returned by the DBCPService.
> >
> > Regards
> > Anil
> >
> >
> > On Tue, Mar 14, 2017 at 2:43 PM, Russell Bateman <r...@windofkeltia.com>
> > wrote:
> >
> >> Anil,
> >>
> >> Typically, your custom processor should have a property, something like
> >>
> >>public static final PropertyDescriptor DBCP_SERVICE = new
> >> PropertyDescriptor.Builder()
> >>.name("Database Connection Pooling Service")
> >>.description("The Controller Service that is used to obtain
> >> connection to database")
> >>.required(true)
> >>.identifiesControllerService(DBCPService.class)
> >>.build();
> >>
> >> When your NiFi user sets up the flow, he or she puts a reference to
> NiFi's
> >> DBCPConnectionPool in it. In configuring that (a ControllerService, you
> >> tell it that it's Oracle, location, etc.)
> >>
> >> Then, your onTrigger() code would do something like this:
> >>
> >>final DBCPService dbcpService = context.getProperty(DBCP_SERVI
> >> CE).asControllerService(DBCPService.class);
> >>
> >>
> >> and later...
> >>
> >>
> >>try (final Connection connection = dbcpService.getConnection())
> >>{
> >>try (final Statement stmt = connection.createStatement(
> ResultSet.TYPE_FORWARD_ONLY,
> >> ResultSet.CONCUR_READ_ONLY))
> >>
> >> etc.
> >>
> >> Does this help?
> >>
> >> Russ
> >>
> >>
> >>
> >> On 03/14/2017 11:54 AM, Anil Rai wrote:
> >>
> >>> We have a use case to connect to oracle database and subscribe to
> Advanced
> >>> Queuing (https://docs.oracle.com/cd/A58617_01/server.804/a58241/
> ch_aq.htm
> >>> ).
> >>> Below is the java snippet to establish this connection from a java
> client.
> >>> We can run this in eclipse and consume message from the advanced queue.
> >>> **
> >>> Class.forName("oracle.jdbc.driver.OracleDriver");
> >>> connection = DriverManager.getConnection("
> >>> jdbc:oracle:thin:@xxx-yyy.zzz.com:1521/DB1","user", "pwd");
> >>> connection.setAutoCommit(true);
> >>> Class.forName("oracle.AQ.AQOracleDriver");
> >>> aqSession = AQDriverManager.createAQSession(connection);
> >>> System.out.println("AQ Session --->" + aqSession);
> >>> 
> >>>
> >>> We have created a custom processor in Nifi. This processor is getting
> the
> >>> connection string using getConnection function of Standard DBCP
> service.
> >>> The problem is, the connection object that is retrieved from eclipse
> >>> versus
> >>> what is returned from DBCP service is different. We have made sure we
> are
> >>> referring to the same jar both in eclipse and Nifi (ojdbc7.jar)
> >>> It fails @  aqSession = AQDriverManager.createAQSession(connection);
> >>> The connection object that comes from DBCP is not what is expected by
> >>> AQDriverManager.
> >>>
> >>> Any help is greatly appreciated.
> >>>
> >>> Thanks
> >>> Anil
> >>>
> >>>
> >>
>
>

Re: Request flow within a cluster

[Anil Rai]

Thank you.
Is there any documentation on the ideal cluster set up for production
environment? I saw the 3 zoo keepers on 3 VM's and 3 nodes on 3 VM's is the
best. Could you confirm?
Also the ideal system requirement for each node and each zoo keeper.in
terms of CPU, memory, disk space etc?
and finally archivinglogs, flow files etc and their frequency

Thanks
Anil


On Tue, Mar 7, 2017 at 3:11 PM, Bryan Bende <bbe...@gmail.com> wrote:

> Anil,
>
> The short answer is that each node in the cluster is running the
> entire data flow you see in UI, and NiFi does not transfer flow files
> between nodes unless the data flow is created to do that.
>
> This article describes approaches to distributing data across the
> cluster, which is dependent on how the data is entering the cluster:
> https://community.hortonworks.com/articles/16120/how-do-i-
> distribute-data-across-a-nifi-cluster.html
>
> Thanks,
>
> Bryan
>
>
> On Tue, Mar 7, 2017 at 3:03 PM, Anil Rai <anilrain...@gmail.com> wrote:
> > Thanks Andy. I am looking for the processing of a flow file.If i create a
> > flow, with multiple processor, how does this request flow in nifi in a
> > cluster.
> >
> >
> > On Tue, Mar 7, 2017 at 2:58 PM, Andy LoPresto <alopre...@apache.org>
> wrote:
> >
> >> Anil,
> >>
> >> Are you talking about processing a flowfile or responding to a REST HTTP
> >> request? The Admin Guide section on clustering [1] offers more
> information
> >> but if you can clarify your question you may get better answers from the
> >> mailing list.
> >>
> >> [1] https://nifi.apache.org/docs/nifi-docs/html/administration-
> >> guide.html#clustering
> >>
> >>
> >> Andy LoPresto
> >> alopre...@apache.org
> >> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >>
> >> On Mar 7, 2017, at 11:48 AM, Anil Rai <anilrain...@gmail.com> wrote:
> >>
> >> Team,
> >>
> >> Is there any documentation that explains how Nifi processes a request
> once
> >> it reach nifi in a cluster settings?
> >>
> >> Regards
> >> Anil
> >>
> >>
> >>
>

Re: Request flow within a cluster

[Anil Rai]

Thanks Andy. I am looking for the processing of a flow file.If i create a
flow, with multiple processor, how does this request flow in nifi in a
cluster.


On Tue, Mar 7, 2017 at 2:58 PM, Andy LoPresto <alopre...@apache.org> wrote:

> Anil,
>
> Are you talking about processing a flowfile or responding to a REST HTTP
> request? The Admin Guide section on clustering [1] offers more information
> but if you can clarify your question you may get better answers from the
> mailing list.
>
> [1] https://nifi.apache.org/docs/nifi-docs/html/administration-
> guide.html#clustering
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Mar 7, 2017, at 11:48 AM, Anil Rai <anilrain...@gmail.com> wrote:
>
> Team,
>
> Is there any documentation that explains how Nifi processes a request once
> it reach nifi in a cluster settings?
>
> Regards
> Anil
>
>
>