t 23, 2018 1:28 PM
To: users@nifi.apache.org
Subject: Re: ConvertAvroSchema with unmapped fields
Hi Dave,
Can you provide the two schemas (redact anything necessary). There is a way to
specify an “optional” field [1] by setting the type to an array of null and the
type you support. Yo
Hi - I've got a scenario where I'm trying to convert the implicit AVRO schema
associated with a database record to a different AVRO schema. I'm trying to use
the ConvertAvroSchema processor, but it won't validate because there is an
'unmapped' field that exists in the second schema but not the
to NiFi 1.6.0 this was a bug [1], please see the Jira for more details
(and an example).
Regards,
Matt
[1] https://issues.apache.org/jira/browse/NIFI-4893
On Thu, Aug 23, 2018 at 2:29 PM David Gallagher
wrote:
>
> Hi Andy – this is the schema we’re getting from the database query:
>
Hi - I have a dev registry server (1.7.1) set up with git, and am trying to use
git bundle to move the buckets and flows to a test registry instance (I'm using
git bundle because there's going to be an air gap in production). Everything
looks fine, but when I start the production registry it
pache.org>>
Subject: Re: Nifi Registry + Git: target registry ignores git flows
It probably has to do with the database the registry used not being migrated to
your new environment. Try setting up a snapshot of that too.
On Sat, Sep 8, 2018 at 1:06 PM David Gallagher
mailto:dgallag...@cle
Hi – not sure if it helps, but you can set a static hostname in your
docker-compose.
Thanks,
Dave
From: Juan Pablo Gardella
Sent: Sunday, September 23, 2018 3:43 PM
To: users@nifi.apache.org
Subject: Nifi with docker and LDAP
Hi all,
I'm using Nifi with docker and it's secure.
I'm facing
that one, but its still hard to know if this is really
> the right service. What if it was for a different database and someone
> didn't realize?
>
> -Bryan
>
> On Thu, Nov 29, 2018 at 12:34 PM David Gallagher
> wrote:
> >
> > Hi - I'm using nifi-1.7.1 and nifi-registry-0.
Hi - I'm using nifi-1.7.1 and nifi-registry-0.2.0. I'd like to have 'global'
DBCPConnectionPool instances at the Nifi Flow level, then import flows from
the registry and have them use the global pools, e.g. in a PutDatabaseRecord
processor. When I try that, though, the processor is invalid and
t it is also unclear how to select the
>>> > appropriate one. What if there were multiple DBCP connection pools
>>> > visible to where the versioned flow is being imported? There would be
>>> > no way to know which one to use.
>>> >
>>> >
’t exist
because the bucket id and flow id are not the same.
On Sep 11, 2018, at 4:55 PM, David Gallagher
mailto:dgallag...@cleverdevices.com>> wrote:
Ok, Last question on this one (promise!). I’ve set up the dev / git repository
and it’s working great. I can export flows from dev
[1]
https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#system-requirements
On Wed, Sep 12, 2018 at 6:40 PM, David Gallagher
wrote:
> Hi – ok, now I’ve got a different (but maybe related?) problem. I’ve
> set up the nifi registry on a remote server with a git reposito
Just following up for the benefit of others - git works just fine with a Linux
VM. Thanks for all your help!
Dave
-Original Message-
From: David Gallagher
Sent: Monday, September 17, 2018 1:38 PM
To: users@nifi.apache.org
Subject: RE: Nifi Registry + Git: target registry ignores git
outside of the NiFi Registry API/UI
I hope this helps you get on the right track. Let me know if you need further
assistance.
Cheers,
Kevin
From: David Gallagher
mailto:dgallag...@cleverdevices.com>>
Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
mailto:users@n
host as
the user that nifi registry runs as, you should be able to push using ssh+git
instead of username / password. then NiFi Registry will be able to do the same.
On Thu, Mar 28, 2019 at 12:09 PM David Gallagher
wrote:
>
> Hi – does anyone know if it is possible to somehow e
Hi - does anyone know if it is possible to somehow encrypt the providers.xml
file? I'm using the Git persistence provider and want to store the remote user
and password there, but I don't want to leave that value in there unencrypted
if I don't have to. I downloaded the nifi-toolkit, but I
for this to work (your suspicion was correct!). So we recommend doing that and
securing the key using OS filesystem permissions (ie, only the run as user
should have read access to that ssk key file)
On Thu, Mar 28, 2019 at 3:54 PM David Gallagher
wrote:
>
> Thanks for the reply! I've set
talking about.
Thanks
On Fri, May 17, 2019 at 12:30 PM David Gallagher
mailto:dgallag...@cleverdevices.com>> wrote:
Hi – I have a requirement to read one message off RabbitMQ and fully process
that message before I take another message off the queue. I found some great
guidance to sett
Hi - I have a requirement to read one message off RabbitMQ and fully process
that message before I take another message off the queue. I found some great
guidance to setting up a 'traffic light' pattern here
(https://gist.github.com/ijokarumawak/9e1a4855934f2bb9661f88ca625bd244), but it
won't
Jf0%2FvM%2F4VPKe0z916zcHyVx5UpgzCttu2yUZbCbk%3D=0>
Thanks
Shawn
Sent from my iPhone
On May 17, 2019, at 1:59 PM, David Gallagher
mailto:dgallag...@cleverdevices.com>> wrote:
Hi Joe – thanks for answering. I can actually live with the message ack at the
moment. But, I can’t have two mess
ype":"long",
"logicalType":"timestamp-millis"} }
Thanks
-Mark
On Jun 6, 2019, at 11:21 AM, David Gallagher
mailto:dgallag...@cleverdevices.com>> wrote:
Hi – I’ve got an incoming JSON message with a timestamp that I want to
validate. I have a Valid
Hi - I've got an incoming JSON message with a timestamp that I want to
validate. I have a ValidateRecord (1.8.0) processor set up with a
JSONPathReader for the message. The relevant field is defined in the schema as:
{"name": "activationDate", "type": { "type":"string",
illis" only for a long field, not
for a String field.
So I think you'd need to use:
{"name": "activationDate", "type": { "type":"long",
"logicalType":"timestamp-millis"} }
Thanks
-Mark
On Jun 6, 2019, at 11:21
are trying to change the same
key because the Wait sees the value as it flip/flops.
Thanks
Shawn
From: David Gallagher
mailto:dgallag...@cleverdevices.com>>
Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
mailto:users@nifi.apache.org>>
Date: Wednesda
Hi - I have a gate set up with a Wait processor listening for a target signal
count of '1'. I have several Notify processors that set the value to '1' when
the system is able to take work. If there is work waiting in the queue while
the 'gate' is closed, I sometimes get this error:
Timestamps
David,
Can you send a template of your flow and a sample piece of data?
Thanks
-Mark
On Jun 6, 2019, at 11:47 AM, David Gallagher
mailto:dgallag...@cleverdevices.com>> wrote:
Thanks, Mark. I tried that but it causes even messages with the correct format
to be rej
the full user experience here. I'm not certain if
this processor (ExecuteStreamCommand) is like this but hopefully that makes
sense.
In short, the behavior you're describing should be treated like a bug and the
processor logic should be improved.
Thanks
On Fri, Apr 26, 2019 at 12:12 PM David
Hi - I'm working on a flow with lots of processors and tons of reasons that
something could fail (failure to match schema, bad return codes, file not
found, etc). The intent is to send these failures to a ProduceAMQP processor
with an appropriate routing key and error message. It makes the flow
just a preference.
On Wed, Apr 24, 2019 at 12:16 PM David Gallagher
wrote:
>
> Hi - I'm working on a flow with lots of processors and tons of reasons that
> something could fail (failure to match schema, bad return codes, file not
> found, etc). The intent is to send th
Hi - I have an issue where certain processors may hang due to misconfiguration.
For example, an ExecuteStreamCommand with an incorrect path for the executable.
In this scenario, the flowfile waits in the queue for the processor and cannot
proceed. It generates an error message, both visually
Hi - I'm looking to remove records from a table using the results returned from
a QueryDatabaseTable processor. I want to do this in batches because it's going
to require a lot of deletes, and flowfile per record would be too slow. I've
tried using the PutDatabaseRecord processor, but the
before PutDatabaseRecord
to remove the unindexed / unnecessary fields, then the generated statement
should only include the necessary fields and be able to take advantage of any
indexes.
Regards,
Matt
On Fri, Sep 20, 2019 at 4:55 PM David Gallagher
wrote:
>
> Hi – I’m looking to
Hi - I'm currently converting our flows from 1.8.0 to 1.11.1, with a parallel
upgrade from registry 0.2.0 to registry 0.5.0. Currently stuck due to this set
of steps:
1. Upgraded registry from 0.2.0 to 0.5.0
2. Upgraded NiFi from 1.8.0 to 1.11.1
3. Imported a new version of a flow to
iFi 1.8.0 to 1.11.1 upgrade issue
Hello,
Can you elaborate on what step 3 means? Where did this new version come from
that was imported to registry?
-Bryan
On Thu, Feb 13, 2020 at 9:11 AM David Gallagher
wrote:
>
> Hi – I’m currently converting our flows from 1.8.0 to 1.11.1, with a
be scheduled on the primary node,
and maybe you previously had it schedule on all nodes?
On Thu, Feb 13, 2020 at 9:57 AM David Gallagher
wrote:
>
> Hi Bryan -
>
> The new version of the flow came from a development environment. Thinking
> back on it, I upgraded that development envir
269sdata=Iy9%2FN5O
> cRxfE7E78NOFWkxj6xX3uGKX7fseKPvFSqI0%3Dreserved=0
>
> On Thu, Feb 13, 2020 at 11:55 AM David Gallagher
> wrote:
> >
> > @Bryan Bende, to add to this it looks like 'All' was an option in 1.8.0 but
> > is no longer an option in 1.11.1. This is why it
ubject: Re: Loss of state on QueryDatabaseTable during upgrade
>
>
>
> Hi David,
>
>
>
> How did you perform the upgrade? Do you have a cluster with external ZK, a
> cluster with embedded ZK, or a single node?
>
>
>
> Thanks,
>
> Pierre
>
>
>
>
Hi - I'm upgrading flows from 1.8.0 to 1.11.1. Everything is working nicely,
but it looks like I may have lost the State values of my QueryDatabaseTable
processors (I'm using a Maximum ID). I've looked in the state file
(\state\local\snapshot) and the value is there, but it's not showing in the
the upgrade? Do you have a cluster with external ZK, a
cluster with embedded ZK, or a single node?
Thanks,
Pierre
Le jeu. 6 févr. 2020 à 09:15, David Gallagher
mailto:dgallag...@cleverdevices.com>> a écrit :
Hi – I’m upgrading flows from 1.8.0 to 1.11.1. Everything is working nicely,
but it looks
Actually this is probably on me - In the config file I specified the path to
state as ..\..\state, not ..\state. I'll be able to test again in a little
while and see.
-Original Message-
From: David Gallagher
Sent: Thursday, February 6, 2020 10:07 AM
To: users@nifi.apache.org; Bryan
Bryan / Pierre - problem was definitely on my end. Can confirm that state is
preserved during upgrade. Thanks for your help!
-Original Message-
From: David Gallagher
Sent: Thursday, February 6, 2020 11:44 AM
To: users@nifi.apache.org; Bryan Bende
Subject: RE: Loss of state
to do this
automatically, unfortunately it doesn't look like there's a PUT option for
state.
Thanks,
Dave
-Original Message-
From: David Gallagher
Sent: Thursday, February 13, 2020 3:32 PM
To: users@nifi.apache.org
Subject: RE: NiFi 1.8.0 to 1.11.1 upgrade issue
Yes, recreating the PG
Hi - we have a setup with a single Development registry and multiple Production
NiFi Registries. There's an air gap between the environments. Our intent is to
work on flows in the Development environment, export the flows, and then deploy
them to production environments. We're finding now that
42 matches
Mail list logo
