Matt, you rock!! thank you!!
On Tue, Aug 7, 2018 at 5:16 PM Matt Burgess wrote:
> Sounds good, it makes the underlying code a bit more complicated but I see
> from y’all’s points that a “separate” processor is a better user
> experience. I’m knee deep in it as we speak, hope to have a PR up in
Sounds good, it makes the underlying code a bit more complicated but I see from
y’all’s points that a “separate” processor is a better user experience. I’m
knee deep in it as we speak, hope to have a PR up in a few days.
Thanks,
Matt
> On Aug 7, 2018, at 5:07 PM, Andrew Grande wrote:
>
>
I'd really like to see the Record suffix on the processor for
discoverability, as already mentioned.
Andrew
On Tue, Aug 7, 2018, 2:16 PM Matt Burgess wrote:
> Yeah that's definitely doable, most of the logic for writing a
> ResultSet to a Flow File is localized (currently to JdbcCommon but
>
I commented the jira about that. We do have validators, or at least the
start of some, but they are in the serialization libraries and would need
to be moved to be usable by both processors and records.
On August 7, 2018 at 13:58:27, Ryan Hendrickson (
ryan.andrew.hendrick...@gmail.com) wrote:
Yeah that's definitely doable, most of the logic for writing a
ResultSet to a Flow File is localized (currently to JdbcCommon but
also in ResultSetRecordSet), so I wouldn't think it would be too much
refactor. What are folks thoughts on whether to add a Record Writer
property to the existing
now this is really slick! thanks Mark for educating me!
On Tue, Aug 7, 2018 at 1:15 PM Mark Payne wrote:
> Boris,
>
> Using a Record-based processor does not mean that you need to define a
> schema upfront. This is
> necessary if the source itself cannot provide a schema. However, since it
> is
Matt,
Would extending the core ExecuteSQL processor with an ExecuteSQLRecord
processor also work? I wonder about discoverability if only one processor is
present and in other places we explicitly name the processors which handle
records as such. If the ExecuteSQL processor handled all the SQL
Yea, just finished testing, it works.. I added an additional property
called "Date Format" and I passed in a java date format there. Now I'm
just wondering if there's a way to validate it's a good date format before
it tries to use it when it's passing docs through. Maybe a custom
validator?
Haha, yea I'm testing out some code-hacking here to see what I can do too.
If it works, I can try to submit a Pull Request for it. I haven't done one
before, and this seems pretty easy to make it configurable.
Ryan
On Tue, Aug 7, 2018 at 1:48 PM Mike Thomsen wrote:
> Add, I should have
Add, I should have read your comment on the Jira ticket because it's
even easier than that!
On Tue, Aug 7, 2018 at 1:47 PM Mike Thomsen wrote:
> I just checked the code, and it's using the default Jackson mapping
> behavior for that. The Mongo driver returns a Date, and looks like Jackson
>
I just checked the code, and it's using the default Jackson mapping
behavior for that. The Mongo driver returns a Date, and looks like Jackson
is just turning that into an ISO8601 string without that level of
precision. A custom mapper for Date objects should be able to solve that.
I'll work it
Bryan,
Thanks, that confirms what I was thinking would be the case.
Mike
On Tue, Aug 7, 2018 at 12:54 PM Andrew Grande wrote:
> There are also registry event hooks to help with the automation.
>
> Andrew
>
>
> On Tue, Aug 7, 2018, 12:46 PM Bryan Bende wrote:
>
>> Mike,
>>
>> It is not really
As a side note, one has to ha e a serious justification _not_ to use
record-based processors. The benefits, including performance, are too
numerous to call out here.
Andrew
On Tue, Aug 7, 2018, 1:15 PM Mark Payne wrote:
> Boris,
>
> Using a Record-based processor does not mean that you need to
Boris,
Using a Record-based processor does not mean that you need to define a schema
upfront. This is
necessary if the source itself cannot provide a schema. However, since it is
pulling structured data
and the schema can be inferred from the database, you wouldn't need to. As Matt
was saying,
Hi all,
I'm using GetMongo configured with JSON Type of "Standard JSON". The
document I've got in Mongo has a date field that looks like the following:
{
...
"date" : ISODate("2018-08-06T16:20:10.912Z"
...
}
When GetMongo spits it out, the date comes out as:
There are also registry event hooks to help with the automation.
Andrew
On Tue, Aug 7, 2018, 12:46 PM Bryan Bende wrote:
> Mike,
>
> It is not really made to have multiple instances pointing at the same git
> repo.
>
> Each registry has a metadata DB and a flow storage component which can
> be
Matt thanks for the quick reply.
So I changed the attribute name and set it to now()
and it worked.
I then changed it back to ${now():format("-MM-dd HH:mm",'Z')}
and it worked!!! too
Either I did have something funky with my attribute name on the spec or I
wonder if the setting in
Mike,
It is not really made to have multiple instances pointing at the same git repo.
Each registry has a metadata DB and a flow storage component which can
be filesystem or git.
So the metadata DB won't know about the stuff created in the other instance.
Generally it is either a single shared
thanks for all the responses! it means I am not the only one interested in
this topic.
Record-aware version would be really nice, but a lot of times I do not want
to use record-based processors since I need to define a schema for
input/output upfront and just want to run SQL query and get
Has anyone tried having two or more registry instances pointing to the same
repo and keeping them in sync?
We have a NiFi deployment where it would be an easier sell to have 3
instances of the registry sharing the same repo than to have one instance
that is a big exception to the network security
Juan,
What version of NiFi are you using? I tried with the latest master,
using a GenerateFlowFile (where I set the attribute using the now()
expression) into JoltTransformJSON with your spec above, and it works
fine. Are you sure there's not a double-quote missing somewhere, or
something extra
Hello all,
I am trying to "enrich" my json.
My problem happens when I try to give it a timestamp.
My requirement is to give it time in this format:
${now():format("-MM-dd HH:mm",'Z')}
So at the top level and every element of my json should have an entry
called
EX:
"createTime":"2018-08-07
The variable registry is hierarchical, so variables defined in a
parent group will be visible to all children, unless a child group has
a variable with the same name which will override the parent's value.
So you can put any common variables in the root group.
On Tue, Aug 7, 2018 at 9:40 AM, l
What's about level higher - common variables for several process groups?
Thanks.
On Mon, Aug 6, 2018 at 7:06 PM, Charlie Meyer <
charlie.me...@civitaslearning.com> wrote:
> You might want to take a look at utilizing the variable registry for this:
>
I'm definitely interested in supporting a record-aware version as well
(I wrote the Jira up last year [1] but haven't gotten around to
implementing it), however I agree with Peter's comment on the Jira.
Since ExecuteSQL is an oft-touched processor, if we had two processors
that only differed in
I would also add that the pattern of splitting to 1 record per flow
file was common before the record processors existed, and generally
this can/should be avoided now in favor of processing/manipulating
records in place, and keeping them together in large batches.
On Tue, Aug 7, 2018 at 9:10
Careful, that makes too much sense, Joe ;)
On Tue, Aug 7, 2018, 8:45 AM Joe Witt wrote:
> i think we just need to make an ExecuteSqlRecord processor.
>
> thanks
>
> On Tue, Aug 7, 2018, 8:41 AM Mike Thomsen wrote:
>
>> My guess is that it is due to the fact that Avro is the only record type
>>
You could look at using ExecuteStreamCommand + JQ. We use this pattern a
lot...
Regards
Dano
On Tue, Aug 7, 2018, 5:09 AM Mahendra prabhu
wrote:
> Expected behaivor is similiar to this task created -
> https://issues.apache.org/jira/browse/NIFI-4398 I couldnt achieve this
> using
i think we just need to make an ExecuteSqlRecord processor.
thanks
On Tue, Aug 7, 2018, 8:41 AM Mike Thomsen wrote:
> My guess is that it is due to the fact that Avro is the only record type
> that can match sql pretty closely feature to feature on data types.
> On Tue, Aug 7, 2018 at 8:33 AM
My guess is that it is due to the fact that Avro is the only record type
that can match sql pretty closely feature to feature on data types.
On Tue, Aug 7, 2018 at 8:33 AM Boris Tyukin wrote:
> I've been wondering since I started learning NiFi why ExecuteSQL processor
> only returns AVRO
I've been wondering since I started learning NiFi why ExecuteSQL processor
only returns AVRO formatted data. All community examples I've seen then
convert AVRO to json and pretty much all of them then split json to
multiple flows.
I found myself doing the same thing over and over and over again.
Expected behaivor is similiar to this task created -
https://issues.apache.org/jira/browse/NIFI-4398 I couldnt achieve this
using ConvertRecord. Can you guide to proceed further to handle nested json
elements ?
Thanks in advance
On Tue, Aug 7, 2018 at 4:31 PM Mahendra prabhu
wrote:
> Hi
Hi Pierre,
I need to move nested json data into SQL table after using expression
language. So I thought to use this processor, and do transformation for the
resultant CSV and move to SQL.
Regards,
Prabhu Mahendran
On Tue, Aug 7, 2018 at 3:59 PM Pierre Villard
wrote:
> Hi,
>
> I believe it can
Hi,
I believe it can be closed because it's doable with ConvertRecord processor
(which is the recommended approach).
Does it answer your requirement ?
Thanks,
Pierre
2018-08-07 11:43 GMT+02:00 Mahendra prabhu :
> Hi Folks,
>
> Is there is plan to include this feature or dropped -
>
Hi Folks,
Is there is plan to include this feature or dropped -
https://issues.apache.org/jira/browse/NIFI-1583 ?
Is this PR closed because of licensing or incomplete rebase of code ?
Regards,
Prabhu Mahendran
35 matches
Mail list logo
