r performance), then for each flow file use the custom
> processors/script to read the value of the field where the Avro blob is,
> and overwrite the flow file content with that value, then send all of these
> to a MergeRecord.
>
> -Bryan
>
>
> On Mon, Sep 14,
Anyone have thoughts on this? Essentially we have binary avro stored as a
BLOB in Oracle, and I want to extract it via Nifi and read and write out
the contents.
Thanks,
Jason
On Mon, Aug 17, 2020 at 10:04 AM Jason Iannone wrote:
> Hi all,
>
> I have a scenario where an Avro binary
Hi all,
I have a scenario where an Avro binary is being stored as a BLOB in an
RDBMS. What's the recommended approach for querying this in bulk,
extracting this specific field, and batching it to HDFS?
1. GenerateTableFetch OR QueryDatabaseTableRecord
2. Extract Avro column and assemble
ext to maximize the odds if things start to be more clear
> so we can help more effectively.
>
> thanks
>
> On Fri, Jul 3, 2020 at 7:47 AM Jason Iannone wrote:
>
>> Hi Joe,
>>
>> We've seen the issue with stock ConsumeKafka_2_0 against Nifi 1.10 and
>> 1.11
me matches any of the names in ConsumerConfig, and if
> so it adds the key/value to the config map.
>
> So since the property is named "max.poll.records", that is the same name
> in ConsumerConfig.
>
> Thanks,
>
> Bryan
>
> On Tue, Jul 7, 2020 at 9:32 AM J
Hi all,
I have been digging through the ConsumeKafka_2_0 (and record) code and
noticed that the PropertyDescriptor for MAX_POLL_RECORDS isn't connected to
anything. Is this intentional, and a "deprecated field" or am I missing
something?
Thanks,
Jason
ust
> refactor this and properly use the properties explicitly. The current
> implementation is a bit “magical” and not at all straight forward.
>
>
> On Jul 7, 2020, at 9:58 AM, Jason Iannone wrote:
>
> Hi Bryan,
>
> Thanks, I completely missed that! It also makes more sens
ted to zk.
>
> you mention a fork of nifi for consume kafka. have you tried using stock
> items/albeit without whatever feature you needed so you can narrow in on
> the problem?
>
> joe
>
> On Fri, Jul 3, 2020 at 7:16 AM Jason Iannone wrote:
>
>> What role does Zookeep
ed all traces of custom code what are peoples thoughts on
possible causes? Could this be an OS issue, or are there any known issues
with specific versions of RHEL?
Logically I think it makes sense to remove JSON from the equation as a
whole.
Thanks,
Jason
On Wed, Jun 24, 2020 at 2:54 PM Jason I
I spoke too soon, and must be the magic of sending an email! We found what
appears to be corrupted content and captured the binary, hoping to play it
through the code and see what's going on.
Thanks,
Jason
On Mon, Jun 22, 2020 at 4:35 PM Jason Iannone wrote:
> Hey Mark,
>
> We hit
d together by MergeContent and see if
> any of those is corrupt.
>
> Thanks
> -Mark
>
> [1]
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data_provenance
>
> On Jun 10, 2020, at 2:07 PM, Jason Iannone wrote:
>
> Hey Mark,
>
> I was think
ocessor. And when
> you see which processor resulted in corruption, you can easily download the
> data as it looks when it went into the processor to make it easy to
> re-ingest and test.
>
> Thanks
> -Mark
>
>
> On Jun 22, 2020, at 4:46 PM, Jason Iannone wrote:
>
&g
doesn’t always happen on
> the same input, it sounds like a threading/concurrency bug. Do you have a
> buffer or anything like that as a member variable?
>
> On Jun 22, 2020, at 10:02 PM, Jason Iannone wrote:
>
> I'm now thinking its due to how we handled reading the flowfile cont
Hi all,
We will have multiple processors which will be integrating with various
sources and sinks (i.e. Kafka, HDFS, RDBMS) that will all have differing
credentials and tickets (same cache location). At this point keytabs are
not an option as the ticket/credential cache is more secure due to more
Hi all,
Within Nifi 1.10.0 we're seeing unexpected behavior with mergecontent. The
processor is being fed in many flowfiles with individual JSON records. The
records have various field types including a hex-encoded byte[]. We are not
trying to merge JSON records themselves but rather consolidate
s pretty straight forward and hasn’t been modified in over
> 3 years, so I would expect to see it happen often if it were a bug in the
> MergeContent processor itself. Any chance that you can create a flow
> template/sample data that recreates the issue? Anything particularly unique
>
> -Mark
>
>
> > On Jun 9, 2020, at 6:47 PM, Jason Iannone wrote:
> >
> > Hi all,
> >
> > Within Nifi 1.10.0 we're seeing unexpected behavior with mergecontent.
> The processor is being fed in many flowfiles with individual JSON records.
> The records hav
at the FlowFiles that were joined together by MergeContent and see if
> any of those is corrupt.
>
> Thanks
> -Mark
>
> [1]
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data_provenance
>
> On Jun 10, 2020, at 2:07 PM, Jason Iannone wrote:
>
> Hey
g to that FlowFile.
>>
>> My recommendation to track this down would be to find a FlowFile that is
>> corrupt, and then use the data provenance feature [1] to view its lineage.
>> Look at the FlowFiles that were joined together by MergeContent and see if
>> any of those
ded ability
> to get the InputStreams and OutputStreams directly and callers can just use
> try-with-resources. This is probably preferred now for most cases just
> because it results in cleaner code.
>
> Thanks
> -Mark
>
> On Jun 11, 2020, at 10:43 AM, Jason Iannone wrote:
inside the jvm for your specific scenario.
>
> Sent from my Verizon, Samsung Galaxy smartphone
> Get Outlook for Android <https://aka.ms/ghei36>
>
> ------
> *From:* Jason Iannone
> *Sent:* Tuesday, June 30, 2020 10:36:02 PM
> *To:* users@n
21 matches
Mail list logo
