Re: Parsing a template to identify processor names

[Koji Kawamura]

Hello,

I'm not sure if this is what you're looking for, but I wrote a test
case before that loads template and use flow information to automate
tests.
The code can be a reference.
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/test/java/org/apache/nifi/atlas/reporting/ITReportLineageToAtlas.java#L100

Thanks,
Koji
On Wed, Nov 14, 2018 at 7:23 AM Vitaly Krivoy
 wrote:
>
> If a process group template contains multiple processors of the same type, I 
> need to make sure that they have different names. Is there an easier way of 
> parsing a template outside of instantiating it and then getting all 
> processors from a resulting process group? I do already have the code to do 
> the latter, but I am curious if there is another way. Thanks.
>
>
>
> STATEMENT OF CONFIDENTIALITY The information contained in this email message 
> and any attachments may be confidential and legally privileged and is 
> intended for the use of the addressee(s) only. If you are not an intended 
> recipient, please: (1) notify me immediately by replying to this message; (2) 
> do not use, disseminate, distribute or reproduce any part of the message or 
> any attachment; and (3) destroy all copies of this message and any 
> attachments.

Parsing a template to identify processor names

[Vitaly Krivoy]

If a process group template contains multiple processors of the same type, I 
need to make sure that they have different names. Is there an easier way of 
parsing a template outside of instantiating it and then getting all processors 
from a resulting process group? I do already have the code to do the latter, 
but I am curious if there is another way. Thanks.

STATEMENT OF CONFIDENTIALITY The information contained in this email message 
and any attachments may be confidential and legally privileged and is intended 
for the use of the addressee(s) only. If you are not an intended recipient, 
please: (1) notify me immediately by replying to this message; (2) do not use, 
disseminate, distribute or reproduce any part of the message or any attachment; 
and (3) destroy all copies of this message and any attachments.

Re: ConsumeKafkaRecord won't pull new events from Kafka

[Mike Thomsen]

I had it running and was adding new content to the queue that other
consumers were able to see.

On Tue, Nov 13, 2018 at 1:15 PM Mark Payne  wrote:

> Mike,
>
> Is there new data coming into the Kafka topic? By default, when the
> Processor is started, it uses
> an auto commit offset of 'latest'. So that means that if you started the
> Processor with this setting,
> the commit offset is saved pointing to the end of the topic. So if no more
> data is coming into the topic,
> you'll not see anything out of the processor, while the ConsumeKafka
> processor was started with the
> offset at 'earliest'?
>
> Thanks
> -Mark
>
> On Nov 13, 2018, at 12:54 PM, Mike Thomsen  wrote:
>
> That would appear to be the case. So here's what I was doing:
>
> 1. Used this sort of code to serialize the Avro:
>
> private byte[] serialize(Object obj, Class clz) throws Exception {
> SpecificDatumWriter writer = new SpecificDatumWriter<>(clz);
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(out,
> null);
> //writer.write(obj, encoder);
> //encoder.flush();
> out.close();
>
> return out.toByteArray();
> }
>
> So that got me a byte array with just the binary representation.
>
> 2. Produced that to the Kafka topic.
>
> 3. Had an AvroReader variously configured to use schema.name hard set to
> an entry on the AvroSchemaRegistry or EmbeddedSchema.
>
> Didn't see any flowfiles get emitted to parse.failure or success.
>
> On Tue, Nov 13, 2018 at 12:50 PM Joe Witt  wrote:
>
>> Mike - so does this mean the parse.failure relationship wasn't working
>> though?  We should try to dig into this more if you're up for it or
>> sharing more details.  We dont want the behavior you ran into for
>> sure...
>> On Tue, Nov 13, 2018 at 12:49 PM Mike Thomsen 
>> wrote:
>> >
>> > So after a lot of diving into 1.9.X, it **appears** that there was some
>> sort of combination w/ the AvroReader + ConsumeKafkaRecord that was
>> suppressing errors from being reported. Haven't been able to fully figure
>> out what was doing on, but I know one of the contributing factors was that
>> my producer (spring boot service, not NiFi) was incorrectly building the
>> binary form of the Avro (didn't have embedded schema).
>> >
>> > Mike
>> >
>> > On Tue, Nov 13, 2018 at 12:33 PM Joe Witt  wrote:
>> >>
>> >> Viking
>> >>
>> >> Just to clarify it isn't that it is trying to merge records to create
>> >> larger flow files but rather it is 'playing the kafka interaction as
>> >> it lies' so to speak.  When polling from Kafka we can get one or more
>> >> records.  We're just taking advantage of that without trading off
>> >> speed.
>> >>
>> >> Thanks
>> >> On Tue, Nov 13, 2018 at 12:24 PM Viking K 
>> wrote:
>> >> >
>> >> > Mike, are we talking about complete avro messages or bare records
>> (schema is contained inside the avro file  or do they use a schema registry)
>> >> >
>> >> > From my own testing the ConsumeKafkaRecord tries to bundle in the
>> incoming messages to create larger flow files.
>> >> > Do you use any Kafka headers in the processor?
>> >> >
>> >> > Also what happens if you try to replicate the behavior of
>> ConsumeKafkaRecord like this. I don't know if you need the ConvertRecord
>> but it might be needed to pick out a schema name to use as Merge strategy.
>> >> > ConsumeKafka -> (ConvertRecord) -> Merge Content
>> >> >
>> >> > /Viking
>> >> > 
>> >> > From: Mike Thomsen 
>> >> > Sent: Tuesday, November 13, 2018 3:02 PM
>> >> > To: users@nifi.apache.org
>> >> > Subject: Re: ConsumeKafkaRecord won't pull new events from Kafka
>> >> >
>> >> > Closest thing I see to something that implies something might be
>> awry is this in nifi-app.log:
>> >> >
>> >> > javax.management.InstanceAlreadyExistsException:
>> kafka.consumer:type=app-info,id=consumer-1
>> >> > at
>> com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
>> >> > at
>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
>> >> > at
>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
>> >> > at
>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
>> >> > at
>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
>> >> > at
>> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
>> >> > at
>> org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:57)
>> >> > at
>> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:640)
>> >> > at
>> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:512)
>> >> > at
>> 

Re: JsonPath expression language exception with QueryRecord processor

[Mandeep Gill]

Hi all,

Have created as an issue in Jira: NIFI-5817 (
https://issues.apache.org/jira/browse/NIFI-5817).

Thanks Mark, yes can confirm that adding `replaceNull` to the function
chain as described works around the issue - have updated the ticket to
reflect this.

Cheers,
Mandeep

On Tue, 13 Nov 2018 at 18:21 Mark Payne  wrote:

> Hi Mandeep,
>
> I think this may actually be a bug in QueryRecord, in the way that it's
> handling the Expression Language.
> Do please file a JIRA for that. In the meantime, you can probably work
> around the issue using the replaceNull() method.
> So if your expression was ${myAttribute:jsonPath('/hello')} you could
> instead use ${myAttribute:replaceNull('{}'):jsonPath('/hello')}
>
> Thanks
> -Mark
>
> On Nov 13, 2018, at 10:16 AM, Mandeep Gill  wrote:
>
> Hi there,
>
> I'm hitting an issue using the `jsonPath` expression language function to
> extract a query to use with the QueryRecord processor. The processor works
> fine if the expression language subject is contained within the process
> group variable registry, but fails upon starting with an
> AttributeExpressionLanguageException if the subject was expected to to
> exist within a flowfile attribute [1]
>
> I've attached a template generated on NiFi 1.8.0 demonstrating the problem
> - it only appears to be an issue with dynamic outputs from the QueryRecord
> processors, as the same expression language statement works fine when used
> as part of UpdateAttribute processor with the subject in a flowfile
> attribute as per the template. I've dug into the codebase and can trace the
> error to the evaluate function within the JsonPathEvaluator class, which
> throws the exception if the variable can not be referenced. I have a local
> fix at https://github.com/apache/nifi/compare/master...nstack:fix/jsonpath 
> that
> returns StringQueryResult("") if the subject is empty instead of throwing
> the exception and this appears to work however I wonder that as
> UpdateAttribute works if the problem is instead in QueryRecord eagerly
> evaluating the queries.
>
> Any help would be appreciated.
>
> Cheers,
> Mandeep
>
> [1] 2018-11-13 14:46:24,899 ERROR [Timer-Driven Process Thread-1]
> o.a.nifi.processors.standard.QueryRecord
> QueryRecord[id=0d5684e2-0167-1000-74c1-eb29a1401981] Failed to properly
> initialize Processor. If still scheduled to run, NiFi will attempt to
> initialize and run the Processor again after the 'Administrative Yield
> Duration' has elapsed. Failure is due to
> java.lang.reflect.InvocationTargetException:
> java.lang.reflect.InvocationTargetException
> java.lang.reflect.InvocationTargetException: null
> at sun.reflect.GeneratedMethodAccessor916.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
> at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
> at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75)
> at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52)
> at
> org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$4(StandardProcessorNode.java:1499)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException:
> Subject is empty
> at
> org.apache.nifi.attribute.expression.language.evaluation.functions.JsonPathEvaluator.evaluate(JsonPathEvaluator.java:66)
> at
> org.apache.nifi.attribute.expression.language.Query.evaluate(Query.java:315)
> at
> org.apache.nifi.attribute.expression.language.Query.evaluateExpression(Query.java:203)
> at
> org.apache.nifi.attribute.expression.language.CompiledExpression.evaluate(CompiledExpression.java:58)
> at
> org.apache.nifi.attribute.expression.language.StandardPreparedQuery.evaluateExpressions(StandardPreparedQuery.java:51)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:148)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:113)
> at
> 

Re: JsonPath expression language exception with QueryRecord processor

[Mark Payne]

Hi Mandeep,

I think this may actually be a bug in QueryRecord, in the way that it's 
handling the Expression Language.
Do please file a JIRA for that. In the meantime, you can probably work around 
the issue using the replaceNull() method.
So if your expression was ${myAttribute:jsonPath('/hello')} you could instead 
use ${myAttribute:replaceNull('{}'):jsonPath('/hello')}

Thanks
-Mark

On Nov 13, 2018, at 10:16 AM, Mandeep Gill 
mailto:mand...@nstack.com>> wrote:

Hi there,

I'm hitting an issue using the `jsonPath` expression language function to 
extract a query to use with the QueryRecord processor. The processor works fine 
if the expression language subject is contained within the process group 
variable registry, but fails upon starting with an 
AttributeExpressionLanguageException if the subject was expected to to exist 
within a flowfile attribute [1]

I've attached a template generated on NiFi 1.8.0 demonstrating the problem - it 
only appears to be an issue with dynamic outputs from the QueryRecord 
processors, as the same expression language statement works fine when used as 
part of UpdateAttribute processor with the subject in a flowfile attribute as 
per the template. I've dug into the codebase and can trace the error to the 
evaluate function within the JsonPathEvaluator class, which throws the 
exception if the variable can not be referenced. I have a local fix at 
https://github.com/apache/nifi/compare/master...nstack:fix/jsonpath that 
returns StringQueryResult("") if the subject is empty instead of throwing the 
exception and this appears to work however I wonder that as UpdateAttribute 
works if the problem is instead in QueryRecord eagerly evaluating the queries.

Any help would be appreciated.

Cheers,
Mandeep

[1] 2018-11-13 14:46:24,899 ERROR [Timer-Driven Process Thread-1] 
o.a.nifi.processors.standard.QueryRecord 
QueryRecord[id=0d5684e2-0167-1000-74c1-eb29a1401981] Failed to properly 
initialize Processor. If still scheduled to run, NiFi will attempt to 
initialize and run the Processor again after the 'Administrative Yield 
Duration' has elapsed. Failure is due to 
java.lang.reflect.InvocationTargetException: 
java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException: null
at sun.reflect.GeneratedMethodAccessor916.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
at 
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
at 
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75)
at 
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52)
at 
org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$4(StandardProcessorNode.java:1499)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: 
org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException:
 Subject is empty
at 
org.apache.nifi.attribute.expression.language.evaluation.functions.JsonPathEvaluator.evaluate(JsonPathEvaluator.java:66)
at org.apache.nifi.attribute.expression.language.Query.evaluate(Query.java:315)
at 
org.apache.nifi.attribute.expression.language.Query.evaluateExpression(Query.java:203)
at 
org.apache.nifi.attribute.expression.language.CompiledExpression.evaluate(CompiledExpression.java:58)
at 
org.apache.nifi.attribute.expression.language.StandardPreparedQuery.evaluateExpressions(StandardPreparedQuery.java:51)
at 
org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
at 
org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:148)
at 
org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:113)
at 
org.apache.nifi.processors.standard.QueryRecord.setupQueues(QueryRecord.java:443)
... 14 common frames omitted

--

Mandeep Gill

nstack.com / +44 7961822575

Re: ConsumeKafkaRecord won't pull new events from Kafka

[Mark Payne]

Mike,

Is there new data coming into the Kafka topic? By default, when the Processor 
is started, it uses
an auto commit offset of 'latest'. So that means that if you started the 
Processor with this setting,
the commit offset is saved pointing to the end of the topic. So if no more data 
is coming into the topic,
you'll not see anything out of the processor, while the ConsumeKafka processor 
was started with the
offset at 'earliest'?

Thanks
-Mark

On Nov 13, 2018, at 12:54 PM, Mike Thomsen 
mailto:mikerthom...@gmail.com>> wrote:

That would appear to be the case. So here's what I was doing:

1. Used this sort of code to serialize the Avro:

private byte[] serialize(Object obj, Class clz) throws Exception {
SpecificDatumWriter writer = new SpecificDatumWriter<>(clz);
ByteArrayOutputStream out = new ByteArrayOutputStream();
BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(out, null);
//writer.write(obj, encoder);
//encoder.flush();
out.close();

return out.toByteArray();
}

So that got me a byte array with just the binary representation.

2. Produced that to the Kafka topic.

3. Had an AvroReader variously configured to use 
schema.name hard set to an entry on the AvroSchemaRegistry 
or EmbeddedSchema.

Didn't see any flowfiles get emitted to parse.failure or success.

On Tue, Nov 13, 2018 at 12:50 PM Joe Witt 
mailto:joe.w...@gmail.com>> wrote:
Mike - so does this mean the parse.failure relationship wasn't working
though?  We should try to dig into this more if you're up for it or
sharing more details.  We dont want the behavior you ran into for
sure...
On Tue, Nov 13, 2018 at 12:49 PM Mike Thomsen 
mailto:mikerthom...@gmail.com>> wrote:
>
> So after a lot of diving into 1.9.X, it **appears** that there was some sort 
> of combination w/ the AvroReader + ConsumeKafkaRecord that was suppressing 
> errors from being reported. Haven't been able to fully figure out what was 
> doing on, but I know one of the contributing factors was that my producer 
> (spring boot service, not NiFi) was incorrectly building the binary form of 
> the Avro (didn't have embedded schema).
>
> Mike
>
> On Tue, Nov 13, 2018 at 12:33 PM Joe Witt 
> mailto:joe.w...@gmail.com>> wrote:
>>
>> Viking
>>
>> Just to clarify it isn't that it is trying to merge records to create
>> larger flow files but rather it is 'playing the kafka interaction as
>> it lies' so to speak.  When polling from Kafka we can get one or more
>> records.  We're just taking advantage of that without trading off
>> speed.
>>
>> Thanks
>> On Tue, Nov 13, 2018 at 12:24 PM Viking K 
>> mailto:cyber_v...@hotmail.com>> wrote:
>> >
>> > Mike, are we talking about complete avro messages or bare records (schema 
>> > is contained inside the avro file  or do they use a schema registry)
>> >
>> > From my own testing the ConsumeKafkaRecord tries to bundle in the incoming 
>> > messages to create larger flow files.
>> > Do you use any Kafka headers in the processor?
>> >
>> > Also what happens if you try to replicate the behavior of 
>> > ConsumeKafkaRecord like this. I don't know if you need the ConvertRecord 
>> > but it might be needed to pick out a schema name to use as Merge strategy.
>> > ConsumeKafka -> (ConvertRecord) -> Merge Content
>> >
>> > /Viking
>> > 
>> > From: Mike Thomsen mailto:mikerthom...@gmail.com>>
>> > Sent: Tuesday, November 13, 2018 3:02 PM
>> > To: users@nifi.apache.org
>> > Subject: Re: ConsumeKafkaRecord won't pull new events from Kafka
>> >
>> > Closest thing I see to something that implies something might be awry is 
>> > this in nifi-app.log:
>> >
>> > javax.management.InstanceAlreadyExistsException: 
>> > kafka.consumer:type=app-info,id=consumer-1
>> > at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
>> > at 
>> > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
>> > at 
>> > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
>> > at 
>> > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
>> > at 
>> > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
>> > at 
>> > com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
>> > at 
>> > org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:57)
>> > at 
>> > org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:640)
>> > at 
>> > org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:512)
>> > at 
>> > org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:494)
>> > at 
>> > 

Re: JsonPath expression language exception with QueryRecord processor

[Otto Fowler]

Can you create a JIRA with this information?


On November 13, 2018 at 10:16:36, Mandeep Gill (mand...@nstack.com) wrote:

Hi there,

I'm hitting an issue using the `jsonPath` expression language function to
extract a query to use with the QueryRecord processor. The processor works
fine if the expression language subject is contained within the process
group variable registry, but fails upon starting with an
AttributeExpressionLanguageException if the subject was expected to to
exist within a flowfile attribute [1]

I've attached a template generated on NiFi 1.8.0 demonstrating the problem
- it only appears to be an issue with dynamic outputs from the QueryRecord
processors, as the same expression language statement works fine when used
as part of UpdateAttribute processor with the subject in a flowfile
attribute as per the template. I've dug into the codebase and can trace the
error to the evaluate function within the JsonPathEvaluator class, which
throws the exception if the variable can not be referenced. I have a local
fix at https://github.com/apache/nifi/compare/master...nstack:fix/jsonpath that
returns StringQueryResult("") if the subject is empty instead of throwing
the exception and this appears to work however I wonder that as
UpdateAttribute works if the problem is instead in QueryRecord eagerly
evaluating the queries.

Any help would be appreciated.

Cheers,
Mandeep

[1] 2018-11-13 14:46:24,899 ERROR [Timer-Driven Process Thread-1]
o.a.nifi.processors.standard.QueryRecord
QueryRecord[id=0d5684e2-0167-1000-74c1-eb29a1401981] Failed to properly
initialize Processor. If still scheduled to run, NiFi will attempt to
initialize and run the Processor again after the 'Administrative Yield
Duration' has elapsed. Failure is due to
java.lang.reflect.InvocationTargetException:
java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException: null
at sun.reflect.GeneratedMethodAccessor916.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52)
at
org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$4(StandardProcessorNode.java:1499)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by:
org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException:
Subject is empty
at
org.apache.nifi.attribute.expression.language.evaluation.functions.JsonPathEvaluator.evaluate(JsonPathEvaluator.java:66)
at
org.apache.nifi.attribute.expression.language.Query.evaluate(Query.java:315)
at
org.apache.nifi.attribute.expression.language.Query.evaluateExpression(Query.java:203)
at
org.apache.nifi.attribute.expression.language.CompiledExpression.evaluate(CompiledExpression.java:58)
at
org.apache.nifi.attribute.expression.language.StandardPreparedQuery.evaluateExpressions(StandardPreparedQuery.java:51)
at
org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
at
org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:148)
at
org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:113)
at
org.apache.nifi.processors.standard.QueryRecord.setupQueues(QueryRecord.java:443)
... 14 common frames omitted

--

Mandeep Gill

nstack.com  / +44 7961822575

Re: ConsumeKafkaRecord won't pull new events from Kafka

[Mike Thomsen]

That would appear to be the case. So here's what I was doing:

1. Used this sort of code to serialize the Avro:

private byte[] serialize(Object obj, Class clz) throws Exception {
SpecificDatumWriter writer = new SpecificDatumWriter<>(clz);
ByteArrayOutputStream out = new ByteArrayOutputStream();
BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(out,
null);
//writer.write(obj, encoder);
//encoder.flush();
out.close();

return out.toByteArray();
}

So that got me a byte array with just the binary representation.

2. Produced that to the Kafka topic.

3. Had an AvroReader variously configured to use schema.name hard set to an
entry on the AvroSchemaRegistry or EmbeddedSchema.

Didn't see any flowfiles get emitted to parse.failure or success.

On Tue, Nov 13, 2018 at 12:50 PM Joe Witt  wrote:

> Mike - so does this mean the parse.failure relationship wasn't working
> though?  We should try to dig into this more if you're up for it or
> sharing more details.  We dont want the behavior you ran into for
> sure...
> On Tue, Nov 13, 2018 at 12:49 PM Mike Thomsen 
> wrote:
> >
> > So after a lot of diving into 1.9.X, it **appears** that there was some
> sort of combination w/ the AvroReader + ConsumeKafkaRecord that was
> suppressing errors from being reported. Haven't been able to fully figure
> out what was doing on, but I know one of the contributing factors was that
> my producer (spring boot service, not NiFi) was incorrectly building the
> binary form of the Avro (didn't have embedded schema).
> >
> > Mike
> >
> > On Tue, Nov 13, 2018 at 12:33 PM Joe Witt  wrote:
> >>
> >> Viking
> >>
> >> Just to clarify it isn't that it is trying to merge records to create
> >> larger flow files but rather it is 'playing the kafka interaction as
> >> it lies' so to speak.  When polling from Kafka we can get one or more
> >> records.  We're just taking advantage of that without trading off
> >> speed.
> >>
> >> Thanks
> >> On Tue, Nov 13, 2018 at 12:24 PM Viking K 
> wrote:
> >> >
> >> > Mike, are we talking about complete avro messages or bare records
> (schema is contained inside the avro file  or do they use a schema registry)
> >> >
> >> > From my own testing the ConsumeKafkaRecord tries to bundle in the
> incoming messages to create larger flow files.
> >> > Do you use any Kafka headers in the processor?
> >> >
> >> > Also what happens if you try to replicate the behavior of
> ConsumeKafkaRecord like this. I don't know if you need the ConvertRecord
> but it might be needed to pick out a schema name to use as Merge strategy.
> >> > ConsumeKafka -> (ConvertRecord) -> Merge Content
> >> >
> >> > /Viking
> >> > 
> >> > From: Mike Thomsen 
> >> > Sent: Tuesday, November 13, 2018 3:02 PM
> >> > To: users@nifi.apache.org
> >> > Subject: Re: ConsumeKafkaRecord won't pull new events from Kafka
> >> >
> >> > Closest thing I see to something that implies something might be awry
> is this in nifi-app.log:
> >> >
> >> > javax.management.InstanceAlreadyExistsException:
> kafka.consumer:type=app-info,id=consumer-1
> >> > at
> com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> >> > at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> >> > at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> >> > at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> >> > at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> >> > at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> >> > at
> org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:57)
> >> > at
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:640)
> >> > at
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:512)
> >> > at
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:494)
> >> > at
> org.apache.nifi.processors.kafka.pubsub.ConsumerPool.createKafkaConsumer(ConsumerPool.java:143)
> >> > at
> org.apache.nifi.processors.kafka.pubsub.ConsumerPool.obtainConsumer(ConsumerPool.java:107)
> >> > at
> org.apache.nifi.processors.kafka.pubsub.ConsumeKafka.onTrigger(ConsumeKafka.java:359)
> >> > at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> >> > at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> >> > at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> >> > at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> >> > at
> 

Re: ConsumeKafkaRecord won't pull new events from Kafka

[Viking K]

Joe,

Yea I was thinking in the lines of;
Maybe its a broken Avro record or something similar inside the Kafka Topic 
thats causing ConsumeKafkaRecord to lock up.
Since the ConsumeKafkaRecord uses the newer record readers while ConsumeKafka 
don't.
If that is the case it should materialize itself in the ConvertRecord 
processor, and maybe it should be possible to grab out the error from there 
instead.

/Viking

From: Joe Witt 
Sent: Tuesday, November 13, 2018 5:32 PM
To: users@nifi.apache.org
Subject: Re: ConsumeKafkaRecord won't pull new events from Kafka

Viking

Just to clarify it isn't that it is trying to merge records to create
larger flow files but rather it is 'playing the kafka interaction as
it lies' so to speak.  When polling from Kafka we can get one or more
records.  We're just taking advantage of that without trading off
speed.

Thanks
On Tue, Nov 13, 2018 at 12:24 PM Viking K  wrote:
>
> Mike, are we talking about complete avro messages or bare records (schema is 
> contained inside the avro file  or do they use a schema registry)
>
> From my own testing the ConsumeKafkaRecord tries to bundle in the incoming 
> messages to create larger flow files.
> Do you use any Kafka headers in the processor?
>
> Also what happens if you try to replicate the behavior of ConsumeKafkaRecord 
> like this. I don't know if you need the ConvertRecord but it might be needed 
> to pick out a schema name to use as Merge strategy.
> ConsumeKafka -> (ConvertRecord) -> Merge Content
>
> /Viking
> 
> From: Mike Thomsen 
> Sent: Tuesday, November 13, 2018 3:02 PM
> To: users@nifi.apache.org
> Subject: Re: ConsumeKafkaRecord won't pull new events from Kafka
>
> Closest thing I see to something that implies something might be awry is this 
> in nifi-app.log:
>
> javax.management.InstanceAlreadyExistsException: 
> kafka.consumer:type=app-info,id=consumer-1
> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> at 
> org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:57)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:640)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:512)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:494)
> at 
> org.apache.nifi.processors.kafka.pubsub.ConsumerPool.createKafkaConsumer(ConsumerPool.java:143)
> at 
> org.apache.nifi.processors.kafka.pubsub.ConsumerPool.obtainConsumer(ConsumerPool.java:107)
> at 
> org.apache.nifi.processors.kafka.pubsub.ConsumeKafka.onTrigger(ConsumeKafka.java:359)
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at 
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> On Tue, Nov 13, 2018 at 10:00 AM Pierre Villard  
> wrote:
>
> Hey Mike,
>
> Anything in the logs?
>
> Pierre
>
> Le mar. 13 nov. 2018 à 15:56, Mike Thomsen  a écrit :
>
> I have an odd situation where I have ConsumeKafkaRecord and ConsumeKafka 
> pulling from the same topic under different consumer groups, but only the 
> latter will pull new events. I ran into a situation where the reader didn't 
> like the Avro data being pulled from the queue and so I created new topics 
> and configured both processors to use the new ones. However, only the 
> non-record version of the processor will read.
>
> Anyone got suggestions on how to debug this? I'm reasonably familiar with 

Re: ConsumeKafkaRecord won't pull new events from Kafka

[Joe Witt]

Mike - so does this mean the parse.failure relationship wasn't working
though?  We should try to dig into this more if you're up for it or
sharing more details.  We dont want the behavior you ran into for
sure...
On Tue, Nov 13, 2018 at 12:49 PM Mike Thomsen  wrote:
>
> So after a lot of diving into 1.9.X, it **appears** that there was some sort 
> of combination w/ the AvroReader + ConsumeKafkaRecord that was suppressing 
> errors from being reported. Haven't been able to fully figure out what was 
> doing on, but I know one of the contributing factors was that my producer 
> (spring boot service, not NiFi) was incorrectly building the binary form of 
> the Avro (didn't have embedded schema).
>
> Mike
>
> On Tue, Nov 13, 2018 at 12:33 PM Joe Witt  wrote:
>>
>> Viking
>>
>> Just to clarify it isn't that it is trying to merge records to create
>> larger flow files but rather it is 'playing the kafka interaction as
>> it lies' so to speak.  When polling from Kafka we can get one or more
>> records.  We're just taking advantage of that without trading off
>> speed.
>>
>> Thanks
>> On Tue, Nov 13, 2018 at 12:24 PM Viking K  wrote:
>> >
>> > Mike, are we talking about complete avro messages or bare records (schema 
>> > is contained inside the avro file  or do they use a schema registry)
>> >
>> > From my own testing the ConsumeKafkaRecord tries to bundle in the incoming 
>> > messages to create larger flow files.
>> > Do you use any Kafka headers in the processor?
>> >
>> > Also what happens if you try to replicate the behavior of 
>> > ConsumeKafkaRecord like this. I don't know if you need the ConvertRecord 
>> > but it might be needed to pick out a schema name to use as Merge strategy.
>> > ConsumeKafka -> (ConvertRecord) -> Merge Content
>> >
>> > /Viking
>> > 
>> > From: Mike Thomsen 
>> > Sent: Tuesday, November 13, 2018 3:02 PM
>> > To: users@nifi.apache.org
>> > Subject: Re: ConsumeKafkaRecord won't pull new events from Kafka
>> >
>> > Closest thing I see to something that implies something might be awry is 
>> > this in nifi-app.log:
>> >
>> > javax.management.InstanceAlreadyExistsException: 
>> > kafka.consumer:type=app-info,id=consumer-1
>> > at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
>> > at 
>> > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
>> > at 
>> > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
>> > at 
>> > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
>> > at 
>> > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
>> > at 
>> > com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
>> > at 
>> > org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:57)
>> > at 
>> > org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:640)
>> > at 
>> > org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:512)
>> > at 
>> > org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:494)
>> > at 
>> > org.apache.nifi.processors.kafka.pubsub.ConsumerPool.createKafkaConsumer(ConsumerPool.java:143)
>> > at 
>> > org.apache.nifi.processors.kafka.pubsub.ConsumerPool.obtainConsumer(ConsumerPool.java:107)
>> > at 
>> > org.apache.nifi.processors.kafka.pubsub.ConsumeKafka.onTrigger(ConsumeKafka.java:359)
>> > at 
>> > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> > at 
>> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
>> > at 
>> > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
>> > at 
>> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>> > at 
>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> > at 
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> > at 
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> > at 
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> > at 
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> > at java.lang.Thread.run(Thread.java:748)
>> >
>> >
>> >
>> > On Tue, Nov 13, 2018 at 10:00 AM Pierre Villard 
>> >  wrote:
>> >
>> > Hey Mike,
>> >
>> > Anything in the logs?
>> >
>> > Pierre
>> >
>> > Le mar. 13 nov. 2018 à 15:56, Mike Thomsen  a 
>> > écrit :
>> >
>> > I have an odd situation where I have 

Re: ConsumeKafkaRecord won't pull new events from Kafka

[Mike Thomsen]

So after a lot of diving into 1.9.X, it **appears** that there was some
sort of combination w/ the AvroReader + ConsumeKafkaRecord that was
suppressing errors from being reported. Haven't been able to fully figure
out what was doing on, but I know one of the contributing factors was that
my producer (spring boot service, not NiFi) was incorrectly building the
binary form of the Avro (didn't have embedded schema).

Mike

On Tue, Nov 13, 2018 at 12:33 PM Joe Witt  wrote:

> Viking
>
> Just to clarify it isn't that it is trying to merge records to create
> larger flow files but rather it is 'playing the kafka interaction as
> it lies' so to speak.  When polling from Kafka we can get one or more
> records.  We're just taking advantage of that without trading off
> speed.
>
> Thanks
> On Tue, Nov 13, 2018 at 12:24 PM Viking K  wrote:
> >
> > Mike, are we talking about complete avro messages or bare records
> (schema is contained inside the avro file  or do they use a schema registry)
> >
> > From my own testing the ConsumeKafkaRecord tries to bundle in the
> incoming messages to create larger flow files.
> > Do you use any Kafka headers in the processor?
> >
> > Also what happens if you try to replicate the behavior of
> ConsumeKafkaRecord like this. I don't know if you need the ConvertRecord
> but it might be needed to pick out a schema name to use as Merge strategy.
> > ConsumeKafka -> (ConvertRecord) -> Merge Content
> >
> > /Viking
> > 
> > From: Mike Thomsen 
> > Sent: Tuesday, November 13, 2018 3:02 PM
> > To: users@nifi.apache.org
> > Subject: Re: ConsumeKafkaRecord won't pull new events from Kafka
> >
> > Closest thing I see to something that implies something might be awry is
> this in nifi-app.log:
> >
> > javax.management.InstanceAlreadyExistsException:
> kafka.consumer:type=app-info,id=consumer-1
> > at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> > at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> > at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> > at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> > at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> > at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> > at
> org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:57)
> > at
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:640)
> > at
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:512)
> > at
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:494)
> > at
> org.apache.nifi.processors.kafka.pubsub.ConsumerPool.createKafkaConsumer(ConsumerPool.java:143)
> > at
> org.apache.nifi.processors.kafka.pubsub.ConsumerPool.obtainConsumer(ConsumerPool.java:107)
> > at
> org.apache.nifi.processors.kafka.pubsub.ConsumeKafka.onTrigger(ConsumeKafka.java:359)
> > at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> > at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> > at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> > at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> >
> >
> >
> > On Tue, Nov 13, 2018 at 10:00 AM Pierre Villard <
> pierre.villard...@gmail.com> wrote:
> >
> > Hey Mike,
> >
> > Anything in the logs?
> >
> > Pierre
> >
> > Le mar. 13 nov. 2018 à 15:56, Mike Thomsen  a
> écrit :
> >
> > I have an odd situation where I have ConsumeKafkaRecord and ConsumeKafka
> pulling from the same topic under different consumer groups, but only the
> latter will pull new events. I ran into a situation where the reader didn't
> like the Avro data being pulled from the queue and so I created new topics
> and configured both processors to use the new ones. However, only the
> non-record version of the processor will read.
> >
> > Anyone got suggestions on how to debug this? I'm 

Re: Can a nar bundle host a servlet outside of the NiFi web framework?

[Nathan Gough]

Hi Vijay,

You can take a look at the code for ListenHTTP to get an idea of how you could 
write a processor to do what you want with your own servlet. ListenHTTP uses 
embedded Jetty:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java

There's a few articles around that describe how to write your own processor:
https://medium.com/hashmapinc/creating-custom-processors-and-controllers-in-apache-nifi-e14148740ea
https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html

Cheers,
Nathan


On 11/13/18, 10:27 AM, "Vijay Chhipa"  wrote:

Hi Bryan

So we implemented it based on your suggestion, and it works, however, for 
concerns unrelated to NiFi  from the team, I  will need to go back to the 
original idea which is standup  a servlet from within my .nar file.

Is it possible and if so can you point me to an example?

Thank you, 

Vijay



> On Nov 6, 2018, at 4:35 PM, Vijay Chhipa  wrote:
> 
> Bryan
> 
> Thats perfect, I was planning to launch my own Jetty instance from within 
the nar, but ListenHttp or HandleHttpRequest is even better 
> 
> Thank you, 
> 
> 
>> On Nov 6, 2018, at 4:24 PM, Bryan Bende  wrote:
>> 
>> Not sure if this is what you are asking for, but ListenHttp and
>> HandleHttpRequest both do this.
>> 
>> They start an embedded Jetty server on a different port, unrelated to
>> NiFi's web server, and they are not part of NiFi's security model,
>> although they can be secured with 2-way TLS, or basic auth (I tthink).
>> 
>> The incoming request gets turned into a flow file and be processed by
>> the rest of the flow, and a response can be sent with
>> HandleHttpResponse.
>> On Tue, Nov 6, 2018 at 4:52 PM Vijay Chhipa  wrote:
>>> 
>>> Hello
>>> 
>>> I would like to create a nar bundle that can serve up some content, I 
would like this to be outside of the secure NiFI web context. Essentially it 
has nothing to do with the "https://:/nifi"  and the 
nifi.properties
>>> 
>>> This would be a unsecured endpoint  accessible from the host thats 
running NiFi.
>>> http:/localhost:8089/blabla
>>> 
>>> Essentially, my nar  would  be running a Jetty server other than the 
NiFi Jetty server
>>> 
>>> Any technical issues with this approach ( obviously there is a security 
concerns but apart from that)
>>> 
>>> Thanks
>>> 
>>> Vijay
>>> 
>>> 
>>> 
>>> 
>>> 
> 

Re: ConsumeKafkaRecord won't pull new events from Kafka

[Joe Witt]

Viking

Just to clarify it isn't that it is trying to merge records to create
larger flow files but rather it is 'playing the kafka interaction as
it lies' so to speak.  When polling from Kafka we can get one or more
records.  We're just taking advantage of that without trading off
speed.

Thanks
On Tue, Nov 13, 2018 at 12:24 PM Viking K  wrote:
>
> Mike, are we talking about complete avro messages or bare records (schema is 
> contained inside the avro file  or do they use a schema registry)
>
> From my own testing the ConsumeKafkaRecord tries to bundle in the incoming 
> messages to create larger flow files.
> Do you use any Kafka headers in the processor?
>
> Also what happens if you try to replicate the behavior of ConsumeKafkaRecord 
> like this. I don't know if you need the ConvertRecord but it might be needed 
> to pick out a schema name to use as Merge strategy.
> ConsumeKafka -> (ConvertRecord) -> Merge Content
>
> /Viking
> 
> From: Mike Thomsen 
> Sent: Tuesday, November 13, 2018 3:02 PM
> To: users@nifi.apache.org
> Subject: Re: ConsumeKafkaRecord won't pull new events from Kafka
>
> Closest thing I see to something that implies something might be awry is this 
> in nifi-app.log:
>
> javax.management.InstanceAlreadyExistsException: 
> kafka.consumer:type=app-info,id=consumer-1
> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> at 
> org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:57)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:640)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:512)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:494)
> at 
> org.apache.nifi.processors.kafka.pubsub.ConsumerPool.createKafkaConsumer(ConsumerPool.java:143)
> at 
> org.apache.nifi.processors.kafka.pubsub.ConsumerPool.obtainConsumer(ConsumerPool.java:107)
> at 
> org.apache.nifi.processors.kafka.pubsub.ConsumeKafka.onTrigger(ConsumeKafka.java:359)
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at 
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> On Tue, Nov 13, 2018 at 10:00 AM Pierre Villard  
> wrote:
>
> Hey Mike,
>
> Anything in the logs?
>
> Pierre
>
> Le mar. 13 nov. 2018 à 15:56, Mike Thomsen  a écrit :
>
> I have an odd situation where I have ConsumeKafkaRecord and ConsumeKafka 
> pulling from the same topic under different consumer groups, but only the 
> latter will pull new events. I ran into a situation where the reader didn't 
> like the Avro data being pulled from the queue and so I created new topics 
> and configured both processors to use the new ones. However, only the 
> non-record version of the processor will read.
>
> Anyone got suggestions on how to debug this? I'm reasonably familiar with 
> Kafka, but can't figure out why ConsumeKafka and the console consumer can 
> read the topic, but ConsumeKafkaRecord is acting like there's nothing there 
> at all.
>
> Thanks,
>
> Mike

Re: ConsumeKafkaRecord won't pull new events from Kafka

[Viking K]

Mike, are we talking about complete avro messages or bare records (schema is 
contained inside the avro file  or do they use a schema registry)

>From my own testing the ConsumeKafkaRecord tries to bundle in the incoming 
>messages to create larger flow files.
Do you use any Kafka headers in the processor?

Also what happens if you try to replicate the behavior of ConsumeKafkaRecord 
like this. I don't know if you need the ConvertRecord but it might be needed to 
pick out a schema name to use as Merge strategy.
ConsumeKafka -> (ConvertRecord) -> Merge Content

/Viking

From: Mike Thomsen 
Sent: Tuesday, November 13, 2018 3:02 PM
To: users@nifi.apache.org
Subject: Re: ConsumeKafkaRecord won't pull new events from Kafka

Closest thing I see to something that implies something might be awry is this 
in nifi-app.log:

javax.management.InstanceAlreadyExistsException: 
kafka.consumer:type=app-info,id=consumer-1
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at 
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at 
org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:57)
at 
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:640)
at 
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:512)
at 
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:494)
at 
org.apache.nifi.processors.kafka.pubsub.ConsumerPool.createKafkaConsumer(ConsumerPool.java:143)
at 
org.apache.nifi.processors.kafka.pubsub.ConsumerPool.obtainConsumer(ConsumerPool.java:107)
at 
org.apache.nifi.processors.kafka.pubsub.ConsumeKafka.onTrigger(ConsumeKafka.java:359)
at 
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
at 
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)



On Tue, Nov 13, 2018 at 10:00 AM Pierre Villard 
mailto:pierre.villard...@gmail.com>> wrote:
Hey Mike,

Anything in the logs?

Pierre

Le mar. 13 nov. 2018 à 15:56, Mike Thomsen 
mailto:mikerthom...@gmail.com>> a écrit :
I have an odd situation where I have ConsumeKafkaRecord and ConsumeKafka 
pulling from the same topic under different consumer groups, but only the 
latter will pull new events. I ran into a situation where the reader didn't 
like the Avro data being pulled from the queue and so I created new topics and 
configured both processors to use the new ones. However, only the non-record 
version of the processor will read.

Anyone got suggestions on how to debug this? I'm reasonably familiar with 
Kafka, but can't figure out why ConsumeKafka and the console consumer can read 
the topic, but ConsumeKafkaRecord is acting like there's nothing there at all.

Thanks,

Mike

RE: Easiest way to list all NiFi processors in a release.

[Vitaly Krivoy]

Awesome. Thanks!

From: Pierre Villard 
Sent: Monday, November 12, 2018 1:22 PM
To: users@nifi.apache.org
Subject: Re: Easiest way to list all NiFi processors in a release.

Hi Vitaly,

Yes there is one [1]: GET on /flow/processor-types

Easiest way to find out is to use the developer tool of your browser. Anything 
viewable/doable in the NiFi UI is a doable through REST API endpoints.

[1] https://nifi.apache.org/docs/nifi-docs/rest-api/

Thanks,
Pierre

Le lun. 12 nov. 2018 à 18:36, Vitaly Krivoy 
mailto:vitaly_kri...@jhancock.com>> a écrit :
Is there a REST API to list all NiFi processors in a current NiFi release? 
Thanks.

STATEMENT OF CONFIDENTIALITY The information contained in this email message 
and any attachments may be confidential and legally privileged and is intended 
for the use of the addressee(s) only. If you are not an intended recipient, 
please: (1) notify me immediately by replying to this message; (2) do not use, 
disseminate, distribute or reproduce any part of the message or any attachment; 
and (3) destroy all copies of this message and any attachments.

STATEMENT OF CONFIDENTIALITY The information contained in this email message 
and any attachments may be confidential and legally privileged and is intended 
for the use of the addressee(s) only. If you are not an intended recipient, 
please: (1) notify me immediately by replying to this message; (2) do not use, 
disseminate, distribute or reproduce any part of the message or any attachment; 
and (3) destroy all copies of this message and any attachments.

Re: LogMessage Processor Fails to Transfer FlowFile when Log Level is not Enabled

[Dann]

Sure is the one.  Thanks for the quick reply!

On Tue, Nov 13, 2018 at 8:41 AM Matt Burgess  wrote:

> Dann,
>
> Looks like you're running into NIFI-5652 [1], I'll try to get a fix up
> soon.
>
> Regards,
> Matt
>
> [1] https://issues.apache.org/jira/browse/NIFI-5652
>
> On Tue, Nov 13, 2018 at 10:06 AM Dann  wrote:
> >
> > It looks like to me that the LogMessage processor doesn't work when the
> log level isn't enabled.  It fails to transfer the flow file and then the
> processor gets administratively yielded.
> >
> > This is what looks like is happening to me.
> > * At the beginning of onTrigger, session.get() is called.
> > * Then if the log level isn't enabled, the transferChunk method is
> called with the session
> > * The transfer chunk method gets more FlowFiles from the session and
> transfers those to REL_SUCCESS
> > * The original flow file from the beginning of on Trigger is never
> transferred and the causes the problem.
> >
> > A second pair of eyes might verify where I am thinking wrong or that
> there is an issue.
> >
> > Thanks,
> >
> > Dann Ormond
>

Re: LogMessage Processor Fails to Transfer FlowFile when Log Level is not Enabled

[Matt Burgess]

Dann,

Looks like you're running into NIFI-5652 [1], I'll try to get a fix up soon.

Regards,
Matt

[1] https://issues.apache.org/jira/browse/NIFI-5652

On Tue, Nov 13, 2018 at 10:06 AM Dann  wrote:
>
> It looks like to me that the LogMessage processor doesn't work when the log 
> level isn't enabled.  It fails to transfer the flow file and then the 
> processor gets administratively yielded.
>
> This is what looks like is happening to me.
> * At the beginning of onTrigger, session.get() is called.
> * Then if the log level isn't enabled, the transferChunk method is called 
> with the session
> * The transfer chunk method gets more FlowFiles from the session and 
> transfers those to REL_SUCCESS
> * The original flow file from the beginning of on Trigger is never 
> transferred and the causes the problem.
>
> A second pair of eyes might verify where I am thinking wrong or that there is 
> an issue.
>
> Thanks,
>
> Dann Ormond

Re: Can a nar bundle host a servlet outside of the NiFi web framework?

[Vijay Chhipa]

Hi Bryan

So we implemented it based on your suggestion, and it works, however, for 
concerns unrelated to NiFi  from the team, I  will need to go back to the 
original idea which is standup  a servlet from within my .nar file.

Is it possible and if so can you point me to an example?

Thank you, 

Vijay



> On Nov 6, 2018, at 4:35 PM, Vijay Chhipa  wrote:
> 
> Bryan
> 
> Thats perfect, I was planning to launch my own Jetty instance from within the 
> nar, but ListenHttp or HandleHttpRequest is even better 
> 
> Thank you, 
> 
> 
>> On Nov 6, 2018, at 4:24 PM, Bryan Bende  wrote:
>> 
>> Not sure if this is what you are asking for, but ListenHttp and
>> HandleHttpRequest both do this.
>> 
>> They start an embedded Jetty server on a different port, unrelated to
>> NiFi's web server, and they are not part of NiFi's security model,
>> although they can be secured with 2-way TLS, or basic auth (I tthink).
>> 
>> The incoming request gets turned into a flow file and be processed by
>> the rest of the flow, and a response can be sent with
>> HandleHttpResponse.
>> On Tue, Nov 6, 2018 at 4:52 PM Vijay Chhipa  wrote:
>>> 
>>> Hello
>>> 
>>> I would like to create a nar bundle that can serve up some content, I would 
>>> like this to be outside of the secure NiFI web context. Essentially it has 
>>> nothing to do with the "https://:/nifi"  and the 
>>> nifi.properties
>>> 
>>> This would be a unsecured endpoint  accessible from the host thats running 
>>> NiFi.
>>> http:/localhost:8089/blabla
>>> 
>>> Essentially, my nar  would  be running a Jetty server other than the NiFi 
>>> Jetty server
>>> 
>>> Any technical issues with this approach ( obviously there is a security 
>>> concerns but apart from that)
>>> 
>>> Thanks
>>> 
>>> Vijay
>>> 
>>> 
>>> 
>>> 
>>> 
> 



smime.p7s
Description: S/MIME cryptographic signature

JsonPath expression language exception with QueryRecord processor

[Mandeep Gill]

Hi there,

I'm hitting an issue using the `jsonPath` expression language function to
extract a query to use with the QueryRecord processor. The processor works
fine if the expression language subject is contained within the process
group variable registry, but fails upon starting with an
AttributeExpressionLanguageException if the subject was expected to to
exist within a flowfile attribute [1]

I've attached a template generated on NiFi 1.8.0 demonstrating the problem
- it only appears to be an issue with dynamic outputs from the QueryRecord
processors, as the same expression language statement works fine when used
as part of UpdateAttribute processor with the subject in a flowfile
attribute as per the template. I've dug into the codebase and can trace the
error to the evaluate function within the JsonPathEvaluator class, which
throws the exception if the variable can not be referenced. I have a local
fix at https://github.com/apache/nifi/compare/master...nstack:fix/jsonpath that
returns StringQueryResult("") if the subject is empty instead of throwing
the exception and this appears to work however I wonder that as
UpdateAttribute works if the problem is instead in QueryRecord eagerly
evaluating the queries.

Any help would be appreciated.

Cheers,
Mandeep

[1] 2018-11-13 14:46:24,899 ERROR [Timer-Driven Process Thread-1]
o.a.nifi.processors.standard.QueryRecord
QueryRecord[id=0d5684e2-0167-1000-74c1-eb29a1401981] Failed to properly
initialize Processor. If still scheduled to run, NiFi will attempt to
initialize and run the Processor again after the 'Administrative Yield
Duration' has elapsed. Failure is due to
java.lang.reflect.InvocationTargetException:
java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException: null
at sun.reflect.GeneratedMethodAccessor916.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52)
at
org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$4(StandardProcessorNode.java:1499)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by:
org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException:
Subject is empty
at
org.apache.nifi.attribute.expression.language.evaluation.functions.JsonPathEvaluator.evaluate(JsonPathEvaluator.java:66)
at
org.apache.nifi.attribute.expression.language.Query.evaluate(Query.java:315)
at
org.apache.nifi.attribute.expression.language.Query.evaluateExpression(Query.java:203)
at
org.apache.nifi.attribute.expression.language.CompiledExpression.evaluate(CompiledExpression.java:58)
at
org.apache.nifi.attribute.expression.language.StandardPreparedQuery.evaluateExpressions(StandardPreparedQuery.java:51)
at
org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
at
org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:148)
at
org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:113)
at
org.apache.nifi.processors.standard.QueryRecord.setupQueues(QueryRecord.java:443)
... 14 common frames omitted

-- 

Mandeep Gill

nstack.com  / +44 7961822575



0d515341-0167-1000-9a39-f4b2d7745941
QueryRecord JsonPath Issue


b290e741-f388-33f0--
33811b00-fd04-3d0e--
1 GB
1

33811b00-fd04-3d0e--
c2132d5e-86cc-3d8a--
PROCESSOR

0 sec
1

success

33811b00-fd04-3d0e--
40b2a725-a0dc-3517--
PROCESSOR

0


c499d71c-2cb4-3b01--
33811b00-fd04-3d0e--
1 GB
1

LogMessage Processor Fails to Transfer FlowFile when Log Level is not Enabled

[Dann]

It looks like to me that the LogMessage processor doesn't work when the log
level isn't enabled.  It fails to transfer the flow file and then the
processor gets administratively yielded.

This is what looks like is happening to me.
* At the beginning of onTrigger, session.get() is called.
* Then if the log level isn't enabled, the transferChunk method is called
with the session
* The transfer chunk method gets more FlowFiles from the session and
transfers those to REL_SUCCESS
* The original flow file from the beginning of on Trigger is never
transferred and the causes the problem.

A second pair of eyes might verify where I am thinking wrong or that there
is an issue.

Thanks,

Dann Ormond

Re: ConsumeKafkaRecord won't pull new events from Kafka

[Mike Thomsen]

Closest thing I see to something that implies something might be awry is
this in nifi-app.log:

javax.management.InstanceAlreadyExistsException:
kafka.consumer:type=app-info,id=consumer-1
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at
org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:57)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:640)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:512)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:494)
at
org.apache.nifi.processors.kafka.pubsub.ConsumerPool.createKafkaConsumer(ConsumerPool.java:143)
at
org.apache.nifi.processors.kafka.pubsub.ConsumerPool.obtainConsumer(ConsumerPool.java:107)
at
org.apache.nifi.processors.kafka.pubsub.ConsumeKafka.onTrigger(ConsumeKafka.java:359)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)



On Tue, Nov 13, 2018 at 10:00 AM Pierre Villard 
wrote:

> Hey Mike,
>
> Anything in the logs?
>
> Pierre
>
> Le mar. 13 nov. 2018 à 15:56, Mike Thomsen  a
> écrit :
>
>> I have an odd situation where I have ConsumeKafkaRecord and ConsumeKafka
>> pulling from the same topic under different consumer groups, but only the
>> latter will pull new events. I ran into a situation where the reader didn't
>> like the Avro data being pulled from the queue and so I created new topics
>> and configured both processors to use the new ones. However, only the
>> non-record version of the processor will read.
>>
>> Anyone got suggestions on how to debug this? I'm reasonably familiar with
>> Kafka, but can't figure out why ConsumeKafka and the console consumer can
>> read the topic, but ConsumeKafkaRecord is acting like there's nothing there
>> at all.
>>
>> Thanks,
>>
>> Mike
>>
>

Re: ConsumeKafkaRecord won't pull new events from Kafka

[Joe Witt]

Mike,

In previous versions that processor didn't have a parse.failure
relationship I believe.  In that case data would not be pulled when
unable to parse it.  In newer versions of the processor it has it.  So
check there first.

Thanks
On Tue, Nov 13, 2018 at 9:56 AM Mike Thomsen  wrote:
>
> I have an odd situation where I have ConsumeKafkaRecord and ConsumeKafka 
> pulling from the same topic under different consumer groups, but only the 
> latter will pull new events. I ran into a situation where the reader didn't 
> like the Avro data being pulled from the queue and so I created new topics 
> and configured both processors to use the new ones. However, only the 
> non-record version of the processor will read.
>
> Anyone got suggestions on how to debug this? I'm reasonably familiar with 
> Kafka, but can't figure out why ConsumeKafka and the console consumer can 
> read the topic, but ConsumeKafkaRecord is acting like there's nothing there 
> at all.
>
> Thanks,
>
> Mike

Re: ConsumeKafkaRecord won't pull new events from Kafka

[Pierre Villard]

Hey Mike,

Anything in the logs?

Pierre

Le mar. 13 nov. 2018 à 15:56, Mike Thomsen  a
écrit :

> I have an odd situation where I have ConsumeKafkaRecord and ConsumeKafka
> pulling from the same topic under different consumer groups, but only the
> latter will pull new events. I ran into a situation where the reader didn't
> like the Avro data being pulled from the queue and so I created new topics
> and configured both processors to use the new ones. However, only the
> non-record version of the processor will read.
>
> Anyone got suggestions on how to debug this? I'm reasonably familiar with
> Kafka, but can't figure out why ConsumeKafka and the console consumer can
> read the topic, but ConsumeKafkaRecord is acting like there's nothing there
> at all.
>
> Thanks,
>
> Mike
>

ConsumeKafkaRecord won't pull new events from Kafka

[Mike Thomsen]

I have an odd situation where I have ConsumeKafkaRecord and ConsumeKafka
pulling from the same topic under different consumer groups, but only the
latter will pull new events. I ran into a situation where the reader didn't
like the Avro data being pulled from the queue and so I created new topics
and configured both processors to use the new ones. However, only the
non-record version of the processor will read.

Anyone got suggestions on how to debug this? I'm reasonably familiar with
Kafka, but can't figure out why ConsumeKafka and the console consumer can
read the topic, but ConsumeKafkaRecord is acting like there's nothing there
at all.

Thanks,

Mike

Re: EnforceOrder processor

[Boris Tyukin]

good deal, I am going to play with it next week and see if I can
contribute. The processor is really neat and I can see Koji put a lot of
thoughts and time designing it.

Boris

On Tue, Nov 13, 2018 at 3:31 AM Pierre Villard 
wrote:

> Hey Boris,
>
> It's certainly a processor that would benefit an "additional details"
> documentation page. If you're willing to file a JIRA and submit a PR
> explaining the processor with your words, that would certainly be much
> appreciated and I'm sure Koji will be happy to review it.
>
> Thanks,
> Pierre
>
> Le lun. 12 nov. 2018 à 21:13, Boris Tyukin  a
> écrit :
>
>> I was really confused how EnforceOrder processor works and NiFi
>> documentation made it even more confusing. After some time looking for an
>> explanation, I found this gist
>>  which
>> I think was created by a developer who created this processor. It does much
>> better job explaining the processor and various options and relationships
>> than NiFi doc so wanted to post it here for someone who is looking for the
>> same information.
>>
>> https://gist.github.com/ijokarumawak/88fc30a2300845b3c27a79113fc72d41
>>
>>
>>

Re: Easiest way to list all NiFi processors in a release.

[Daniel Chaffelson]

There is also a shortcut function NiPyApi for this, as it is handy when
procedurally deploying flows:
https://nipyapi.readthedocs.io/en/latest/_modules/nipyapi/canvas.html#list_all_processor_types


On Mon, Nov 12, 2018 at 6:23 PM Pierre Villard 
wrote:

> Hi Vitaly,
>
> Yes there is one [1]: GET on /flow/processor-types
>
> Easiest way to find out is to use the developer tool of your browser.
> Anything viewable/doable in the NiFi UI is a doable through REST API
> endpoints.
>
> [1] https://nifi.apache.org/docs/nifi-docs/rest-api/
>
> Thanks,
> Pierre
>
> Le lun. 12 nov. 2018 à 18:36, Vitaly Krivoy 
> a écrit :
>
>> Is there a REST API to list all NiFi processors in a current NiFi
>> release? Thanks.
>>
>>
>>
>> STATEMENT OF CONFIDENTIALITY The information contained in this email
>> message and any attachments may be confidential and legally privileged and
>> is intended for the use of the addressee(s) only. If you are not an
>> intended recipient, please: (1) notify me immediately by replying to this
>> message; (2) do not use, disseminate, distribute or reproduce any part of
>> the message or any attachment; and (3) destroy all copies of this message
>> and any attachments.
>>
>

Re: EnforceOrder processor

[Pierre Villard]

Hey Boris,

It's certainly a processor that would benefit an "additional details"
documentation page. If you're willing to file a JIRA and submit a PR
explaining the processor with your words, that would certainly be much
appreciated and I'm sure Koji will be happy to review it.

Thanks,
Pierre

Le lun. 12 nov. 2018 à 21:13, Boris Tyukin  a écrit :

> I was really confused how EnforceOrder processor works and NiFi
> documentation made it even more confusing. After some time looking for an
> explanation, I found this gist
>  which
> I think was created by a developer who created this processor. It does much
> better job explaining the processor and various options and relationships
> than NiFi doc so wanted to post it here for someone who is looking for the
> same information.
>
> https://gist.github.com/ijokarumawak/88fc30a2300845b3c27a79113fc72d41
>
>
>