Christopher McDermott created NIFI-1642:
-------------------------------------------
Summary: PutKafka should validate topic expression and calculated
value
Key: NIFI-1642
URL: https://issues.apache.org/jira/browse/NIFI-1642
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Reporter: Christopher McDermott
Priority: Minor
PutKafka does not validate the expression supplied for the topic property, like
most other processors. It should also try to validate the evaluated value of
the topic to see if it is a legal topic name. Note I'm not suggesting that the
topic need to exist, just that the name is compliant to what Kafka will accept.
This would be most helpful because if certain (probably not all) illegal
names are used, the Kafka client throws bizarre and most unhelpful exceptions.
-----------------
Chris,
Assuming the client can validate #2 i am with you. Please do feel
free to fire up a JIRA for this.
Thanks
Joe
On Wed, Mar 16, 2016 at 1:24 PM, McDermott, Chris Kevin (MSDU -
STaTS/StorefrontRemote) <[email protected]> wrote:
It turns out the root cause of the problem was an invalid topic name. Strange
error for that!
I think there are a couple of improvements could be made to PutKafka.
1. Check the validity of the the expression in the topic property.
2. Check the validity of the topic name before attempting to write to the topic.
Chris
On 3/16/16, 11:41 AM, "McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)"
<[email protected]> wrote:
Joe,
I’ll checkout the disk-space. We are running 0.9. If disk space is not the
issue we’ll give 0.8 a try.
Thanks very much for your quick reply.
Cheers,
Chris
On 3/16/16, 11:04 AM, "Joe Witt" <[email protected]> wrote:
Chris,
I have seen that when the diskspace kafka relies on is full. We've
seen a number of interesting exceptions recently in testing various
configurations. But recommend checking that.
Also, what version of Kafka broker are you using? With Apache NiFi
0.5.x we moved to the kafka client 0.9. In doing that we messed up
support for 0.8. So...with the upcoming release we will move back to
the 0.8 client and thus it works great with Kafka 0.8 and 0.9 brokers
albeit without the new SSL and Kerberos support they added in their
0.9 work. We have a JIRA item to go after that for our next feature
bearing release.
Thanks
Joe
On Wed, Mar 16, 2016 at 11:01 AM, McDermott, Chris Kevin (MSDU -
STaTS/StorefrontRemote) <[email protected]> wrote:
I say strange because the timeout (63ms) is so very short. The communication
timeout I’ve set is 30 sec. Has anyone overseen this?
2016-03-16 14:41:38,227 ERROR [Timer-Driven Process Thread-8]
o.apache.nifi.processors.kafka.PutKafka
PutKafka[id=852c8d42-a2fa-3478-b06b-84ceb6\
6f8b0b] Failed to send
StandardFlowFileRecord[uuid=a0074162-0066-49e7-918b-cea1cfc5a955,claim=StandardContentClaim
[resourceClaim=StandardResour\
ceClaim[id=1458079089737-67, container=default, section=67], offset=377796,
length=743],offset=0,name=2349680613178720,size=743] to Kafka; routi\
ng to 'failure'; last failure reason reported was
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata
after 63 ms.;: org.\
apache.kafka.common.errors.TimeoutException: Failed to update metadata after 63
ms.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
