Github user jvwing commented on the pull request:
https://github.com/apache/nifi/pull/239#issuecomment-222057686
Your suggestion about the plain old SDK Kinesis API is interesting, but I
guess I don't recommend it. Using the plain SDK would let NiFi handle the
multithreading, retries, batching, etc. It would be a win for the NiFi model.
But then we would also have to pick up at least some of the KCL's other
features, like balancing traffic across shards, or leave that as an excercise
for the user. It sounds complicated and hard, and I'm lazy. Also, the KCL is
recommended by Amazon, as you pointed out earlier, and not using it might be a
point of confusion as to why we didn't do it the "right" way, especially if a
new version of the KCL is released with features we hadn't thought of. So even
though the KCL is an awkward fit in NiFi, as long as it is an opt-in feature,
it seems like a good addition to NiFi's AWS interoperability story.
Part of why I would prefer to not change the base AWS processor classes is
to keep it opt-in. I'm not sure how the other AWS Get* processors would
benefit from the Kinesis-like threading model, they do not have applications or
threads to drive activity outside of NiFi's scheduling. But I can see lobbying
for a change to `AbstractProcessor` to remove `onTrigger`'s `final` modifier
for cases like these without divorcing the class hierarchies.
A recently merged [change in the 0.x
branch](https://github.com/apache/nifi/commit/de7ecd719a2e9907042628ea3a8283cfe2d4fbac)
for NIFI-786 has some shared credential property descriptors and validation
logic that you may be able to use, that will hopefully make it easier to
implement separate base classes. Let me know if I can help with that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
