Mark,
I this point on my project we are evaluating different workflow
technologies. I was asked to take a look at NiFi which looks pretty
cool. At some point the powers that be will make a determination as to
what software we will be using. If we go with NiFi I would definitely
work on solving this problem. I can't commit at this time since its not
in my tasking. I'll keep you posted.
Thanks,
Mike
On 07/05/2016 09:15 AM, Mark Payne wrote:
Hi Mike,
Personally, I'm not familiar with DynamoDB either, but a quick look at
the source & configuration do indeed
show that it expects the user to fetch only a single 'attribute' from
a Dynamo Item. Looking at the AWS javadocs,
it appears that this is the typical usage, as they don't seem to
provide a method for obtaining the entire item
as JSON (unless I'm missing something).
One approach, though, would be to allow the 'Json Document attribute'
property to be optional instead of required
and if not specified, simply iterate over all of the keys in the
Dynamo DB Item and create a JSON object ourselves
that we write to the content of the FlowFile.
Often what happens, though, is that a member of the NiFi community
develops a processor to provide the functionality
that they need and sometimes makes the processor a little more
'generic' to handle additional use cases, if it makes sense.
As the community begins to use the processor, it begins to evolved
over time to meet more and more use cases. This is
a fairly new processor and so likely will evolve over time to meet
more & more use cases. I think the use case of pulling back
an entire DynamoDB item is very reasonable and likely will be needed
by others.
Is there any chance that you're interested in working on building out
that capability and contributing it to the community?
We would be more than happy to help shepherd you through the process
of contributing it back to make NiFi's dynamo
support better!
Thanks
-Mark
On Jul 5, 2016, at 8:51 AM, Mike Connolly <[email protected]
<mailto:[email protected]>> wrote:
James,
Hmmm. I'm not too familiar with DynamoDB or NiFi, but having a
processor that can only return one attribute at a time doesn't seem
very useful. I would think the processor would retrieve all the
attributes and package them into one variable and send it back as one
unit so when the user configures the processor they could do
something like set the json attribute value to "ALL" or something
like that. I tried modifying the processor source code to call
item.getJSON() to populate the return value with everything, but
something isn't liking { brace in the string value that gets sent
back. Its complaining about unacceptable characters in the string. I
might need to swich to MongoDB as my datasource and see if that works.
Thanks,
Mike
On 07/01/2016 02:41 AM, James Wing wrote:
Mike,
I believe it was an intentional design choice to separate the
hash/range keys from the flowfile content for validation and
handling. There is currently no option in PutDynamoDB or
GetDynamoDB to match the entire flowfile content to the entire
DynamoDB item, but I believe that is certainly a possible extension.
Thanks,
James
On Thu, Jun 30, 2016 at 9:57 AM, Mike Connolly
<[email protected] <mailto:[email protected]>> wrote:
I checked out the master branch of Nifi and noticed the
GetDynamoDB processor which was something I really needed for a
workflow. I tried using it to return the json representation of
a DynamoDB item. It looks like it will only return a json
representation of a a single attribute of a DynamoDB item. I'm
wondering it this was the intention or if there is a specific
value you need to set in the processor configuration that will
return the whole item. Anybody ever use this?
Thanks,
Mike Connolly
