Created NIFI-6881 and I've got an almost working implementation. Something I'm
trying to work out is how to store the keys. Most databases don't support byte
arrays for primary keys. I know in virtually every case in NiFi the key's will
be Strings but I'm wondering what to do if they weren't. I could base64
everything but that will make it hard to figure out the column sizing for keys.
I'm trying to keep the SQL generic enough that it should work on Oracle, MySQL,
Postgres, DB2, Derby, H2 and maybe SQL Server if it doesn't do something odd.
Thanks
On 11/14/19, 8:17 AM, "Matt Burgess" <mattyb...@apache.org> wrote:
Shawn,
There are also Redis and Couchbase distributed map cache clients
already in NiFi. I don't see any Jiras or PRs related to a DynamoDB or
JDBC ones. I thought about making ones for JDBC, Hazelcast and/or
Nitrite [1] (with or without a DataGate server), but never got around
to it. I think DynamoDB and JDBC implementations would be helpful, the
latter could support DynamoDB in the meantime using the Simba JDBC
driver [2].
While thinking about the JDBC one, I figured it might be nice to be
able to cache the table locally for X amount of time or N number of
entries, in case you pre-populate the cache and are just reading it
with the client. Any write (from the NiFi client) would invalidate the
cache and the table would be re-fetched on the next read operation. I
did something similar for the DatabaseRecordLookupService, but that's
a read-only service so I didn't have to worry about writes, I was just
trying to improve performance where possible.
Regards,
Matt
[1] https://www.dizitart.org/nitrite-database.html
[2] https://www.simba.com/drivers/dynamodb-odbc-jdbc/
On Thu, Nov 14, 2019 at 8:31 AM Shawn Weeks <swe...@weeksconsulting.us>
wrote:
>
> Has anyone already done some work on adding new services for distribute
map cache? I’m looking at moving to aws and I really don’t want to have to run
emr just for hbase. I’ve been thinking about starting on either a DynamoDB or
simple jdbc implementation.
>
> Thanks
> Shawn
>
> Sent from my iPhone
