Gene Peters created NIFI-3449:
---------------------------------
Summary: Create Google Cloud Platform/Google Cloud Storage
Processors
Key: NIFI-3449
URL: https://issues.apache.org/jira/browse/NIFI-3449
Project: Apache NiFi
Issue Type: New Feature
Components: Extensions
Reporter: Gene Peters
Hi all,
We had a need in our production deployments to interact with Google Cloud
Storage. At the time, NIFI-2809 hadn't seen much movement, and after applying
the patch I found that the configuration was too specific for my needs (it's
hardcoded to use "Application Default" credentials, everything uploaded to GCS
is uploaded with the "public" ACL, etc). So I created a series of Processors /
Controller Services based off of the AWS NiFi library, and would like to
contribute them.
Features:
* All credentialing is handled by a controller service, allowing multiple
processors to use the same service / credentials
* An Abstract processor is provided which forms the basis for all GCP related
processors.
* The standard Google Cloud Storage operations are supported, very similarly to
the AWS S3 processors: ListGCSBucket, DeleteGCSObject, FetchGCSObject,
PutGCSObject
* Everything is documented and unit tested.
* I've also provided integration tests, but they're disabled by default (as
they require Google Cloud credentials). To run them, use the flag
{{skipGCPIntegrationTests=false}}
Todo:
* The GCP Java library's ReadChannel objects implement the "restorable"
interface, which allows for state saving / checkpointing. I'd really like to
leverage this with the State support that NiFi provides, but it would require
serializing / deserializing the object.
I'm going to be submitting this as a pull request through GitHub.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
