I'm interested although it might consume some time since I don't know how big it is going to be. And I suppose it is better to capture it in a separate Jira. The summary could be upgrade and refactor GCP processor codebase or something like that. We could then make NIFI-5133 a dependent of the new Jira. Thoughts?
Meanwhile, anyone who has any other recommendations, feel free to share. :) - Sivaprasanna On Thu, May 3, 2018 at 9:43 PM, Joe Witt <joe.w...@gmail.com> wrote: > Sivaprasanna > > Ok makes sense. Are you in a position of interest/expertise/time to > make those changes as well? > > Thanks > > On Thu, May 3, 2018 at 12:11 PM, Sivaprasanna <sivaprasanna...@gmail.com> > wrote: > > Hi. As I had mentioned, upgrading to the latest version of the library is > > not as simple as I thought. Google Cloud team introduced many breaking > > changes. Many of the APIs (classes & methods) have been > > scrapped/replaced/modified/refactored/renamed. > > > > In short, a simple change of version may demand changes on the > processor's > > code, especially on the AbstractProcessors (AbstractGCS, AbstractGCP) > which > > may pose backward compatibility issues, I'm afraid. > > > > Thanks, > > Sivaprasanna > > > > On Thu, May 3, 2018 at 9:26 PM, Joe Witt <joe.w...@gmail.com> wrote: > > > >> Sivaprasanna > >> > >> I might not completely follow but is there a 3rd option to upgrade to > >> a more recent library and solve the use of the proper jars > >> problem(smaller nar)? > >> > >> Thanks > >> > >> On Thu, May 3, 2018 at 11:51 AM, Sivaprasanna < > sivaprasanna...@gmail.com> > >> wrote: > >> > Hi > >> > > >> > I've started the initial works on implementing Google Cloud Pub/Sub > >> > processors. The associated Jira ID is NIFI-5133 > >> > <https://issues.apache.org/jira/browse/NIFI-5133>. This will go to > the > >> > existing GCP bundle which currently has only the storage processors. > Upon > >> > some inspection, I noticed the following: > >> > > >> > - As of now, the bundle uses google-cloud > >> > <http://mvnrepository.com/artifact/com.google.cloud/google-cloud> > as > >> its > >> > dependency which is like uber/fat jar that contains most of the > Google > >> > Cloud's client library SDKs including storage, bigquery, pubsub, > etc. > >> The > >> > main point is it is using a very older version (0.8.0) > >> > - I thought of using google-cloud-bom > >> > <http://mvnrepository.com/artifact/com.google.cloud/ > google-cloud-bom> > >> in > >> > the bundle's POM > >> > <https://github.com/apache/nifi/blob/master/nifi-nar- > >> bundles/nifi-gcp-bundle/pom.xml> > >> > and then use the required artifacts in the processor's POM > >> > <https://github.com/apache/nifi/blob/master/nifi-nar- > >> bundles/nifi-gcp-bundle/nifi-gcp-processors/pom.xml>. > >> > The benefit is, it will help us reduce the overall size of the NAR. > >> > > >> > When I tried to do #2, I realized this is not a simple version change > >> but a > >> > change that brings backward compatibility issues. Ex: Some APIs used > in > >> the > >> > older version i.e. 0.8.0 have now been entirely scrapped and moved to > >> > different library. We can do either two things: > >> > > >> > 1. User the Pub/Sub APIs from the older version but the problem is > >> it's > >> > very old and the problem of upgrading would soon catchup with us. > >> > 2. Or we can continue to use the older version of > google-cloud-storage > >> > only for the storage processors and introduce the #2 mentioned > above > >> but I > >> > don't think then the new processors can't properly extend the > existing > >> > AbstractGCPProcessor > >> > <https://github.com/apache/nifi/blob/master/nifi-nar- > >> bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/ > >> org/apache/nifi/processors/gcp/AbstractGCPProcessor.java>. > >> > > >> > > >> > A quick glance on the processor code and the POM would help you > >> understand > >> > my concern. > >> > > >> > I'm stuck up here so any help & guidance in this regard is very much > >> > appreciated. :) > >> > > >> > Thanks, > >> > > >> > Sivaprasanna > >> >
