Hi All, One of our main goals is to make configuration as simple as possible. I am all in to create a metadata object for VFS, I agree with Nicolas that a VFS object could be like we do our database connections, pick a type and specific option fields become visible to define the connection.
I also think we could go a bit further than only saving the global settings/authentication. For S3 the general idea I have would be to create a vfs object named "test-bucket" it would contain the root path to s3 and how to authenticate, be it the system variables, a keyfile,... In our input transforms we could then get files using this named object and an added path. "vfs:test-bucket:/path/to/file.csv" (format subjected to change). This also adds the ability to use multiple different buckets/accounts with different keys/locations making Hop even more flexible. Let me know if this idea is totally bananas or a possible path to follow. Cheers, Hans On Wed, Feb 17, 2021 at 8:25 PM Nicolas ADMENT <[email protected]> wrote: > Dear Hop thinker, > > I think we should create a new type of metadata to share OAUTH, VFS, FTP, > SSH, IMAP, POP3, AS400, SAP... connection settings that we could reuse in > different plugins. > This new type of metadata object could be called something like > "IConnection" with a name, protocol type, and custom properties depending > on protocol type (host/login/password, Secret ID/Key, ....), much like > databases. > > For VFS, the connection name must be the same as the protocol scheme of the > URL used. > To configure an amazon connection, you must define the predefined name S3 > and select the "Amazon Web Services Authentication" protocol, then set > authentication the properties. > > Some web based protocols may have common properties like proxy settings: > > We could be inspired by: > https://www.visualcron.com/doc/html/connection-visualcron.html > > Maybe it will give some idea. > > Regards, > Nicolas > > > > <#m_-6113234865807150246_m_-2555803484072009614_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > > Le mer. 17 févr. 2021 à 11:29, Matt Casters <[email protected] > .invalid> > a écrit : > > > Dear Hoppiverse, > > > > Apache VFS is a very useful tool for supporting alternative ways of > storing > > files. > > We have a plugin type for VFS plugins as well as for allowing us to > extend > > functionality. > > Where we're not doing a great job is in supporting specific driver > features > > like authentication where this is needed. > > For example, Amazon AWS needs an access key and a secret key (among a > list > > of authentication options). > > > > Usually the configuration of these access APIs underpinning the VFS > drivers > > is left as standard as possible. > > However, this causes it to be dragged outside of the Hop-isphere in the > > sense that you have effects like the following: > > - variables can't be used. > > - obfuscation/encryption of keys and passwords is not done > > - No GUI or configuration elements to configure the security > > - No metadata wrappers are available or supported > > ... > > > > To get past these shortcomings I propose to turn HopVfs from a singleton > > into a proper class. > > Class HopVfs is used about 800 times in the source code. It's a wrapper > > around the Apache VFS API. > > > > In the ideal case we would have a new metadata object type called > something > > like : > > "Amazon Web Services Authentication" which would have fields like access > > and secret key but also perhaps a checkbox: [x] configured by the system. > > We could then have a global configuration option for the S3 VFS driver > > which simply says which "AWS Auth" object to use. > > > > This would address many if not all concerns. The "only" thing we need to > > change is moving from: > > > > HopVfs.getFileObject(filename) > > > > to > > > > new HopVfs(metadataProvider, variables).getFileObject(filename) > > > > Obviously a case can be made to cache the HopVfs objects at various > > locations (one per pipeline, workflow, ...). > > You get the idea. > > > > I have a feeling that if we don't do this we'll continue to carry around > > this architectural debt for quite a while more into other territories > like > > Azure and Google. > > > > Thoughts? > > > > Matt > > -- > > Neo4j Chief Solutions Architect > > *✉ *[email protected] > > ☎ +32486972937 > > > > < > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > > > Garanti > sans virus. www.avast.com > < > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > > > <#m_-6113234865807150246_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >
