So indeed I know cases as well where a company has over 30 FTP accounts to access.
My idea was to have them all listed as VFS Authentication metadata objects and that you could shield the technical details in the metadata with a physical to logical mapping. Maybe it could be a new VFS driver like link://ftp1/ which maps to ftp://user:pass@server1:/somefolder essentially protecting you from changes like IP address or authentication. There's a metadata input transform as well which would allow us to list all these URLs then and so on. Just an idea. Cheers Matt Op vr 25 feb. 2022 19:29 schreef Brandon Jackson <[email protected]>: > I definitely agree with the use cases. Frequently, I gather data from > different accounts. My examples below are probably more of the same that > you have in your original requirements observation. > > Getting files from one SFTP site using keys for auth and putting the files > to a different location using passwords. > Dropbox two different accounts. - Same scenario. > Different google accounts with access to different projects. > In the past, with SSH, when I configured it at the user level with key > authorization, I remembered being able to use VFS to access the locations > and not worry about the system prompting for passwords. That was really > nice because I did not have to bury the credentials in transforms. > Getting and placing files using Box. > > One thing I find myself doing more of these days is writing and running > python scripts using the requests and json modules to authenticate and hit > against rest endpoints and save the files for further processing in Hop. > It is otherwise pretty tricky to gather all the headers, sessions, and > cookies to interact with endpoints. I mention this because the > authentication against sharepoint is a bit tricky. I wanted enough > authentication landscape out there to come up with a good scheme for > storing and using credentials, whether it is VFS or other parts of the > system like Rest Client etc. It would be nice to have visibility of all > the different credentials enabling access from a centralized standpoint, > even if in practice certs are placed here and there or environment > variables walk in, or parameters pass in. I need a way to see the auth > landscape. > > Brandon > > On Fri, Feb 25, 2022 at 8:12 AM Matt Casters <[email protected] > .invalid> > wrote: > > > Dear Hoppists, > > > > Once in a blue moon the topic of authentication against Apache VFS > > protocols comes up again. > > The issues discussed focus on a number of different requirements that > come > > up during discussions in various forms on the chat channels and > elsewhere. > > > > * It should be possible to use non-system level variables when > configuring > > VFS features. This is because we don't expose any IVariables objects > into > > the various HopVfs calls that resolve FileObjects, > InputStream/OutputStream > > and so on. Important features should be configurable through a standard > > environment configuration file to match our lifecycle management vision. > > > > * We should be able to tie the authentication of VFS file systems against > > the Authentication Provider plugins which are already present in Apache > Hop > > but go unused. For example if we have a secure FTP target and we want to > > authenticate using a secure key we should be able to provide that key and > > use that to authenticate against a certain hostname. > > > > * It should be possible to authenticate differently to several systems in > > the same project or environment. For example it should be possible to > > authenticate to different hdfs:// servers at the same time or use > multiple > > Amazon s3:// accounts at the same time. This would allow us to transfer > > data from one account to another for example. > > > > * It should be possible to create symbolic links through our VFS driver > > usage where we map one folder to another hiding the implementation > > details. > > > > See also: https://issues.apache.org/jira/browse/HOP-3795 > > > > Architecturally I think that what this means is that we first need to > > create Authentication metadata elements which use the Authentication > > Provider plugins. > > > > https://issues.apache.org/jira/browse/HOP-3796 > > > > After that we can use those authentication provider metadata elements to > > make it super easy to configure VFS authentication simply by pointing to > > whichever method we want to use. We can do that in Hop VFS > Authentication > > metadata elements. > > > > Technically what this means is that almost 1000 method uses of class > HopVfs > > need to be fed the appropriate IVariables and IHopMetadataProvider > instance > > at the location where the method is used (the lowest level possible). > This > > sounds like a lot of work but I feel like that we should get this out of > > the way sooner rather than later since the topic just keeps coming back. > > The idea that nobody else out there is doing even remotely like it is > not a > > deterrent, it's an incentive :-) > > > > I would love to hear your ideas and check if anyone else had any > > requirements that we should take into account. > > > > Thanks in advance, > > > > Matt > > >
