Wonderful suggestions. I think I would like to think about how the samples themselves operate though. Often we will need to read from files or even a database on occasion. We can use a H2 database for that but in any case we'd need a place for the samples to read from and also to write to. Perhaps it's as easy as having a consensus on input files being placed in an input/ folder and output to go to /output? Any temporary files perhaps can go in a tmp/ folder? What if the same file exists in multiple folders? Can we safely overwrite or should we have a separate logic for shared input files?
"Data sets" as such are used for unit testing and I think should be seen separate from the various file formats that Hop can read. I've seen people mix the terminology and I really don't think that's a good idea going forward. Sure, a data set is stored as a CSV file but it has a particular format. Cheers, Matt On Tue, Jan 26, 2021 at 9:05 AM Sergio Ramazzina (SERASOFT) < [email protected]> wrote: > OK I will do! > > Il 25/01/2021 21:19, Hans Van Akelyen ha scritto: > > Hey Sergio, > > > > You have my go! > > I would create a fixed project (with default run engine/configuration) > and > > a "datasets" directory in it where we can add some common data that can > be > > used. > > The plugins can then add their own "special" data there and indeed add a > > folder per plugin where the pipelines can reside. > > I think it will take some time and tweaking to get everything right but > it > > will be a great start. > > > > Cheers, > > Hans > > > > On Mon, Jan 25, 2021 at 9:13 PM Sergio Ramazzina < > > [email protected]> wrote: > > > >> Hi all, > >> > >> I do agree with Hans about the idea of putting plugin's samples together > >> docs in a subdirectory of the plugin module. If you agree I can start > >> implementing this way the samples for Enhanced JSON Output. > >> > >> I can also modify the build procedure so that they can be packaged all > >> together in a samples directory where each plugin's samples are > separated > >> by a specific subdirectory. > >> > >> Let me know what you think > >> > >> Sergio Ramazzina > >> Sent from my iPhone > >> > >>> Il giorno 25 gen 2021, alle ore 19:58, Bart Maertens < > >> [email protected]> ha scritto: > >>> Hi All, > >>> > >>> Why not aim for the best of both worlds? Imho, samples are similar to > >> docs: > >>> each action and transform plugin should come with at least one sample > >>> workflow or pipeline that shows its functionality with as few other > >> actions > >>> or transforms as possible. > >>> We can bundle those into a 'samples' project, and add more complex > >> example > >>> workflows and pipelines that show users how to work in more complex > >>> situations, implement a number of sample design patterns etc > >>> > >>> Regards, > >>> Bart > >>> > >>> > >>> > >>>> On Mon, Jan 25, 2021 at 11:16 AM Hans Van Akelyen < > >>>> [email protected]> wrote: > >>>> > >>>> Hi Sergio, > >>>> > >>>> So I think there are a couple things we should consider. The simplest > >>>> approach will be to, as you suggested, create a folder in static or > some > >>>> other location and dump all our samples in there. Another idea that > has > >>>> come up here and there but should be discussed is to look at the > >> samples in > >>>> the same way we are doing our docs, create a sample folder per plugin. > >> This > >>>> gives us a clear overview which plugins have samples and in the future > >> we > >>>> could add extra checks or warnings to our build process to indicate > the > >>>> quality of a plugin. For now we are also planning to ship all our > >>>> transforms and actions in our distribution release, but at the speed > our > >>>> set of tools is growing we will have to start choosing at some point > in > >>>> time and this could mean that samples break. > >>>> > >>>> Creating samples for each plugin will require a bit more work/thinking > >> as > >>>> we need something "smart" to combine all these samples into an actual > >>>> sample project that is included in our release. But my personal > >> preference > >>>> would be to look at something like this. And apply the same logic to > our > >>>> integration tests. > >>>> > >>>> Cheers, > >>>> Hans > >>>> > >>>> On Thu, Jan 21, 2021 at 11:03 PM Sergio Ramazzina (SERASOFT) < > >>>> [email protected]> wrote: > >>>> > >>>>> Ciao Hoppers, > >>>>> > >>>>> 'm here to start a discussion around how we can create a sample > >>>>> distribution because I was preparing samples for my Enchanced JSON > >>>>> Output but I discovered that I was unaware about where I can put them > >> in > >>>>> our project's codebase structure. Hans told me in our dev chat that > >>>>> there isn't any rule or guideline on how to do that and proposed me > to > >>>>> start a discussion in the dev list to collect ideas on how to do that > >> so > >>>>> here I am. > >>>>> > >>>>> As Matt said in one of our discussions, a good idea could be to > >> generate > >>>>> a samples project and put the project's directory our codebase. I > >> agree > >>>>> on that. For example under the assembly/static module, in resouce > >> folder > >>>>> we could create a new folder named samples and put the project there. > >>>>> Then we can include it hop's application archive for distribution. > >>>>> > >>>>> Any ideas comments on this topic? > >>>>> > >>>>> Cheers > >>>>> > >>>>> Sergio > >>>>> > >>>>> -- > >>>>> > -- > Best Regards/Distinti Saluti > Sergio Ramazzina > Senior Solution Architect > Via Milano 78 > 20013 Magenta (MI) - ITALY > mobile : +39 347 2103689 > Tel: +39 02 87158700 > Fax: +39 02 87151947 > website: http://www.serasoft.it <http://www.serasoft.it> - email : > [email protected] <mailto:[email protected]> > skype : sramazzina - Follow me on twitter: sramazzina > View my profile on LinkedIn: http://www.linkedin.com/in/sramazzina > <http://www.linkedin.com/in/sramazzina> > > -- Neo4j Chief Solutions Architect *✉ *[email protected] ☎ +32486972937
