I agree that convention over configuration seems to be the more efficient approach for samples, but can probably also be applied to integration testing.
How about referring to "data sets" as "test data" or "test data sets" going forward to avoid confusion? It probably is not obvious for new Hop users that a "data set" is related to testing... Regards, Bart On Tue, Jan 26, 2021 at 9:14 AM Matt Casters <[email protected]> wrote: > Wonderful suggestions. > I think I would like to think about how the samples themselves operate > though. > Often we will need to read from files or even a database on occasion. We > can use a H2 database for that but in any case we'd need a place for the > samples to read from and also to write to. > Perhaps it's as easy as having a consensus on input files being placed in > an input/ folder and output to go to /output? > Any temporary files perhaps can go in a tmp/ folder? > What if the same file exists in multiple folders? Can we safely overwrite > or should we have a separate logic for shared input files? > > "Data sets" as such are used for unit testing and I think should be seen > separate from the various file formats that Hop can read. I've seen people > mix the terminology and I really don't think that's a good idea going > forward. Sure, a data set is stored as a CSV file but it has a particular > format. > > Cheers, > Matt > > > On Tue, Jan 26, 2021 at 9:05 AM Sergio Ramazzina (SERASOFT) < > [email protected]> wrote: > > > OK I will do! > > > > Il 25/01/2021 21:19, Hans Van Akelyen ha scritto: > > > Hey Sergio, > > > > > > You have my go! > > > I would create a fixed project (with default run engine/configuration) > > and > > > a "datasets" directory in it where we can add some common data that can > > be > > > used. > > > The plugins can then add their own "special" data there and indeed add > a > > > folder per plugin where the pipelines can reside. > > > I think it will take some time and tweaking to get everything right but > > it > > > will be a great start. > > > > > > Cheers, > > > Hans > > > > > > On Mon, Jan 25, 2021 at 9:13 PM Sergio Ramazzina < > > > [email protected]> wrote: > > > > > >> Hi all, > > >> > > >> I do agree with Hans about the idea of putting plugin's samples > together > > >> docs in a subdirectory of the plugin module. If you agree I can start > > >> implementing this way the samples for Enhanced JSON Output. > > >> > > >> I can also modify the build procedure so that they can be packaged all > > >> together in a samples directory where each plugin's samples are > > separated > > >> by a specific subdirectory. > > >> > > >> Let me know what you think > > >> > > >> Sergio Ramazzina > > >> Sent from my iPhone > > >> > > >>> Il giorno 25 gen 2021, alle ore 19:58, Bart Maertens < > > >> [email protected]> ha scritto: > > >>> Hi All, > > >>> > > >>> Why not aim for the best of both worlds? Imho, samples are similar to > > >> docs: > > >>> each action and transform plugin should come with at least one sample > > >>> workflow or pipeline that shows its functionality with as few other > > >> actions > > >>> or transforms as possible. > > >>> We can bundle those into a 'samples' project, and add more complex > > >> example > > >>> workflows and pipelines that show users how to work in more complex > > >>> situations, implement a number of sample design patterns etc > > >>> > > >>> Regards, > > >>> Bart > > >>> > > >>> > > >>> > > >>>> On Mon, Jan 25, 2021 at 11:16 AM Hans Van Akelyen < > > >>>> [email protected]> wrote: > > >>>> > > >>>> Hi Sergio, > > >>>> > > >>>> So I think there are a couple things we should consider. The > simplest > > >>>> approach will be to, as you suggested, create a folder in static or > > some > > >>>> other location and dump all our samples in there. Another idea that > > has > > >>>> come up here and there but should be discussed is to look at the > > >> samples in > > >>>> the same way we are doing our docs, create a sample folder per > plugin. > > >> This > > >>>> gives us a clear overview which plugins have samples and in the > future > > >> we > > >>>> could add extra checks or warnings to our build process to indicate > > the > > >>>> quality of a plugin. For now we are also planning to ship all our > > >>>> transforms and actions in our distribution release, but at the speed > > our > > >>>> set of tools is growing we will have to start choosing at some point > > in > > >>>> time and this could mean that samples break. > > >>>> > > >>>> Creating samples for each plugin will require a bit more > work/thinking > > >> as > > >>>> we need something "smart" to combine all these samples into an > actual > > >>>> sample project that is included in our release. But my personal > > >> preference > > >>>> would be to look at something like this. And apply the same logic to > > our > > >>>> integration tests. > > >>>> > > >>>> Cheers, > > >>>> Hans > > >>>> > > >>>> On Thu, Jan 21, 2021 at 11:03 PM Sergio Ramazzina (SERASOFT) < > > >>>> [email protected]> wrote: > > >>>> > > >>>>> Ciao Hoppers, > > >>>>> > > >>>>> 'm here to start a discussion around how we can create a sample > > >>>>> distribution because I was preparing samples for my Enchanced JSON > > >>>>> Output but I discovered that I was unaware about where I can put > them > > >> in > > >>>>> our project's codebase structure. Hans told me in our dev chat that > > >>>>> there isn't any rule or guideline on how to do that and proposed me > > to > > >>>>> start a discussion in the dev list to collect ideas on how to do > that > > >> so > > >>>>> here I am. > > >>>>> > > >>>>> As Matt said in one of our discussions, a good idea could be to > > >> generate > > >>>>> a samples project and put the project's directory our codebase. I > > >> agree > > >>>>> on that. For example under the assembly/static module, in resouce > > >> folder > > >>>>> we could create a new folder named samples and put the project > there. > > >>>>> Then we can include it hop's application archive for distribution. > > >>>>> > > >>>>> Any ideas comments on this topic? > > >>>>> > > >>>>> Cheers > > >>>>> > > >>>>> Sergio > > >>>>> > > >>>>> -- > > >>>>> > > -- > > Best Regards/Distinti Saluti > > Sergio Ramazzina > > Senior Solution Architect > > Via Milano 78 > > 20013 Magenta (MI) - ITALY > > mobile : +39 347 2103689 > > Tel: +39 02 87158700 > > Fax: +39 02 87151947 > > website: http://www.serasoft.it <http://www.serasoft.it> - email : > > [email protected] <mailto:[email protected]> > > skype : sramazzina - Follow me on twitter: sramazzina > > View my profile on LinkedIn: http://www.linkedin.com/in/sramazzina > > <http://www.linkedin.com/in/sramazzina> > > > > > > -- > Neo4j Chief Solutions Architect > *✉ *[email protected] > ☎ +32486972937 >
