Well, because I was thinking that "data sets" (what's in a name) can be applied beyond testing in a not so distant future. In a lot of projects you have these small sets of data that we now put in a "Data Grid", an excel file and so on. It would be a lot better IMO to formalize those in some way. I don't care what it's called but as a re-usable concept I like the idea a lot.
On Tue, Jan 26, 2021 at 9:31 AM Bart Maertens <[email protected]> wrote: > I agree that convention over configuration seems to be the more efficient > approach for samples, but can probably also be applied to integration > testing. > > How about referring to "data sets" as "test data" or "test data sets" going > forward to avoid confusion? > It probably is not obvious for new Hop users that a "data set" is related > to testing... > > Regards, > Bart > > > On Tue, Jan 26, 2021 at 9:14 AM Matt Casters <[email protected] > .invalid> > wrote: > > > Wonderful suggestions. > > I think I would like to think about how the samples themselves operate > > though. > > Often we will need to read from files or even a database on occasion. We > > can use a H2 database for that but in any case we'd need a place for the > > samples to read from and also to write to. > > Perhaps it's as easy as having a consensus on input files being placed in > > an input/ folder and output to go to /output? > > Any temporary files perhaps can go in a tmp/ folder? > > What if the same file exists in multiple folders? Can we safely > overwrite > > or should we have a separate logic for shared input files? > > > > "Data sets" as such are used for unit testing and I think should be seen > > separate from the various file formats that Hop can read. I've seen > people > > mix the terminology and I really don't think that's a good idea going > > forward. Sure, a data set is stored as a CSV file but it has a > particular > > format. > > > > Cheers, > > Matt > > > > > > On Tue, Jan 26, 2021 at 9:05 AM Sergio Ramazzina (SERASOFT) < > > [email protected]> wrote: > > > > > OK I will do! > > > > > > Il 25/01/2021 21:19, Hans Van Akelyen ha scritto: > > > > Hey Sergio, > > > > > > > > You have my go! > > > > I would create a fixed project (with default run > engine/configuration) > > > and > > > > a "datasets" directory in it where we can add some common data that > can > > > be > > > > used. > > > > The plugins can then add their own "special" data there and indeed > add > > a > > > > folder per plugin where the pipelines can reside. > > > > I think it will take some time and tweaking to get everything right > but > > > it > > > > will be a great start. > > > > > > > > Cheers, > > > > Hans > > > > > > > > On Mon, Jan 25, 2021 at 9:13 PM Sergio Ramazzina < > > > > [email protected]> wrote: > > > > > > > >> Hi all, > > > >> > > > >> I do agree with Hans about the idea of putting plugin's samples > > together > > > >> docs in a subdirectory of the plugin module. If you agree I can > start > > > >> implementing this way the samples for Enhanced JSON Output. > > > >> > > > >> I can also modify the build procedure so that they can be packaged > all > > > >> together in a samples directory where each plugin's samples are > > > separated > > > >> by a specific subdirectory. > > > >> > > > >> Let me know what you think > > > >> > > > >> Sergio Ramazzina > > > >> Sent from my iPhone > > > >> > > > >>> Il giorno 25 gen 2021, alle ore 19:58, Bart Maertens < > > > >> [email protected]> ha scritto: > > > >>> Hi All, > > > >>> > > > >>> Why not aim for the best of both worlds? Imho, samples are similar > to > > > >> docs: > > > >>> each action and transform plugin should come with at least one > sample > > > >>> workflow or pipeline that shows its functionality with as few other > > > >> actions > > > >>> or transforms as possible. > > > >>> We can bundle those into a 'samples' project, and add more complex > > > >> example > > > >>> workflows and pipelines that show users how to work in more complex > > > >>> situations, implement a number of sample design patterns etc > > > >>> > > > >>> Regards, > > > >>> Bart > > > >>> > > > >>> > > > >>> > > > >>>> On Mon, Jan 25, 2021 at 11:16 AM Hans Van Akelyen < > > > >>>> [email protected]> wrote: > > > >>>> > > > >>>> Hi Sergio, > > > >>>> > > > >>>> So I think there are a couple things we should consider. The > > simplest > > > >>>> approach will be to, as you suggested, create a folder in static > or > > > some > > > >>>> other location and dump all our samples in there. Another idea > that > > > has > > > >>>> come up here and there but should be discussed is to look at the > > > >> samples in > > > >>>> the same way we are doing our docs, create a sample folder per > > plugin. > > > >> This > > > >>>> gives us a clear overview which plugins have samples and in the > > future > > > >> we > > > >>>> could add extra checks or warnings to our build process to > indicate > > > the > > > >>>> quality of a plugin. For now we are also planning to ship all our > > > >>>> transforms and actions in our distribution release, but at the > speed > > > our > > > >>>> set of tools is growing we will have to start choosing at some > point > > > in > > > >>>> time and this could mean that samples break. > > > >>>> > > > >>>> Creating samples for each plugin will require a bit more > > work/thinking > > > >> as > > > >>>> we need something "smart" to combine all these samples into an > > actual > > > >>>> sample project that is included in our release. But my personal > > > >> preference > > > >>>> would be to look at something like this. And apply the same logic > to > > > our > > > >>>> integration tests. > > > >>>> > > > >>>> Cheers, > > > >>>> Hans > > > >>>> > > > >>>> On Thu, Jan 21, 2021 at 11:03 PM Sergio Ramazzina (SERASOFT) < > > > >>>> [email protected]> wrote: > > > >>>> > > > >>>>> Ciao Hoppers, > > > >>>>> > > > >>>>> 'm here to start a discussion around how we can create a sample > > > >>>>> distribution because I was preparing samples for my Enchanced > JSON > > > >>>>> Output but I discovered that I was unaware about where I can put > > them > > > >> in > > > >>>>> our project's codebase structure. Hans told me in our dev chat > that > > > >>>>> there isn't any rule or guideline on how to do that and proposed > me > > > to > > > >>>>> start a discussion in the dev list to collect ideas on how to do > > that > > > >> so > > > >>>>> here I am. > > > >>>>> > > > >>>>> As Matt said in one of our discussions, a good idea could be to > > > >> generate > > > >>>>> a samples project and put the project's directory our codebase. > I > > > >> agree > > > >>>>> on that. For example under the assembly/static module, in resouce > > > >> folder > > > >>>>> we could create a new folder named samples and put the project > > there. > > > >>>>> Then we can include it hop's application archive for > distribution. > > > >>>>> > > > >>>>> Any ideas comments on this topic? > > > >>>>> > > > >>>>> Cheers > > > >>>>> > > > >>>>> Sergio > > > >>>>> > > > >>>>> -- > > > >>>>> > > > -- > > > Best Regards/Distinti Saluti > > > Sergio Ramazzina > > > Senior Solution Architect > > > Via Milano 78 > > > 20013 Magenta (MI) - ITALY > > > mobile : +39 347 2103689 > > > Tel: +39 02 87158700 > > > Fax: +39 02 87151947 > > > website: http://www.serasoft.it <http://www.serasoft.it> - email : > > > [email protected] <mailto:[email protected]> > > > skype : sramazzina - Follow me on twitter: sramazzina > > > View my profile on LinkedIn: http://www.linkedin.com/in/sramazzina > > > <http://www.linkedin.com/in/sramazzina> > > > > > > > > > > -- > > Neo4j Chief Solutions Architect > > *✉ *[email protected] > > ☎ +32486972937 > > > -- Neo4j Chief Solutions Architect *✉ *[email protected] ☎ +32486972937
