Hi, just recognized that the format of my list was totally unreadable in the email I added my list in a propper way here [1]
Florian [1] https://issues.apache.org/jira/browse/STREAMPIPES-581 Am Mittwoch, dem 14.09.2022 um 16:52 +0200 schrieb Florian Micklich: > Hi all, > I made a rouge structure about a possible documentation. > • Introduction: ◦ Overview ◦ Basic Elements (former > StreamPipes Concept) ◦ Time in StreamPipes (Why timestamp is > important in the data?) ◦ Semantic in StreamPipes ◦ > Other > Topics? ◦ Installation ▪ Basic > Installation ▪ Docker Deployment ▪ K8s > / > K3s ▪ Use SSL ▪ Security ◦ > Getting > Help • Overview (Basic GUI Explanation and Introduction for each > GUI > point and why it is implemented (short description text) for example > Dashboard why Dashboard, what you can do. Basically little more deep > diver than Basic Elements section. Section with Most of the > Screenshots, So easy to maintain due changes) ◦ Login Page and > Basics ◦ Welcome Page ◦ Pipelines ◦ > Connect ◦ Dashboard ◦ Data Explorer ◦ Apps > (deprecated?) ◦ Install Pipeline Elements ◦ Asset > Management ◦ File Management ◦ Configuration & User > Prefences ◦ Notification • Data Source Elements > (Adapter) ◦ Why Adapter. Connect to the Real World ◦ > What > Kind of Sources (Data Set vs Data Stream / Categories) ◦ Basic > Adapter Setup in 4 Steps (at the moment) ▪ Step 1 > Settings ▪ Step 2 Select Format • > JSON > with Description for Array Field / GeoJSON / Single Object / Array > with > Short Data Examples • XML • > CSV > • Image ▪ Step 3 configure > fields • Different between Measurement, > Dimension > Header • Add Nested / Add static Value / Add > Timestamp • Edit fields (Big Thinks. > Also) • Preview ▪ Step4 start > adapters • Settings / Description (why Important > and where can I find this info later on?) • > Remove Duplicates / Reduce event rates /Persist events ◦ > Connect > Adapters Overview ▪ Apache Kafka with Example > Sources ▪ Apache Pulsar ▪ File > Set ▪ File Stream ▪ ... ◦ > Connect > Adapters IIOT Overview ▪ ... ◦ How To > Adapter ▪ Setup Adapter Example 1 ▪ > Setup > HTTPS Adapter ◦ Technical Concept of Adapters (Dev > Part???) • > Data Processors • Data Sinks • Create Pipeline • > Dashboard • Assets > The working example will be in the section HowTo and some example > resources in each adapter description.What do you think? > First step would be to check if it possible to move the files so that > the structure is not destroyed and the simply start with the adapters > :)I made already a rebase between dev and STREAMPIPES-581 branch, due > some changes in the dev branch. > Florian > Am Montag, dem 29.08.2022 um 01:07 +0000 schrieb Philipp Zehnder: > > Hi Florian, > > I am not sure if we will find many available public data sources, > > especially for PLCs. But maybe we find some docker containers that > > can be used for the tutorial. If we do not find some, maybe we can > > provide them in our registry.We could also use them for the > > automated > > tests. That would be great, because currently those tests are not > > integrated in the nightly run. > > I think for most users it is already helpful to have a working > > example for the different adapters, so they do not necessarily have > > to try out the examples. > > As you said we should collect all the information somewhere. Should > > we use thwiki for that? > > Cheers,Philipp________________________________Von: Florian Micklich > > < > > [email protected] > > >Gesendet: Samstag, August 27, 2022 1:20 AMAn: > > [email protected] > > < > > [email protected] > > >Betreff: Re: > > [DISCUSS] Documentation - Description for Adapters and Processing > > Elements > > Hi,perfekt, I will create this branch. I will think about a > > possiblestructure over the weekend. I let you know.How are you > > testing and using the adapters and wich one are you usingmost? Do > > you > > have some URL sources you always for developing processor testing > > and what is the typical workflow to add them?Maybe we can start to > > collect them in some way so I can add them later in > > thedocumentation.GreetingsFlorian > > Am Donnerstag, dem 25.08.2022 um 19:38 +0000 schrieb Dominik > > Riemer: > > > Sounds good!We usually use only the issue name as branch > > > name(STREAMPIPES-581) without the "feature" prefix, but both is > > > totallyfine!Dominik > > > -----Original Message-----From: Florian Micklich < > > > [email protected] > > > > > > > Sent: Wednesday, August 24, 2022 9:44 PMTo: > > > > > > [email protected] > > > > > > Subject: Re: [DISCUSS] Documentation - Description for Adapters > > > andProcessing ElementsHi Dominik,thanks for the answer and before > > > I > > > ask more questions,I'll simply start :-)Therefore I created a > > > ticket > > > https://issues.apache.org/jira/browse/STREAMPIPES-581 > > > > > > I will also create a feature branch feature/STREAMPIPES-581 or > > > whatis here best practise naming convention?GreetingFlorian > > > > > > Am Mittwoch, dem 24.08.2022 um 19:07 +0000 schrieb Dominik > > > Riemer: > > > > Hi Florian,happy to help! 1. There is a header in each > > > > markdownfilewith an ID (e.g. in the 01_try-installation.md). > > > > This > > > > idcorrespondsto the id in the sidebar.json2. Yes, all images > > > > are > > > > inthe staticfolder 3. Well, they are not really auto-auto > > > > generated😉 There is amaven plugin which extracts the markdown > > > > files ande.g., createsentries for the sidebar.json and modifies > > > > the imagelinks so thatthey work on the website, but it's not > > > > somethingautomated in Githubactions or so, although it would be > > > > awesome tosupport this4. Youdon't need to manually add the > > > > version, this ismanaged byDocusaurus. The markdown files in the > > > > documentationfolder alwaysrefer to the next unreleased version. > > > > When you startthe website, youcan click on the version (0.69.0 > > > > should be thedefault) and thenselect "master" which points you > > > > to > > > > the unreleasedversion. Aftereach release, we run the "ds- > > > > version" > > > > command andthen Docusauruschecks all files in the documentation > > > > folder againstthe files fromthe previous release and creates a > > > > new docs version.E.g., see olderdocumentation files at [1]. In > > > > case you want tobackport changes tothe documentation pages to > > > > previous versions,these files can also bemodified, but that's > > > > rather rare.5. Goodpoint, I _think_ this shouldbe possible > > > > without any problems, atleast it's worth a try 😉Hope this > > > > helps!CheersDominik[1] > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website/versioned_docs > > > > > > > > > > > > > > > > -----Original Message-----From: Florian Micklich < > > > > [email protected] > > > > > > > > > Sent: Wednesday, August 24, 2022 7:59 PMTo: > > > > > > > > [email protected] > > > > > > > > Subject: Re: [DISCUSS] Documentation - Description for > > > > AdaptersandProcessing ElementsHi,ok, I started docusaurus from > > > > sourceDominik has previouslymentioned and did a build & run > > > > website likementioned in theReadme.Next time I am even faster, > > > > because I willstart the "BuildDocumentation" from the > > > > documentation/websitefolder, as it itclearly mentioned in the > > > > Readme :-D Before I start,I am trying tounderstand the > > > > structure > > > > of this documentation and Ihave somequestions.1.So everything > > > > is > > > > organised by thesidebar.jsonSo the try-installation [1] is > > > > linked > > > > to the 01_try-installation.md file[2]. How is this working if > > > > the names doesnot match?2. I see that all images are stored in > > > > the static websitepath here[3]. Is this correct?3. Do I > > > > understand it correctly thatall filesin the PE folder [4] are > > > > auto generated and will beautogenerated next time again? So > > > > if > > > > I change something here,will thisbe overwritten the next > > > > time?4.Is it OK if I add the 0.70version inthe version file > > > > here > > > > [5]?5. Is it OK to structure thedocuments insome more sup- > > > > folders? We could discuss the sub-foldernames here ofcourse but > > > > I > > > > think that would make the documentationlittle biteasier to > > > > navigate for the authors. This includes alsopictures instep > > > > 2.Greetings Florian[1] > > > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/website/sidebars.json#L5[2] > > > > > > > > > > > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/docs/01_try-installation.md[3] > > > > > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website/static/img[4] > > > > > > > > https://github.com > > > > > > > > /apache/incubator-streampipes- > > > > website/tree/dev/documentation/docs/pe[5] > > > > https://github.com/apache/inc > > > > ubator-streampipes- > > > > website/blob/dev/documentation/website/versions.json > > > > Am Mittwoch, dem 24.08.2022 um 12:32 +0000 schrieb Philipp > > > > Zehnder: > > > > > Hi together,it would be great to have more content > > > > > inthedocumentation.I am alsoin favor of using docusaurus. > > > > > Itusesmarkdown, which leaves us thepossibility to move to > > > > > anothertoolingat some point in the future ifwe want.@Florian > > > > > pleasewrite if yousee something that should be updated > > > > > orchanged in > > > > > thecurrentversion.Cheers,Philipp_____________________________ > > > > > ___Von:DominikRiemer < > > > > > [email protected] > > > > > >Gesendet: Mittwoch,August > > > > > 24,202211:47 AMAn: > > > > > [email protected] > > > > > > > > > > < > > > > > [email protected] > > > > > >Betreff: RE: [DISCUSS] > > > > > Documentation-Description for Adapters andProcessing > > > > > ElementsHi > > > > > Florian,Cool!Thedocumentation on the website is based on > > > > > Docusauruswhichsupportsthings like docs versioning. > > > > > Docusaurus > > > > > usesmarkdown files,so werely on markdown for the > > > > > documentation.Thereis a 'website' folderat [1] where you can > > > > > find the npm scriptstostart docusaurus and a'docs' folder > > > > > which > > > > > includesthedocumentation markdown files forthe next release > > > > > version(selectg"next" in the docs website). Aftera new > > > > > release, > > > > > we run a"npm ds-version" which then detects changesin the > > > > > markdown filesand createsa new release docs > > > > > version.Thedocuments in the "docs"folder can be freely > > > > > modified > > > > > andadded.That's a good place wherewe can add new tutorials. > > > > > The > > > > > "pe"foldercontains thedocumentation of pipeline elements. > > > > > These > > > > > arecurrentlygeneratedfrom the StreamPipes source code using > > > > > thestreampipes-maven-plugin (as some rewriting is needed, > > > > > e.g., > > > > > to fixthe pathsofimages). This process is far from perfect > > > > > and > > > > > needssome manualwork.It would be nice to have some Github > > > > > actionsscriptwhichautomatically updates the Documentation, > > > > > but > > > > > that's abitmoreadvanced task I'd say 😉If you want to make > > > > > changes tothestructure, this can be done in > > > > > the"sidebars.json" > > > > > file [3],whichis also auto-versioned by Docusaurus.[1] > > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website > > > > > > > > > > [2] > > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/docs > > > > > > > > > > [3] > > > > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/website/sidebars.json > > > > > > > > > > > > > > > -----Original Message-----From: Florian Micklich < > > > > > [email protected] > > > > > > > > > > > Sent: Wednesday, August 24, 2022 10:13 AMTo: > > > > > > > > > > [email protected] > > > > > > > > > > Subject: Re: [DISCUSS] Documentation - Description > > > > > forAdaptersandProcessing ElementsHello,I would also prefer a > > > > > shortIn-Appdocumentation and largerdocumentation "outside" > > > > > .Personally, Iprefer to read pdf documentsso I can take > > > > > additionalnotes. ButHTML is also good for a quickonline > > > > > search.We must keepin mindwhat kind of documentation we > > > > > wantto > > > > > create and should notmixthem. Also, cross-references between > > > > > theindividual"chapters"should be avoided. This makes it > > > > > easier > > > > > ifsomethingshouldchange. 1. Quick starter > > > > > documentation. 2.SetupDocumentation 3.Technical > > > > > Documentation 4. Usermanualdocumentation with bestpractiseI > > > > > think we should hit theroad andtake a look along the way :) > > > > > Iknow the in appdocumentation is inmarkdown so far?I > > > > > wouldrecommend to write thedocumentation inasciidoc [1] [2] > > > > > This couldbe set up withasciidoctorj [3]. Thismakes it easy > > > > > to > > > > > includeeverything in Gitand also keep thedocumentation > > > > > alongside thesource code. So ifsomething isrewritten, the > > > > > developer can alsochange thedocumentation. Makesnew > > > > > screenshots > > > > > and deletes oldones.3] Thecreation of a PDF orHTML file can > > > > > be > > > > > done withGithubAction. With every commitor with every > > > > > release. Theresults can bepushed to the StreamPipeswebsite, I > > > > > guess?What doyouthink?Florian[1] > > > > > https://docs.asciidoctor.org/asciidoc/latest/[2] > > > > > > > > > > German Source > > > > > https://blog.ordix.de/docs-as-code-dokumentation-mit-asciidoctor[3] > > > > > > > > > > htt > > > > > ps://github.com/asciidoctor/asciidoctorjAmMittwoch,dem24.08.2 > > > > > 02 > > > > > 2 um 07:07 +0000 schrieb Dominik Riemer: > > > > > > Hi,yes, we can do both: In-app documentation > > > > > > whichonlycoverstheconfiguration and a short description of > > > > > > theadapterandpipelineelement and additional step-by- > > > > > > stepguides/tutorialswithnicescreenshots that are available > > > > > > in > > > > > > theonlinedocumentation. Wecanhave such step-by-step guides > > > > > > for themostfrequently used adapters.@Florian, do you want > > > > > > to > > > > > > work onsuch aguide on thewebsite? I knowyou had some really > > > > > > gooddocumentationideas somewhile ago 😉CheersDominik----- > > > > > > OriginalMessage-----From: Philipp Zehnder < > > > > > > [email protected] > > > > > > > Sent:Sunday, August > > > > > > 21,20221:07PMTo: > > > > > > [email protected] > > > > > > > > > > > > Subject: [DISCUSS] Documentation - > > > > > > DescriptionforAdaptersandProcessing Elements Hi all, here > > > > > > is > > > > > > a new threadtodiscusstheadapter documentation.Here is a > > > > > > link > > > > > > to thepreviousdiscussionin therelease thread [1].@Dominik I > > > > > > know thattheregistration isdifferent, but since alladapters > > > > > > already havethethree files(documentation.md, icon.png, > > > > > > andstring.en),shouldn’tit bepossible to also display the > > > > > > descriptionin theUI.How are wedoingcurrently for the > > > > > > icons?Because they areloaded dynamicallyaswell. I think the > > > > > > maindifference is thatthe icons areloadeddirectly from the > > > > > > workercontainer and notfrom thebackend.Couldn’t we do it > > > > > > similarly forthedocumentation file?(Pleasecorrect me if I > > > > > > am > > > > > > wrong)Regardingthe description on theWebsite.I think it > > > > > > would > > > > > > begreat if we couldgenerate this basedon themarkdown files > > > > > > foreach adapter.The documentation.md filesmostlydescribe > > > > > > theconfigurationparameters of adapters andprocessors. > > > > > > Ithink > > > > > > itwould be great tohave additional exampleshow to use > > > > > > them.(e.g.describe how a CSVfile is uploaded or how aPLC > > > > > > isconnected).Should we include suchexamples > > > > > > intothedocumentation.md file orshould there be > > > > > > anotherlocation forthis?Any thoughts onthat?@Florian you > > > > > > wrote that you would liketo workon thedocumentation.If you > > > > > > need anything to start with,please > > > > > > letusknow.Cheers,Philipp[1] > > > > > > https://lists.apache.org/thread/29w586db1btqo0ps1rgo3fscvt3tvrzg > > > > > >
