Hi all,
I made a rouge structure about a possible documentation.
• Introduction: ◦ Overview ◦ Basic Elements (former
StreamPipes Concept) ◦ Time in StreamPipes (Why timestamp is
important in the data?) ◦ Semantic in StreamPipes ◦ Other
Topics? ◦ Installation ▪ Basic
Installation ▪ Docker Deployment ▪ K8s /
K3s ▪ Use SSL ▪ Security ◦ Getting
Help • Overview (Basic GUI Explanation and Introduction for each GUI
point and why it is implemented (short description text) for example
Dashboard why Dashboard, what you can do. Basically little more deep
diver than Basic Elements section. Section with Most of the
Screenshots, So easy to maintain due changes) ◦ Login Page and
Basics ◦ Welcome Page ◦ Pipelines ◦
Connect ◦ Dashboard ◦ Data Explorer ◦ Apps
(deprecated?) ◦ Install Pipeline Elements ◦ Asset
Management ◦ File Management ◦ Configuration & User
Prefences ◦ Notification • Data Source Elements
(Adapter) ◦ Why Adapter. Connect to the Real World ◦ What
Kind of Sources (Data Set vs Data Stream / Categories) ◦ Basic
Adapter Setup in 4 Steps (at the moment) ▪ Step 1
Settings ▪ Step 2 Select Format • JSON
with Description for Array Field / GeoJSON / Single Object / Array with
Short Data Examples • XML • CSV
• Image ▪ Step 3 configure
fields • Different between Measurement, Dimension
Header • Add Nested / Add static Value / Add
Timestamp • Edit fields (Big Thinks.
Also) • Preview ▪ Step4 start
adapters • Settings / Description (why Important
and where can I find this info later on?) •
Remove Duplicates / Reduce event rates /Persist events ◦ Connect
Adapters Overview ▪ Apache Kafka with Example
Sources ▪ Apache Pulsar ▪ File
Set ▪ File Stream ▪ ... ◦ Connect
Adapters IIOT Overview ▪ ... ◦ How To
Adapter ▪ Setup Adapter Example 1 ▪ Setup
HTTPS Adapter ◦ Technical Concept of Adapters (Dev Part???) •
Data Processors • Data Sinks • Create Pipeline •
Dashboard • Assets
The working example will be in the section HowTo and some example
resources in each adapter description.What do you think?
First step would be to check if it possible to move the files so that
the structure is not destroyed and the simply start with the adapters
:)I made already a rebase between dev and STREAMPIPES-581 branch, due
some changes in the dev branch.
Florian
Am Montag, dem 29.08.2022 um 01:07 +0000 schrieb Philipp Zehnder:
> Hi Florian,
> I am not sure if we will find many available public data sources,
> especially for PLCs. But maybe we find some docker containers that
> can be used for the tutorial. If we do not find some, maybe we can
> provide them in our registry.We could also use them for the automated
> tests. That would be great, because currently those tests are not
> integrated in the nightly run.
> I think for most users it is already helpful to have a working
> example for the different adapters, so they do not necessarily have
> to try out the examples.
> As you said we should collect all the information somewhere. Should
> we use thwiki for that?
> Cheers,Philipp________________________________Von: Florian Micklich <
> [email protected]>Gesendet: Samstag, August 27, 2022 1:20 AMAn:
> [email protected] <[email protected]>Betreff: Re:
> [DISCUSS] Documentation - Description for Adapters and Processing
> Elements
> Hi,perfekt, I will create this branch. I will think about a
> possiblestructure over the weekend. I let you know.How are you
> testing and using the adapters and wich one are you usingmost? Do you
> have some URL sources you always for developing processor testing
> and what is the typical workflow to add them?Maybe we can start to
> collect them in some way so I can add them later in
> thedocumentation.GreetingsFlorian
> Am Donnerstag, dem 25.08.2022 um 19:38 +0000 schrieb Dominik Riemer:
> > Sounds good!We usually use only the issue name as branch
> > name(STREAMPIPES-581) without the "feature" prefix, but both is
> > totallyfine!Dominik
> > -----Original Message-----From: Florian Micklich <
> > [email protected]
> > > Sent: Wednesday, August 24, 2022 9:44 PMTo:
> > [email protected]
> > Subject: Re: [DISCUSS] Documentation - Description for Adapters
> > andProcessing ElementsHi Dominik,thanks for the answer and before I
> > ask more questions,I'll simply start :-)Therefore I created a
> > tickethttps://issues.apache.org/jira/browse/STREAMPIPES-581
> > I will also create a feature branch feature/STREAMPIPES-581 or
> > whatis here best practise naming convention?GreetingFlorian
> >
> > Am Mittwoch, dem 24.08.2022 um 19:07 +0000 schrieb Dominik Riemer:
> > > Hi Florian,happy to help! 1. There is a header in each
> > > markdownfilewith an ID (e.g. in the 01_try-installation.md). This
> > > idcorrespondsto the id in the sidebar.json2. Yes, all images are
> > > inthe staticfolder 3. Well, they are not really auto-auto
> > > generated😉 There is amaven plugin which extracts the markdown
> > > files ande.g., createsentries for the sidebar.json and modifies
> > > the imagelinks so thatthey work on the website, but it's not
> > > somethingautomated in Githubactions or so, although it would be
> > > awesome tosupport this4. Youdon't need to manually add the
> > > version, this ismanaged byDocusaurus. The markdown files in the
> > > documentationfolder alwaysrefer to the next unreleased version.
> > > When you startthe website, youcan click on the version (0.69.0
> > > should be thedefault) and thenselect "master" which points you to
> > > the unreleasedversion. Aftereach release, we run the "ds-version"
> > > command andthen Docusauruschecks all files in the documentation
> > > folder againstthe files fromthe previous release and creates a
> > > new docs version.E.g., see olderdocumentation files at [1]. In
> > > case you want tobackport changes tothe documentation pages to
> > > previous versions,these files can also bemodified, but that's
> > > rather rare.5. Goodpoint, I _think_ this shouldbe possible
> > > without any problems, atleast it's worth a try 😉Hope this
> > > helps!CheersDominik[1]
> > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website/versioned_docs
> > >
> > >
> > > -----Original Message-----From: Florian Micklich <
> > > [email protected]
> > > > Sent: Wednesday, August 24, 2022 7:59 PMTo:
> > > [email protected]
> > > Subject: Re: [DISCUSS] Documentation - Description for
> > > AdaptersandProcessing ElementsHi,ok, I started docusaurus from
> > > sourceDominik has previouslymentioned and did a build & run
> > > website likementioned in theReadme.Next time I am even faster,
> > > because I willstart the "BuildDocumentation" from the
> > > documentation/websitefolder, as it itclearly mentioned in the
> > > Readme :-D Before I start,I am trying tounderstand the structure
> > > of this documentation and Ihave somequestions.1.So everything is
> > > organised by thesidebar.jsonSo the try-installation [1] is linked
> > > to the 01_try-installation.md file[2]. How is this working if
> > > the names doesnot match?2. I see that all images are stored in
> > > the static websitepath here[3]. Is this correct?3. Do I
> > > understand it correctly thatall filesin the PE folder [4] are
> > > auto generated and will beautogenerated next time again? So if
> > > I change something here,will thisbe overwritten the next
> > > time?4.Is it OK if I add the 0.70version inthe version file here
> > > [5]?5. Is it OK to structure thedocuments insome more sup-
> > > folders? We could discuss the sub-foldernames here ofcourse but I
> > > think that would make the documentationlittle biteasier to
> > > navigate for the authors. This includes alsopictures instep
> > > 2.Greetings Florian[1]
> > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/website/sidebars.json#L5[2]
> > >
> > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/docs/01_try-installation.md[3]
> > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website/static/img[4]
> > > https://github.com
> > > /apache/incubator-streampipes-
> > > website/tree/dev/documentation/docs/pe[5]
> > > https://github.com/apache/inc ubator-streampipes-
> > > website/blob/dev/documentation/website/versions.json
> > > Am Mittwoch, dem 24.08.2022 um 12:32 +0000 schrieb Philipp
> > > Zehnder:
> > > > Hi together,it would be great to have more content
> > > > inthedocumentation.I am alsoin favor of using docusaurus.
> > > > Itusesmarkdown, which leaves us thepossibility to move to
> > > > anothertoolingat some point in the future ifwe want.@Florian
> > > > pleasewrite if yousee something that should be updated orchanged in
> > > > thecurrentversion.Cheers,Philipp________________________________Von:DominikRiemer
> > > > <[email protected]>Gesendet: Mittwoch,August
> > > > 24,202211:47 AMAn: [email protected]
> > > > <[email protected]>Betreff: RE: [DISCUSS]
> > > > Documentation-Description for Adapters andProcessing ElementsHi
> > > > Florian,Cool!Thedocumentation on the website is based on
> > > > Docusauruswhichsupportsthings like docs versioning. Docusaurus
> > > > usesmarkdown files,so werely on markdown for the
> > > > documentation.Thereis a 'website' folderat [1] where you can
> > > > find the npm scriptstostart docusaurus and a'docs' folder which
> > > > includesthedocumentation markdown files forthe next release
> > > > version(selectg"next" in the docs website). Aftera new release,
> > > > we run a"npm ds-version" which then detects changesin the
> > > > markdown filesand createsa new release docs
> > > > version.Thedocuments in the "docs"folder can be freely modified
> > > > andadded.That's a good place wherewe can add new tutorials. The
> > > > "pe"foldercontains thedocumentation of pipeline elements. These
> > > > arecurrentlygeneratedfrom the StreamPipes source code using
> > > > thestreampipes-maven-plugin (as some rewriting is needed, e.g.,
> > > > to fixthe pathsofimages). This process is far from perfect and
> > > > needssome manualwork.It would be nice to have some Github
> > > > actionsscriptwhichautomatically updates the Documentation, but
> > > > that's abitmoreadvanced task I'd say 😉If you want to make
> > > > changes tothestructure, this can be done in the"sidebars.json"
> > > > file [3],whichis also auto-versioned by Docusaurus.[1]
> > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website
> > > > [2]
> > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/docs
> > > > [3]
> > > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/website/sidebars.json
> > > >
> > > > -----Original Message-----From: Florian Micklich <
> > > > [email protected]
> > > > > Sent: Wednesday, August 24, 2022 10:13 AMTo:
> > > > [email protected]
> > > > Subject: Re: [DISCUSS] Documentation - Description
> > > > forAdaptersandProcessing ElementsHello,I would also prefer a
> > > > shortIn-Appdocumentation and largerdocumentation "outside"
> > > > .Personally, Iprefer to read pdf documentsso I can take
> > > > additionalnotes. ButHTML is also good for a quickonline
> > > > search.We must keepin mindwhat kind of documentation we wantto
> > > > create and should notmixthem. Also, cross-references between
> > > > theindividual"chapters"should be avoided. This makes it easier
> > > > ifsomethingshouldchange. 1. Quick starter
> > > > documentation. 2.SetupDocumentation 3.Technical
> > > > Documentation 4. Usermanualdocumentation with bestpractiseI
> > > > think we should hit theroad andtake a look along the way :)
> > > > Iknow the in appdocumentation is inmarkdown so far?I
> > > > wouldrecommend to write thedocumentation inasciidoc [1] [2]
> > > > This couldbe set up withasciidoctorj [3]. Thismakes it easy to
> > > > includeeverything in Gitand also keep thedocumentation
> > > > alongside thesource code. So ifsomething isrewritten, the
> > > > developer can alsochange thedocumentation. Makesnew screenshots
> > > > and deletes oldones.3] Thecreation of a PDF orHTML file can be
> > > > done withGithubAction. With every commitor with every
> > > > release. Theresults can bepushed to the StreamPipeswebsite, I
> > > > guess?What doyouthink?Florian[1]
> > > > https://docs.asciidoctor.org/asciidoc/latest/[2]
> > > > German Source
> > > > https://blog.ordix.de/docs-as-code-dokumentation-mit-asciidoctor[3]
> > > > htt
> > > > ps://github.com/asciidoctor/asciidoctorjAmMittwoch,dem24.08.202
> > > > 2 um 07:07 +0000 schrieb Dominik Riemer:
> > > > > Hi,yes, we can do both: In-app documentation
> > > > > whichonlycoverstheconfiguration and a short description of
> > > > > theadapterandpipelineelement and additional step-by-
> > > > > stepguides/tutorialswithnicescreenshots that are available in
> > > > > theonlinedocumentation. Wecanhave such step-by-step guides
> > > > > for themostfrequently used adapters.@Florian, do you want to
> > > > > work onsuch aguide on thewebsite? I knowyou had some really
> > > > > gooddocumentationideas somewhile ago 😉CheersDominik-----
> > > > > OriginalMessage-----From: Philipp Zehnder <
> > > > > [email protected]> Sent:Sunday, August
> > > > > 21,20221:07PMTo: [email protected]
> > > > > Subject: [DISCUSS] Documentation -
> > > > > DescriptionforAdaptersandProcessing Elements Hi all, here is
> > > > > a new threadtodiscusstheadapter documentation.Here is a link
> > > > > to thepreviousdiscussionin therelease thread [1].@Dominik I
> > > > > know thattheregistration isdifferent, but since alladapters
> > > > > already havethethree files(documentation.md, icon.png,
> > > > > andstring.en),shouldn’tit bepossible to also display the
> > > > > descriptionin theUI.How are wedoingcurrently for the
> > > > > icons?Because they areloaded dynamicallyaswell. I think the
> > > > > maindifference is thatthe icons areloadeddirectly from the
> > > > > workercontainer and notfrom thebackend.Couldn’t we do it
> > > > > similarly forthedocumentation file?(Pleasecorrect me if I am
> > > > > wrong)Regardingthe description on theWebsite.I think it would
> > > > > begreat if we couldgenerate this basedon themarkdown files
> > > > > foreach adapter.The documentation.md filesmostlydescribe
> > > > > theconfigurationparameters of adapters andprocessors. Ithink
> > > > > itwould be great tohave additional exampleshow to use
> > > > > them.(e.g.describe how a CSVfile is uploaded or how aPLC
> > > > > isconnected).Should we include suchexamples
> > > > > intothedocumentation.md file orshould there be
> > > > > anotherlocation forthis?Any thoughts onthat?@Florian you
> > > > > wrote that you would liketo workon thedocumentation.If you
> > > > > need anything to start with,please
> > > > > letusknow.Cheers,Philipp[1]
> > > > > https://lists.apache.org/thread/29w586db1btqo0ps1rgo3fscvt3tvrzg
