Re: AW: [DISCUSS] Graduate StreamPipes as TLP?

[Johannes Tex]

Hi Dominik,

i'm not very deep in the project the last months, but i'm looking in from time 
to time and i'm happy about how everything is developing and about the traffic 
on the mail lists :) 

>From my point of view, StreamPiples is ready to become a TLP. 


Best regards
Johannes

On 2022/10/25 15:53:53 Marco Heyden wrote:
> Hi Dominik,
> 
> I am not deeply involved in the project at the moment but I agree with your 
> points.
> Therefore I think that we should go for graduation to a TLP.
> 
> Best
> Marco
> 
> -Ursprüngliche Nachricht-
> Von: Philipp Zehnder  
> Gesendet: Tuesday, 25 October, 2022 10:39
> An: dev@streampipes.apache.org
> Betreff: AW: [DISCUSS] Graduate StreamPipes as TLP?
> 
> Hi Dominik,
> 
> thanks for opening the discussion.
> 
> I really enjoy working with the StreamPipes community and am excited about 
> the development of the project.
> I also think we are ready for graduation and look forward to the next steps 
> and the continued growth of the community.
> 
> Cheers,
> Philipp
> 
> Von: Dominik Riemer 
> Datum: Montag, 24. Oktober 2022 um 18:18
> An: dev@streampipes.apache.org 
> Betreff: [DISCUSS] Graduate StreamPipes as TLP?
> Hi,
> 
> since StreamPipes joined the Apache Incubator in November 2019, a lot has 
> happened:
> 
>   *   We’ve had five releases in the incubator so far, with three different 
> release managers
>   *   We’ve had steady community growth with many new committers & PPMC 
> members
>   *   We have constant traffic on the mailing lists and I think we have 
> really adopted the Apache way for development
>   *   StreamPipes has now over 9.000 commits, has evolved in terms of quality 
> and feature-richness and we had many talks at ApacheCons and many other 
> events to increase attention for our tool
>   *   Our self-assessment of the maturity model [1] looks good and all boxes 
> are ticked 
> 
> So from my personal view, we are quite ready for graduation to a TLP!
> 
> The graduation process foresees that we as a community discuss graduation 
> readiness and have a community vote. Afterwards, we bring this to the 
> Incubator list where another vote happens. Finally, the ASF board makes the 
> final decision on graduation to a top-level project. The whole process will 
> probably take several weeks.
> 
> But first – what do you think? Should we go for graduation?
> 
> Cheers
> Dominik
> 
> [1] 
> https://cwiki.apache.org/confluence/display/STREAMPIPES/StreamPipes+Maturity+Checklist
> 
> 

Re: Re: [DISCUSS] Element Monitoring

[Johannes Tex]

Hi Philipp,

we should minimally store the protocol message itself, the status, the date and 
the source. In the data explorer we should be able to sort by all these values.

I think that for production environments it is necessary to monitor the whole 
installation.

Regards
Johannes

On 2021/10/26 14:17:09 Philipp Zehnder wrote:
> Hi Johannes,
> 
> I like the idea of predefined pipelines that are directly started when 
> installing the system. I recently re-activated the pipeline templates feature 
> to directly store data when creating an adapter. We can use this mechanism 
> for that as well.
> I think it would also be great to store the log data, my suggestion would be 
> to use the data explorer instead of the live dashboard.
> How would you store the data, or what exactly do we need to log and how do we 
> need to represent it? Do you have any thoughts on that?
> 
> I'm not quite sure yet if we should log the entire installation or just the 
> individual elements.
> 
> Philipp
> 
> > On 23. Oct 2021, at 13:34, Johannes Tex  wrote:
> > 
> > Hi Philipp,
> > 
> > I really like your idea in [1] - it's the StreamPipes way :) Everything we 
> > can do with pipelines, weshould do with pipelines! Is it possible to 
> > predefine pipelines that are already running after installation? IMHO a 
> > predefined dashboard would also be necessary.
> > 
> > But...
> > Do you want to monitor only the elements this way or the whole 
> > installation? If you want to monitor all services this way, the 
> > disadvantage is that the UI, pipelines and backend have to be running. If 
> > you use the ELK stack (or an other monitor stack), you can still brows the 
> > logs even if the UI, backend or/and pe's are not running.
> > 
> > Johannes
> > 
> > On 2021/10/23 08:05:26, Philipp Zehnder  wrote: 
> >> Hi Johannes,
> >> 
> >> what do you think should we rather use the ELK stack to collect the logs 
> >> or use the logging mechanism in [1] and transmit the logs over our message 
> >> broker that we use internally.
> >> The advantage  would be that the message broker is already there, so we 
> >> need no additional services.
> >> 
> >> @all Or are there any other ideas how we can collect the logs / errors of 
> >> the individual services including the information of which StreamPipes 
> >> element (adapter / processor / …) was responsible.
> >> 
> >> Philipp
> >> 
> >>> On 21. Oct 2021, at 16:50, Johannes Tex  wrote:
> >>> 
> >>> Hi all,
> >>> 
> >>> some time ago I implemented the EventStatisticLogger [1], which is 
> >>> currently not used. The PE wrappers call this logger for each incoming 
> >>> event. The logs were delivered to elasticsearch/kibana via filebeat.
> >>> 
> >>> Regards
> >>> Johannes
> >>> 
> >>> [1] 
> >>> https://github.com/apache/incubator-streampipes/blob/dev/streampipes-logging/src/main/java/org/apache/streampipes/logging/impl/EventStatisticLogger.java
> >>> 
> >>> 
> >>> 
> >>> On 2021/10/20 17:42:04, Dominik Riemer  wrote: 
> >>>> Hi Philipp,
> >>>> 
> >>>> I think we can either catch exceptions somewhere in the upper parts of 
> >>>> our processing chain and send them to a broker topic, where we collect 
> >>>> exceptions similar to the monitoring feature.
> >>>> We also had the concept of a StreamPipes-specific logging system, which 
> >>>> we can also bring to work so that user-defined logging is available to 
> >>>> the core and user UI. I'm not fully aware of the current status of this 
> >>>> logger.
> >>>> 
> >>>> Dominik
> >>>> 
> >>>> On 2021/10/20 09:48:01, Philipp Zehnder  wrote: 
> >>>>> Hi Domink,
> >>>>> 
> >>>>> yes, I think that would also be a good option to transmit the events.
> >>>>> I was wondering, how can we intercept the exceptions that occur within 
> >>>>> a service?
> >>>>> 
> >>>>> Philipp
> >>>>> 
> >>>>>> On 19. Oct 2021, at 22:32, Dominik Riemer  wrote:
> >>>>>> 
> >>>>>> Hi Philipp,
> >>>>>> 
> >>>>>> looks good! I wonder if we should also think about monitoring errors 
> >>>>>> in the adapter and pipeline elements and ho

Re: [DISCUSS] Element Monitoring

[Johannes Tex]

Hi Philipp,

I really like your idea in [1] - it's the StreamPipes way :) Everything we can 
do with pipelines, weshould do with pipelines! Is it possible to predefine 
pipelines that are already running after installation? IMHO a predefined 
dashboard would also be necessary.

But...
Do you want to monitor only the elements this way or the whole installation? If 
you want to monitor all services this way, the disadvantage is that the UI, 
pipelines and backend have to be running. If you use the ELK stack (or an other 
monitor stack), you can still brows the logs even if the UI, backend or/and 
pe's are not running.

Johannes

On 2021/10/23 08:05:26, Philipp Zehnder  wrote: 
> Hi Johannes,
> 
> what do you think should we rather use the ELK stack to collect the logs or 
> use the logging mechanism in [1] and transmit the logs over our message 
> broker that we use internally.
> The advantage  would be that the message broker is already there, so we need 
> no additional services.
> 
> @all Or are there any other ideas how we can collect the logs / errors of the 
> individual services including the information of which StreamPipes element 
> (adapter / processor / …) was responsible.
> 
> Philipp
> 
> > On 21. Oct 2021, at 16:50, Johannes Tex  wrote:
> > 
> > Hi all,
> > 
> > some time ago I implemented the EventStatisticLogger [1], which is 
> > currently not used. The PE wrappers call this logger for each incoming 
> > event. The logs were delivered to elasticsearch/kibana via filebeat.
> > 
> > Regards
> > Johannes
> > 
> > [1] 
> > https://github.com/apache/incubator-streampipes/blob/dev/streampipes-logging/src/main/java/org/apache/streampipes/logging/impl/EventStatisticLogger.java
> > 
> > 
> > 
> > On 2021/10/20 17:42:04, Dominik Riemer  wrote: 
> >> Hi Philipp,
> >> 
> >> I think we can either catch exceptions somewhere in the upper parts of our 
> >> processing chain and send them to a broker topic, where we collect 
> >> exceptions similar to the monitoring feature.
> >> We also had the concept of a StreamPipes-specific logging system, which we 
> >> can also bring to work so that user-defined logging is available to the 
> >> core and user UI. I'm not fully aware of the current status of this logger.
> >> 
> >> Dominik
> >> 
> >> On 2021/10/20 09:48:01, Philipp Zehnder  wrote: 
> >>> Hi Domink,
> >>> 
> >>> yes, I think that would also be a good option to transmit the events.
> >>> I was wondering, how can we intercept the exceptions that occur within a 
> >>> service?
> >>> 
> >>> Philipp
> >>> 
> >>>> On 19. Oct 2021, at 22:32, Dominik Riemer  wrote:
> >>>> 
> >>>> Hi Philipp,
> >>>> 
> >>>> looks good! I wonder if we should also think about monitoring errors in 
> >>>> the adapter and pipeline elements and how to submit the stack trace of 
> >>>> exceptions (e.g., an adapter or processing element which throws a 
> >>>> runtime error). 
> >>>> 
> >>>> Does it make sense to include an array containing exceptions into the 
> >>>> message payload?
> >>>> 
> >>>> Dominik
> >>>> 
> >>>> On 2021/10/19 14:24:11, Philipp Zehnder  wrote: 
> >>>>> Hi all,
> >>>>> 
> >>>>> I’d like to discuss a feature to monitore our pipeline elements, like 
> >>>>> adapters, processing elements and sinks.
> >>>>> I created a wiki page with a description of the idea [1].
> >>>>> The main idea is to detect if an element is failing (e.g. an adapter 
> >>>>> stops sending events) and notify the admin of the system.
> >>>>> 
> >>>>> I’ll create a prototype according to the description in the wiki. If 
> >>>>> you have any ideas or feature request please write them so that we can 
> >>>>> discuss them.
> >>>>> I am looking forward to your feedback on this feature.
> >>>>> 
> >>>>> Philipp
> >>>>> 
> >>>>> 
> >>>>> [1] 
> >>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335025
> >>>>>  
> >>>>> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335025>
> >>>>> 
> >>>>> 
> >>> 
> >>> 
> >> 
> 
> 

Re: [DISCUSS] Element Monitoring

[Johannes Tex]

Hi all,

some time ago I implemented the EventStatisticLogger [1], which is currently 
not used. The PE wrappers call this logger for each incoming event. The logs 
were delivered to elasticsearch/kibana via filebeat.

Regards
Johannes

[1] 
https://github.com/apache/incubator-streampipes/blob/dev/streampipes-logging/src/main/java/org/apache/streampipes/logging/impl/EventStatisticLogger.java



On 2021/10/20 17:42:04, Dominik Riemer  wrote: 
> Hi Philipp,
> 
> I think we can either catch exceptions somewhere in the upper parts of our 
> processing chain and send them to a broker topic, where we collect exceptions 
> similar to the monitoring feature.
> We also had the concept of a StreamPipes-specific logging system, which we 
> can also bring to work so that user-defined logging is available to the core 
> and user UI. I'm not fully aware of the current status of this logger.
> 
> Dominik
> 
> On 2021/10/20 09:48:01, Philipp Zehnder  wrote: 
> > Hi Domink,
> > 
> > yes, I think that would also be a good option to transmit the events.
> > I was wondering, how can we intercept the exceptions that occur within a 
> > service?
> > 
> > Philipp
> > 
> > > On 19. Oct 2021, at 22:32, Dominik Riemer  wrote:
> > > 
> > > Hi Philipp,
> > > 
> > > looks good! I wonder if we should also think about monitoring errors in 
> > > the adapter and pipeline elements and how to submit the stack trace of 
> > > exceptions (e.g., an adapter or processing element which throws a runtime 
> > > error). 
> > > 
> > > Does it make sense to include an array containing exceptions into the 
> > > message payload?
> > > 
> > > Dominik
> > > 
> > > On 2021/10/19 14:24:11, Philipp Zehnder  wrote: 
> > >> Hi all,
> > >> 
> > >> I’d like to discuss a feature to monitore our pipeline elements, like 
> > >> adapters, processing elements and sinks.
> > >> I created a wiki page with a description of the idea [1].
> > >> The main idea is to detect if an element is failing (e.g. an adapter 
> > >> stops sending events) and notify the admin of the system.
> > >> 
> > >> I’ll create a prototype according to the description in the wiki. If you 
> > >> have any ideas or feature request please write them so that we can 
> > >> discuss them.
> > >> I am looking forward to your feedback on this feature.
> > >> 
> > >> Philipp
> > >> 
> > >> 
> > >> [1] 
> > >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335025
> > >>  
> > >> 
> > >> 
> > >> 
> > 
> > 
> 

Re: Harmonization of Data Lake Query Result

[Johannes Tex]

Hi Philipp,

that's fine for me. Git never forgets ;) 

I recommend rebasing the branch with dev before merging with dev. Then you 
should be able to merge your branch directly with dev without merge conflicts.

Johannes

On 2021/08/12 16:42:15, Philipp Zehnder  wrote: 
> Hi Johannes,
> 
> yes you are right, this is used in [1].
> Currently we mostly work in branch [2] and there class was removed because it 
> was not referenced anymore. 
> If we still need it, we can recreate the class. What do you think?
> 
> In branch [2] we applied many changes, especially Dominik did a lot of 
> refactoring.
> We have tested the code in the branch, but it is possible that there are 
> still some problems. 
> Therefore, we would like to merge it into dev as soon as possible to test 
> everything. 
> 
> Are there any reasons why we should wait with the merge or can fix the 
> resulting bugs directly in dev?
> 
> Philipp
> 
> [2] https://github.com/apache/incubator-streampipes/tree/STREAMPIPES-319
> 
> 
> > On 10. Aug 2021, at 20:08, Johannes Tex  wrote:
> > 
> > Hi Daniel,
> > 
> > the attributes "labels" and "pageSum", which you call obsolete in your 
> > proposals , are used in [1]. Maybe that is also obsolete ;)
> > 
> > Johnnaes
> > 
> > [1] 
> > https://github.com/apache/incubator-streampipes/blob/master/ui/src/app/core-ui/image/image-categorize/image-categorize.component.ts
> > 
> > On 2021/08/10 11:20:58, Daniel Ebi  wrote: 
> >> Hi all,
> >> 
> >> As part of the refactoring and integration of the Data Lake REST API into 
> >> the Data Explorer, it has turned out that it makes sense to harmonize the 
> >> return type of Data Lake queries first. Consequently, I have already 
> >> drafted two proposals for the alignment of DataResult, GroupedDataResult 
> >> and PageResult classes.
> >> I've created a new page in Confluence for these drafts, so that we can 
> >> discuss further ideas (both conceptually and in terms of naming) there in 
> >> the comments [1].
> >> 
> >> The page is thematically divided into two parts. First, I have put 
> >> together a brief overview of the status quo of the return type of Data 
> >> Lake queries. Then follows a part with the two proposals, which differ 
> >> only slightly from each other.
> >> 
> >> It would be great if you leave me feedback and your ideas for adjustments 
> >> there in the comments, so that we can discuss the ideas together before 
> >> implementation.
> >> 
> >> Thanks and regards,
> >> Daniel
> >> 
> >> 
> >> [1] 
> >> https://cwiki.apache.org/confluence/display/STREAMPIPES/Harmonization+of+Data+Lake+Query+Result
> >> 
> >> 
> 
> 

Re: Harmonization of Data Lake Query Result

[Johannes Tex]

Hi Daniel,

the attributes "labels" and "pageSum", which you call obsolete in your 
proposals , are used in [1]. Maybe that is also obsolete ;)

Johnnaes

[1] 
https://github.com/apache/incubator-streampipes/blob/master/ui/src/app/core-ui/image/image-categorize/image-categorize.component.ts

On 2021/08/10 11:20:58, Daniel Ebi  wrote: 
> Hi all,
> 
> As part of the refactoring and integration of the Data Lake REST API into the 
> Data Explorer, it has turned out that it makes sense to harmonize the return 
> type of Data Lake queries first. Consequently, I have already drafted two 
> proposals for the alignment of DataResult, GroupedDataResult and PageResult 
> classes.
> I've created a new page in Confluence for these drafts, so that we can 
> discuss further ideas (both conceptually and in terms of naming) there in the 
> comments [1].
> 
> The page is thematically divided into two parts. First, I have put together a 
> brief overview of the status quo of the return type of Data Lake queries. 
> Then follows a part with the two proposals, which differ only slightly from 
> each other.
> 
> It would be great if you leave me feedback and your ideas for adjustments 
> there in the comments, so that we can discuss the ideas together before 
> implementation.
> 
> Thanks and regards,
> Daniel
> 
> 
> [1] 
> https://cwiki.apache.org/confluence/display/STREAMPIPES/Harmonization+of+Data+Lake+Query+Result
> 
> 

Re: AW: Review of Data Lake Sink and REST API

[Johannes Tex]

Hi Daniel,

I added my comments.

The issue with file storage can also be another issue and a follow-up ticket. 
But you might want to tackle it right away. :)

Johannes

On 2021/04/26 15:06:24, Daniel Ebi  wrote: 
> Hi all,
> 
> I have just created a first draft for the revised endpoint definitions. You 
> can find it in the comments below the confluence page about the Data Lake 
> sink [1].
> 
> I am very interested in your comments and suggestions for adjustments.
> 
> Regards,
> Daniel
> 
> 
> [1] https://cwiki.apache.org/confluence/display/STREAMPIPES/Data+Lake+Sink 
> 
> 
> 
> -Ursprüngliche Nachricht-
> Von: Daniel Ebi 
> Gesendet: Montag, 26. April 2021 10:03
> An: 'dev@streampipes.apache.org' 
> Betreff: AW: Review of Data Lake Sink and REST API
> 
> Hi Dominik, hi Johannes,
> 
> first, thank you both for your thoughts on the Data Lake API adaptions and 
> improvements.
> 
> @Dominik: Great, since you're already working on harmonizing authentication, 
> we can focus primarily on the REST interface and endpoint definition here for 
> now.
> 
> Using query parameters instead of long paths should definitely increase the 
> user convenience. That's a good point.
> I started this morning with working on a concept with adapted and simplified 
> endpoint definitions. If you want to provide input there, Dominik, you are 
> very welcome to do so. Then I can incorporate this one right away. Otherwise 
> I can make a first draft, which we can then discuss and adapt in accordance 
> to your input.
> 
> Regards,
> Daniel
> 
> -Ursprüngliche Nachricht-
> Von: Dominik Riemer  
> Gesendet: Montag, 26. April 2021 09:27
> An: dev@streampipes.apache.org
> Betreff: RE: Review of Data Lake Sink and REST API
> 
> Hi Daniel,
> 
> thanks for bringing this up!
> 
> I added some comments, in my opinion the redesign of the resource endpoint is 
> the most important thing right now - the authentication concept is something 
> I'm currently working on within the scope of STREAMPIPES-319, where I'm first 
> harmonizing service discovery and one of the next step is to inject an 
> instance of the StreamPipes client into the EventProcessorRuntimeContext, so 
> that pipeline elements can easily request data from the core APIs.
> 
> Dominik
> 
> -Original Message-
> From: Johannes Tex  
> Sent: Saturday, April 24, 2021 4:40 PM
> To: dev@streampipes.apache.org
> Subject: Re: Review of Data Lake Sink and REST API
> 
> Hi Daniel,
> 
> I added some comments.
> 
> Greetings
> Johannes
> 
> On 2021/04/23 08:09:54, Daniel Ebi  wrote: 
> > Hi all,
> > 
> > I have spent the last few days reviewing the implementation of the Data 
> > Lake sink and in particular the corresponding REST API. The goal was to 
> > identify open TODOs and some possible adaptations or extensions for the 
> > REST API.
> > I've created a new page in Confluence for the results of the review, so 
> > that we can discuss further ideas there in the comments [1].
> > 
> > The page is thematically divided into two parts. First, I have put together 
> > a concise description of the status quo of the sink implementation and the 
> > specification of the REST API including open TODOs. Then follows a part 
> > with the ideas for possible adaptations and extensions of the interface.
> > 
> > It would be great if you leave me feedback and your ideas for adjustments 
> > of the sink / REST API there in the comments, so that we can discuss the 
> > ideas together. Because in a next step I would like to work out a draft of 
> > the API with updated endpoint definitions.
> > 
> > Thanks and regards,
> > Daniel
> > 
> > 
> > [1] https://cwiki.apache.org/confluence/display/STREAMPIPES/Data+Lake+Sink
> > 
> > 
> 
> 

Re: Review of Data Lake Sink and REST API

[Johannes Tex]

Hi Daniel,

I added some comments.

Greetings
Johannes

On 2021/04/23 08:09:54, Daniel Ebi  wrote: 
> Hi all,
> 
> I have spent the last few days reviewing the implementation of the Data Lake 
> sink and in particular the corresponding REST API. The goal was to identify 
> open TODOs and some possible adaptations or extensions for the REST API.
> I've created a new page in Confluence for the results of the review, so that 
> we can discuss further ideas there in the comments [1].
> 
> The page is thematically divided into two parts. First, I have put together a 
> concise description of the status quo of the sink implementation and the 
> specification of the REST API including open TODOs. Then follows a part with 
> the ideas for possible adaptations and extensions of the interface.
> 
> It would be great if you leave me feedback and your ideas for adjustments of 
> the sink / REST API there in the comments, so that we can discuss the ideas 
> together. Because in a next step I would like to work out a draft of the API 
> with updated endpoint definitions.
> 
> Thanks and regards,
> Daniel
> 
> 
> [1] https://cwiki.apache.org/confluence/display/STREAMPIPES/Data+Lake+Sink
> 
> 

Re: [VOTE] Apache StreamPipes logo

[Johannes Tex]

Hi,

I vote for #4.

 Johannes

On 2021/01/26 12:41:22, Marco Heyden  wrote: 
> Hey,
> 
> I like #2 the most.
> 
> Best
> Marco
> Am 26. Jan. 2021, 11:59 +0100 schrieb Philipp Zehnder :
> > Hi,
> >
> > I vote for #2
> >
> > Philipp
> >
> > > On 25. Jan 2021, at 21:37, Dominik Riemer  wrote:
> > >
> > > Hi all,
> > >
> > > this is a vote to (finally) select a new logo for Apache StreamPipes!
> > >
> > > We have five candidates which can be found in our wiki:
> > > https://cwiki.apache.org/confluence/display/STREAMPIPES/Logo
> > >
> > > Everyone has one vote, please indicate the logo number (first column) of 
> > > the logo you like most.
> > > The vote will be open for at least 72 hours.
> > >
> > > I'm really excited how our new logo will look like 
> > >
> > > Dominik
> > >
> > >
> > >
> >
> 

Re: RE: [BUG] Accessing Images from Data Lake Identifiers with Underscores

[Johannes Tex]

Hi,

yes, another and better possibility would be to use IDs. In this case we would 
have to store the path and ID together so that we can access the image again. 
To read the image, the response time will increase (database look up), but I 
don't think that will be a problem.

I opened a ticket for that [1].

Johannes 

[1] https://issues.apache.org/jira/browse/STREAMPIPES-265

On 2020/12/03 10:07:07, "Dominik Riemer"  wrote: 
> Hi,
> 
> I'd also prefer something which works with an ID.
> Some application servers disallow encoded slashes by default, for the 
> backend, we need to override this (see ALLOW_ENCODED_SLASH) in [1]. 
> Just in case you ever spend hours finding out why your encoded slash is not 
> working 
> 
> Dominik
> 
> [1] 
> https://github.com/apache/incubator-streampipes/blob/dev/streampipes-backend/src/main/java/org/apache/streampipes/backend/StreamPipesBackendApplication.java
> 
> 
> -Original Message-
> From: udeho  
> Sent: Thursday, December 3, 2020 11:02 AM
> To: dev@streampipes.apache.org
> Subject: Re: [BUG] Accessing Images from Data Lake Identifiers with 
> Underscores
> 
> Hi Johannes,
> 
> I would also assume that an encoding with backslashes is the easiest solution.
> Maybe using IDs instead of encoding the file route could be an option to 
> avoid encoding issues.
> But I do not know whether this can be done without to much effort.
> 
> Best,
> Tim
> 
> On Nov. 28 2020, at 10:24 am, Johannes Tex  wrote:
> > Hello, Tim,
> >
> > You are right. We encode some chars of the image path location when we save 
> > the image in the Datalake [1]. This was necessary that the can images via 
> > an http route [2].
> > I think the easiest way would be to change the encoding. Maybe we should 
> > encode with '%2F', this is the encoding for slashes. But I am not sure if 
> > we can still access the images with [2]. Do you have another idea? The best 
> > solution would be a solution where the encoding can be removed.
> > Johannes
> > [1] 
> > https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampipes-sinks-internal-jvm/src/main/java/org/apache/streampipes/sinks/internal/jvm/datalake/DataLake.java
> > [2] 
> > https://github.com/apache/incubator-streampipes/blob/dev/streampipes-rest/src/main/java/org/apache/streampipes/rest/impl/datalake/DataLakeResourceV3.java
> >
> > On 2020/11/24 14:24:21, udeho  wrote:
> > > Hi all,
> > >
> > > For a small object detection use case I have stored some images in the 
> > > internal data lake to label them with the labeling tool.
> > > I discovered that the use of underscores in the data lake identifier 
> > > prevents the user from accessing the images later in the data explorer 
> > > and the labeling tool.
> > > As far as I can see, the underscore is already used as a delimiter in the 
> > > http-request, which leads to a conflict if the identifier also contains 
> > > an underscore.
> > >
> > > Best regards,
> > > Tim
> > >
> > >
> >
> 
> 
> 

[jira] [Created] (STREAMPIPES-265) Accessing Images from Data Lake Identifiers with Underscores

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-265:


 Summary:  Accessing Images from Data Lake Identifiers with 
Underscores
 Key: STREAMPIPES-265
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-265
 Project: StreamPipes
  Issue Type: Bug
  Components: Data Lake
Reporter: Johannes Tex


The images in the Datalake should be accessible via the ID and not via the path

 

See Discussion:

https://lists.apache.org/thread.html/rbbea5ecba0ba8b5b3dc5f2dcc738c640fb917a696a41c58110a9b8e5%40%3Cdev.streampipes.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Promotional Material

[Johannes Tex]

Hi All,

@Philipp: it is only the middle of december.

That sound greats! Thanks to all :)

Johannes


On 2020/11/30 06:59:20, Patrick Wiener  wrote: 
> Hi,
> 
> that’s fine for me - we can have a default set of slides there and some 
> sample data for a demo.
> 
> Patrick
> 
> > Am 29.11.2020 um 22:52 schrieb Dominik Riemer :
> > 
> > Hi,
> > 
> > +1 for creating a wiki page with a presentation template and an how-to 
> > guide on showing some simple demos.
> > 
> > For instance, we could upload the ApacheCon presentation from this year 
> > along with some less technical slides so that everyone could pick 
> > appropriate slides from this template.
> > 
> > I guess most slides we usually show are from either Philipp, Patrick or me, 
> > so in case Patrick also agrees, we can upload the slides within the next 
> > days.
> > 
> > Dominik
> > 
> > On 2020/11/29 18:05:05, Philipp Zehnder  wrote: 
> >> Hi Johannes,
> >> 
> >> I think that is a great idea. We should put a presentation explaining the 
> >> general functionalities into confluence. Additionally, we should put there 
> >> a description on how to setup and perform a demo.
> >> 
> >> What do the others think?
> >> 
> >> When is your presentation in December? If it is already next week you can 
> >> contact me and I can send you some slides.
> >> 
> >> Philipp
> >> 
> >> 
> >>> On 28. Nov 2020, at 09:59, Johannes Tex  wrote:
> >>> 
> >>> Hi,
> >>> 
> >>> I am planning an internal presentation about StreamPipes in my company in 
> >>> December. I would like to ask if we have "promotional material" that 
> >>> could be used for this. On the website [1] are some slides, but not the 
> >>> "source", so I can’t edit them. Maybe it would be a good idea to store 
> >>> such presentation slides in confluence?
> >>> 
> >>> Johannes
> >>> 
> >>> 
> >>> [1] https://streampipes.apache.org/media.html
> >>> 
> >> 
> >> 
> 
> 

Re: [BUG] Accessing Images from Data Lake Identifiers with Underscores

[Johannes Tex]

Hello, Tim,

You are right. We encode some chars of the image path location when we save the 
image in the Datalake [1]. This was necessary that the can images via an http 
route [2].

I think the easiest way would be to change the encoding. Maybe we should encode 
with '%2F', this is the encoding for slashes. But I am not sure if we can still 
access the images with [2]. Do you have another idea? The best solution would 
be a solution where the encoding can be removed.

Johannes

[1] 
https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampipes-sinks-internal-jvm/src/main/java/org/apache/streampipes/sinks/internal/jvm/datalake/DataLake.java
[2] 
https://github.com/apache/incubator-streampipes/blob/dev/streampipes-rest/src/main/java/org/apache/streampipes/rest/impl/datalake/DataLakeResourceV3.java

On 2020/11/24 14:24:21, udeho  wrote: 
> Hi all,
> 
> For a small object detection use case I have stored some images in the 
> internal data lake to label them with the labeling tool.
> I discovered that the use of underscores in the data lake identifier prevents 
> the user from accessing the images later in the data explorer and the 
> labeling tool.
> As far as I can see, the underscore is already used as a delimiter in the 
> http-request, which leads to a conflict if the identifier also contains an 
> underscore.
> 
> Best regards,
> Tim
> 
> 

Promotional Material

[Johannes Tex]

Hi,

I am planning an internal presentation about StreamPipes in my company in 
December. I would like to ask if we have "promotional material" that could be 
used for this. On the website [1] are some slides, but not the "source", so I 
can’t edit them. Maybe it would be a good idea to store such presentation 
slides in confluence?

Johannes


[1] https://streampipes.apache.org/media.html

Re: Datalake - REST Extension

[Johannes Tex]

Hey,

IMO this is a good place.

Just a suggestion: Should the retention policy also be part of the datalake 
sink configuration?

Johannes

On 2020/09/22 08:16:58, Philipp Zehnder  wrote: 
> Hi,
> 
> this will be very helpful, because currently the user just sees the actual 
> data that is stored in the data explorer and has the possibility to delete 
> everything in the configurations view.
> 
> My suggestion would be to add this information to the configurations view, 
> then we have one place to manage all stored data.
> 
> How do you like that?
> 
> Philipp
> 
> 
> 
> > On 22. Sep 2020, at 10:07, Johannes Tex  wrote:
> > 
> > Hi Felix,
> > 
> > I like the idea of displaying the metadata and the ability to change 
> > configurations.
> > Where do you want to display the metadata in the UI?
> > 
> > Johannes
> > 
> > On 2020/09/22 07:06:21, Felix John  wrote: 
> >> Hey all,
> >> 
> >> I am currently working on the REST interface for our datalake. I would 
> >> like to extend the existing functionality with additional REST calls that 
> >> provide the user the following metadata information
> >> 
> >> - number of events
> >> - current retention policy
> >> - etc. ...
> >> 
> >> as well as the ability to
> >> 
> >> - change and modify the retention policy
> >> - remove individual indices in the database
> >> 
> >> What do you think? Have you got any suggestions?
> >> 
> >> Regards,
> >> Felix
> 
> 

Re: Datalake - REST Extension

[Johannes Tex]

Hi Felix,

I like the idea of displaying the metadata and the ability to change 
configurations.
Where do you want to display the metadata in the UI?

Johannes

On 2020/09/22 07:06:21, Felix John  wrote: 
> Hey all,
> 
> I am currently working on the REST interface for our datalake. I would like 
> to extend the existing functionality with additional REST calls that provide 
> the user the following metadata information
> 
> - number of events
> - current retention policy
> - etc. ...
> 
> as well as the ability to
> 
> - change and modify the retention policy
> - remove individual indices in the database
> 
> What do you think? Have you got any suggestions?
> 
> Regards,
> Felix

Re: Added RunConfigurations as Project Files for IntelliJ

[Johannes Tex]

Hi All,

i usually use *application.properties* file [1] for Spring boot Applications to 
set my enviroments varibales. I think it does the same as setting via POM? 

The advantage is that the Spring boot Application check if the env var is set 
in the system, and if not, it taking the default value. E.g:

application.yml
-
eureka:
  client:
serviceUrl:
  defaultZone: http://${eureka.address:localhost:8761/eureka/} 
#envName:Default


In this case the application checks if the env "eureka.address" is set and if 
not, the default value "localhost:8761/eureka/" is takken (Dev). For production 
I only set the environment variable in docker "eureka.address" to 
"eureka:8761/eureka/".

Maybe this would be also a solution for StreamPipes?

Johannes

[1] https://www.baeldung.com/properties-with-spring#6-alternative---yaml-files

On 2020/07/24 05:22:32, Patrick Wiener  wrote: 
> Indeed. Thats a good point. We need to make sure that this is not leading to 
> some conflicts when
> starting in a docker container and setting the env variables from the 
> docker-compose or k8s manifest.
> 
> Idk but is there any spring boot magic that we can tell it it’s only a dev 
> profile setting?
> 
> Patrick
> 
> > Am 23.07.2020 um 22:49 schrieb Philipp Zehnder :
> > 
> > Hi Patrick,
> > 
> > you are right. The solution did just work for IntelliJ. But this pull 
> > request should fix this problem [1]. There the environment variables are 
> > defined in the pom file. The services can be started direclty from the 
> > command line. I would additionally keep the IntelliJ configuration to ease 
> > the setup for users, but we should remove the environment variables from 
> > this configuration.
> > I was wondering what happens when this runs in docker? Are the new 
> > environment variables then used as well? If this is the case, we need a 
> > solution to change the default values for productions. 
> > 
> > We should definitely update the documentation. Where should we put this 
> > kind of information?
> > 
> > Philipp
> > 
> > 
> > [1] https://github.com/apache/incubator-streampipes/pull/25 
> > 
> > 
> >> On 20. Jul 2020, at 10:46, Patrick Wiener  wrote:
> >> 
> >> Hi all,
> >> 
> >> While it works on Mac/Windows hosts, a problem that still arise and we 
> >> should be aware of is on Linux based development environments,
> >> where the developers host is a Linux OS. The problem occurs due to the 
> >> fact that Docker on Linux cannot resolve host.docker.internal
> >> 
> >> Thus, the developer would need to adjust the env variable in the run 
> >> Configuration manually:
> >> 
> >> as per specified run config - fine for OSX/Windows:
> >> 
> >> SP_PORT=6025;SP_HOST=host.docker.internal;SP_DEBUG=true
> >> 
> >> Linux:
> >> SP_PORT=6025;SP_HOST=;SP_DEBUG=true
> >> 
> >> On Linux, the developer has two options:
> >> 
> >> 1) set SP_HOST to his/her host IP —> Problem: not agnostic to changing 
> >> network environments
> >> 2)  set SP_HOST to docker0 bridge IP —> should be agnostic to changing 
> >> network environments [Preferred]
> >> 
> >> He/she can look up the IP’s via ifconfig.
> >> 
> >> We definitely need to update the documentation as well - not only in the 
> >> repositories. Currently it states that you’d still need the env File 
> >> plugin.
> >> Maybe because we haven’t updated the archetypes as well?
> >> 
> >> One minor: Overall this solution only works with Intellij IDE - not 
> >> Eclipse for instance.
> >> 
> >> 
> >> Patrick
> >> 
> >> 
> >>> Am 02.07.2020 um 15:38 schrieb Philipp Zehnder :
> >>> 
> >>> Hi,
> >>> 
> >>> @Felix, very cool. Everything works fine and I merged it into dev.
> >>> 
> >>> @Patrick, I removed the configurations for the EnvFile plugin in the 
> >>> configuration XML, so we should not need this anymore.
> >>> 
> >>> @All: Any ideas for naming the services? Because now we have a quite long 
> >>> list in the run configurations. I would suggest to  use a prefix for the  
> >>> Processor Containers and the backend services?
> >>> The core and connect master already have the prefix “Apache StreamPipes”, 
> >>> but I think this is a bit too long. Any ideas or suggestions?
> >>> 
> >>> Philipp
> >>> 
>  On 2. Jul 2020, at 11:17, Felix John  wrote:
>  
>  Hi Patrick,
>  
>  good question. I just tested one configuration whilst disabling the 
>  "EnvFile" plugin. I worked out nicely.
>  
>  
>  Greetings,
>  Felix
>  
>  ‐‐‐ Original Message ‐‐‐
>  On Wednesday, 1. July 2020 22:03, Patrick Wiener  
>  wrote:
>  
> > Hi Felix,
> > 
> > cool - that def helps the onboarding process for those using IntelliJ.
> > 
> > Do these runConfigurations also need to have the env-File plugin 
> > pre-installed to work?
> > 
> > Patrick
> > 
> >> Am 01.07.2020 um 18:34 schrieb Felix John j...@axantu.com:
> >> Hi,
> >> I have added 

[jira] [Resolved] (STREAMPIPES-184) Image Enricher

[Johannes Tex (Jira)]

 [ 
https://issues.apache.org/jira/browse/STREAMPIPES-184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johannes Tex resolved STREAMPIPES-184.
--
Resolution: Implemented

> Image Enricher
> --
>
> Key: STREAMPIPES-184
> URL: https://issues.apache.org/jira/browse/STREAMPIPES-184
> Project: StreamPipes
>  Issue Type: Improvement
>  Components: Pipeline Elements
>        Reporter: Johannes Tex
>    Assignee: Johannes Tex
>Priority: Major
>
> The score and label should be shown for each box.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (STREAMPIPES-184) Image Enricher

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-184:


 Summary: Image Enricher
 Key: STREAMPIPES-184
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-184
 Project: StreamPipes
  Issue Type: Improvement
  Components: Pipeline Elements
Reporter: Johannes Tex
Assignee: Johannes Tex


The score and label should be shown for each box.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Adding StreamPipes Python wrapper

[Johannes Tex]

Hello everyone,

I think it would be very useful to have a python wrapper, +1 for this purpose. 
I would also prefer a native Python solution (without Java) to keep the less 
complex solution for the users, although it will be a lot of work to migrate 
everything. There is also a project [1] that tries to convert Java code to 
Python code, but I'm not sure if it works well.

Maybe it's time to introduce a language independent StreamPipes model and/or 
API description? Then a lot of the required boilerplate code could simply be 
generated and it would be easier to create wrappers for other languages.

Johannes

[1] https://github.com/natural/java2python/





On 2020/07/17 08:30:12, Philipp Zehnder  wrote: 
> I totally agree, from the user point of view (PE developer) we need a native 
> python solution.
> The question are, how can we realize that and how can we start.
> 
> I have one addition to your list:
> * Support configuration parameters in the consul key-value store
> 
> 
> 
> > On 17. Jul 2020, at 09:17, Patrick Wiener  wrote:
> > 
> > Idk yet either. However, lastly all a processor really is, is a 
> > microservice with an API, some boilerplate code and the logic iteself put 
> > in inside a docker container.
> > Thus we get a REST request from the backend including configurations to 
> > start the given processor and a REST request from the backend to stop it.
> > 
> > So what would we need?
> > 
> > * a way for this service to register itself in consul
> > * a way for it to describe itself to the backend (declareModel)
> > * a way to consume/produce events (maybe in the first step with a fixed 
> > protocol such as kafka)
> > 
> > anything else that I am missing?
> > 
> > So, yes we could potentially add is as it is as part of the extensions 
> > project. However, in the long rung, there will be base implementations of 
> > the above mentioned
> > points that are more suitable as part of the core, rather than on the 
> > „extension“ side - that’s my only concern right now as the current 
> > prototype is really nothing more
> > than a prototype.
> > 
> > Maybe we should think of a way of how we could not reimplement everything 
> > in Python but rather extending the backend side in a way that makes it 
> > easier to integrate
> > it in general.
> > 
> > So I am open
> > 
> > Patrick
> > 
> > 
> >> Am 17.07.2020 um 08:18 schrieb Philipp Zehnder :
> >> 
> >> Hi,
> >> 
> >> I still don't quite understand how we can integrate the Python code into 
> >> the backend module. 
> >> Because then we would have to put  the Python processor implementation in 
> >> the backend as well, right? 
> >> Or how can we use the Python dependencies in the extensions module?
> >> 
> >> Currently we use ExternalEventProcessor to trigger the invocation of a 
> >> processor. Then a REST request is send from java to python.
> >> Maybe we can add some of the java boilerplate code into the 
> >> streampipes-wrapper-python module. But how about the python REST API?
> >> 
> >> Philipp
> >> 
> >>> On 16. Jul 2020, at 23:29, Dominik Riemer  wrote:
> >>> 
> >>> Hi,
> >>> 
> >>> I'm fully +1 for a complete, plain python wrapper that integrates both 
> >>> runtime and controller interfaces! Also, given our microservice 
> >>> architecture with standalone pipeline elements that communicate over 
> >>> JSON/JSON-LD I don't think we need any code-level integration between 
> >>> Python and Java. 
> >>> 
> >>> Concerning the code structure, I'd suggest to create a 
> >>> streampipes-wrapper-python module in the core project, add the Python 
> >>> code there and to create an example using the current Java 
> >>> ExternalEventProcessor into the streampipes-examples project that 
> >>> explains how to use the Python wrapper. By adding the Python code to the 
> >>> core project, all wrappers would be located in the same repository, while 
> >>> the extensions project solely provides specific pipeline elements and 
> >>> adapters.
> >>> In the meantime, we could add the missing features to the Python wrapper. 
> >>> I agree that it is some work, but it should mainly consist of parsing the 
> >>> graphs (we could use JSON instead of JSON-LD here to simplify parsing), 
> >>> extracting parameters and adding some Flask endpoints.
> >>> 
> >>> As I'm not that familiar with Python, are there any Python experts on the 
> >>> list who want to help building the wrapper? I'd expect that finishing the 
> >>> wrapper could probably be done within a few days if there is a Python 
> >>> expert and someone who is familiar with the StreamPipes model - I'd be 
> >>> happy to support the model side 
> >>> 
> >>> Dominik 
> >>> 
> >>> 
> >>> -Original Message-
> >>> From: Philipp Zehnder  
> >>> Sent: Thursday, July 16, 2020 11:09 PM
> >>> To: dev@streampipes.apache.org
> >>> Subject: Re: Adding StreamPipes Python wrapper
> >>> 
> >>> Hi guys,
> >>> 
> >>> I am also in favor of integrating the current prototype of the python 
> >>> wrapper for 

[jira] [Closed] (STREAMPIPES-171) Datalake sink configurations

[Johannes Tex (Jira)]

 [ 
https://issues.apache.org/jira/browse/STREAMPIPES-171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johannes Tex closed STREAMPIPES-171.

Resolution: Implemented

> Datalake sink configurations
> 
>
> Key: STREAMPIPES-171
> URL: https://issues.apache.org/jira/browse/STREAMPIPES-171
> Project: StreamPipes
>  Issue Type: Improvement
>  Components: Pipeline Elements
>        Reporter: Johannes Tex
>    Assignee: Johannes Tex
>Priority: Major
>
> Datalake sink configurations should be set by the container settings and not 
> in code



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (STREAMPIPES-171) Datalake sink configurations

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-171:


 Summary: Datalake sink configurations
 Key: STREAMPIPES-171
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-171
 Project: StreamPipes
  Issue Type: Improvement
  Components: Pipeline Elements
Reporter: Johannes Tex
Assignee: Johannes Tex


Datalake sink configurations should be set by the container settings and not in 
code



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (STREAMPIPES-78) Image Labeling Tool

[Johannes Tex (Jira)]

 [ 
https://issues.apache.org/jira/browse/STREAMPIPES-78?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johannes Tex closed STREAMPIPES-78.
---
Resolution: Implemented

> Image Labeling Tool
> ---
>
> Key: STREAMPIPES-78
> URL: https://issues.apache.org/jira/browse/STREAMPIPES-78
> Project: StreamPipes
>  Issue Type: New Feature
>  Components: Backend, UI
>        Reporter: Johannes Tex
>    Assignee: Johannes Tex
>Priority: Major
>
> Development of an image labeling tool to label the stored images from the 
> Datalake. The Labels will be stored in the _COCO Annonation_ Format. [1]
>  * Bouding Box Labeling
>  * Polygon Labeling
> [1] [http://cocodataset.org/#format-data]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: RE: [VOTE] Apache StreamPipes (incubating) 0.66.0 RC1 release

[Johannes Tex]

Hi all,


+1 (binding)

I checked:
- Specified url to download staged release artifacts (core, extensions, 
installer) are valid
- Signature is correct for all staged release artifacts
- Signature reference contain valid Apache email address
- Checksum is correct
- LICENSE, NOTICE, README, RELEASE_NOTES, RELEASE_VALIDATION files are included 
in extracted source bundle
- RAT run is valid
- No SNAPSHOT dependencies found
- Compile and build source successful (backend, UI, extensions)
- Build and run Test system on Docker successful

Thanks for all your work Dominik!

Johannes

On 2020/05/06 20:14:55, "Dominik Riemer"  wrote: 
> Hi,
> 
> sorry, I just discovered that the incubating name was missing in the
> installer artifact. I updated it in the dist directory, the file contents
> didn't change.
> Sorry for any inconvenience!
> 
> Dominik
> 
> -Original Message-
> From: Dominik Riemer  
> Sent: Tuesday, May 5, 2020 8:53 PM
> To: dev@streampipes.apache.org
> Subject: [VOTE] Apache StreamPipes (incubating) 0.66.0 RC1 release
> 
> Hi,
> 
> Apache StreamPipes (Incubating) 0.66.0 has been staged and it's time to vote
> on accepting it for release. 
> If approved, we will seek final release approval from the IPMC. Voting will
> be open for 72 hours. 
> A minimum of 3 binding +1 votes and more binding +1 than binding -1 are
> required to pass, but everyone is welcome to vote! 
> 
> Three artifacts are relevant for this vote: 
> 
> incubator-streampipes, staged at [1], available in Nexus at [2], release
> tag: release/0.66.0, hash for the release tag:
> b8c23e6785eaad79da6dd03072f2c1dcbfa467b4
> incubator-streampipes-extensions, staged at [3], release tag:
> release/0.66.0, hash for the release tag:
> 44b7dd13c75058811a001f1aa6b3f6805285eb2d
> incubator-streampipes-installer, staged at [4], release tag: release/0.66.0,
> hash for the release tag: 1a8054239d8f42d66ee95e7b1282708f5f1266b0
> 
> Per [5] "Before voting +1, [P]PMC members are required to download the
> signed source code package, compile it as provided, and test the resulting
> executable on their own platform, along with also verifying that the package
> meets the requirements of the ASF policy on releases." 
> 
> A release validation guide is available at [6]. The KEYS file is available
> at [7] 
> 
> [ ] +1 accept (indicate what you validated - e.g. performed the checklist at
> the end of [6]) [ ] -1 reject (explanation required) 
> 
> Thanks for taking your time for validating this release!
> 
> Dominik
> 
> 
> [1]
> https://dist.apache.org/repos/dist/dev/incubator/streampipes/core/0.66.0/rc1
> [2]
> https://repository.apache.org/content/repositories/orgapachestreampipes-1006
> [3]
> https://dist.apache.org/repos/dist/dev/incubator/streampipes/extensions/0.66
> .0/rc1
> [4]
> https://dist.apache.org/repos/dist/dev/incubator/streampipes/installer/0.66.
> 0/rc1
> [5] https://www.apache.org/dev/release.html#approving-a-release
> [6]
> https://cwiki.apache.org/confluence/display/STREAMPIPES/Validating+a+release
> [7] https://dist.apache.org/repos/dist/dev/incubator/streampipes/KEYS
> 
> 
> 
> 
> 

Re: New Contributer | Time Series Data Labeling Tool

[Johannes Tex]

Hi Daniel,

welcome :)

As you mentioned, I have started to develop an image labeling tool. 
I think some of the components can be resused, for example the "image-labels" 
component [1].
It is a component for selecting labels received from the backend (at the moment 
it is only mocked). You also can use the num-pad change labels, it's for the 
power users ;)
You can have a view and we can discuss it if you have comments or questions.

Johannes

[1] 
https://github.com/apache/incubator-streampipes/tree/b2e92c667d77e7b7e724112463bb25141881ffd6/ui/src/app/core-ui/image/components/image-labels

On 2020/03/24 10:54:12, Daniel Ebi  wrote: 
> Hey there,
> my name is Daniel and I would like to contribute to Apache StreamPipes 
> project. Therefore I recently joined this dev mailing list.
> 
> About me
> I study Information Engineering and Management in the Master's program - with 
> focus on Machine Learning and Artificial Intelligence - at Karlsruhe 
> Institute of Technologie (KIT). Additionally, I work as a student assistant 
> at the FZI in Karlsruhe in order to gain some practical experience in 
> research. There I got in contact with StreamPipes and became interested in 
> the project. In the course of my studies I already gathered much experience 
> in the Machine Learning and Data Science domain.
> 
> What are my current ideas for Apache StreamPipes?
> Specifically, I would like to develop a labeling feature for Apache 
> StreamPipes that allows users to label time series data easily and 
> intuitively (like the image labeling tool of Johannes [1]). My idea is to 
> extend the line chart of the data explorer in such a way that one or more 
> labels can be added by simply marking the data points of interest. The 
> visualization of the data is thereby used as a basis for the labeling, and 
> the labelled data can be afterwards illustrated with colored overlays, too. 
> The long-term goal is to ensure that the data labelled in this way can be 
> used as a starting point for machine learning procedures.
> 
> I will be very happy to hear about your suggestions about my plans. Thanks in 
> advance.
> 
> Best regards,
> Daniel Ebi
> 
> 
> [1] 
> https://issues.apache.org/jira/projects/STREAMPIPES/issues/STREAMPIPES-78?filter=allopenissues
> 
> 

Re: Preparing release

[Johannes Tex]

Hi,

I agree, the new data explorer will not be ready for that release.
Even from my side, we can freeze the current state.

Johannes

On 2020/03/16 21:11:46, Philipp Zehnder  wrote: 
> Hi Dominik,
> 
> Johannes and I are currently working on a new version for the data explorer. 
> But we will not finish it before the next release.
> I am also in favor of freezing the current features in the development branch 
> and focus on the release.
> 
> Philipp
> 
> > On 16. Mar 2020, at 20:50, Dominik Riemer  wrote:
> > 
> > Hi all,
> > 
> > is there anything you are currently working on which should be part of the 
> > next release?
> > 
> > If not, I'd suggest to freeze the current development on the dev branch, do 
> > some testing and then go for our first release ;-)
> > 
> > Dominik 
> > 
> > On 2020/03/06 13:04:35, Johannes Tex  wrote: 
> >> Hi Dominik,
> >> 
> >> yeah, thanks for solving the issues :) 
> >> 
> >> Johannes
> >> 
> >> On 2020/03/03 22:13:51, Dominik Riemer  wrote: 
> >>> Hi Johannes,
> >>> 
> >>> thanks a lot, that a great overview.
> >>> I already fixed some of the issues you mentioned, it seems we are on a 
> >>> good way towards our first release :-)
> >>> 
> >>> Dominik
> >>> 
> >>> On 2020/02/28 18:05:18, Johannes Tex  wrote: 
> >>>> Hi all,
> >>>> 
> >>>> a few days ago we started thinking about the first Apache release [1]. 
> >>>> So I started to do functional tests: I created some issues that I think 
> >>>> we should fix before the release, I marked the issues with release 
> >>>> version '0.66.0'. You can find them all here: [2].
> >>>> 
> >>>> Regards
> >>>> Johannes
> >>>> 
> >>>> [1] 
> >>>> http://mail-archives.apache.org/mod_mbox/streampipes-dev/202002.mbox/%3C018b01d5e6fd%24842f62b0%248c8e2810%24%40apache.org%3E
> >>>> [2] https://issues.apache.org/jira/projects/STREAMPIPES/versions/12347025
> >>>> 
> >>>> 
> >>>> 
> >>> 
> >> 
> 
> .
> M. Sc. Philipp Zehnder
> Wissenschaftlicher Mitarbeiter | Research Scientist
> Information Process Engineering (IPE)
>  
> FZI Forschungszentrum Informatik
> Haid-und-Neu-Str. 10–14 
> 76131 Karlsruhe, Germany
> Tel.: +49 721 9654-805
> Fax: +49 721 9654-806
> 
> zehn...@fzi.de <mailto:zehn...@fzi.de>
> https://www.fzi.de/mitarbeiter/philipp-zehnder
>  
> .
> FZI Forschungszentrum Informatik
> Stiftung des bürgerlichen Rechts
> Stiftung Az: 14-0563.1 Regierungspräsidium Karlsruhe
> Vorstand: Prof. Dr. Andreas Oberweis, Jan Wiesenberger, Prof. Dr.-Ing. J. 
> Marius Zöllner
> Vorsitzender des Kuratoriums: Ministerialdirigent Günther Leßnerkraus
> .
> 
> 

Re: [Help wanted] Jslod Serialization

[Johannes Tex]

Thank you! :)

Johannes

On 2020/03/06 16:15:01, Philipp Zehnder  wrote: 
> Hi Johannes,
> 
> I just fixed it. 
> The problem was that the REST call returned and Array of DataLakeMeasure as 
> JSON-LD.
> To do that you have to add asContainer:
> 
> public Response getAllInfos() {
>   List result = this.dataLakeManagement.getInfos();
> 
>   return ok(asContainer(result));
> }
> I’ll continue to work on the new data explorer and push the latest version 
> later.
> 
> Philipp
> 
> 
> 
> > On 6. Mar 2020, at 17:02, Dominik Riemer  wrote:
> > 
> > Hi Johannes,
> > 
> > did you try to call the constructor defined in UnnamedStreamPipesEntity 
> > (just call the empty contructor with super() in all constructors of 
> > DataLakeMeasure)?
> > 
> > Dominik
> > 
> > -Original Message-
> > From: Johannes Tex  
> > Sent: Friday, March 6, 2020 2:00 PM
> > To: dev@streampipes.apache.org
> > Subject: [Help wanted] Jslod Serialization
> > 
> > Hello everyone,
> > 
> > the REST endpoint for receiving the information about the stored data in 
> > the data lake ("../datalake/info") is currently serializing the class 
> > 'DataLakeMeasure' 
> > (org.apache.streampipes.rest.impl.datalake.DataLakeMeasure) in JSON, but it 
> > should be JSONLD.
> > 
> > I tried to change the serialization, but it does not work. I always get the 
> > error:
> > "No identifier was found for 
> > [org.apache.streampipes.model.datalake.DataLakeMeasure@3165a907]!  The 
> > instance should implement Identifiable, have one or more properties 
> > annotated with @RdfId, or have an id function provided to the mapper.". 
> > The 'DataLakeMeasure' extends the 'UnnamedStreamPipesEntity' 
> > (org.apache.streampipes.model.base.UnnamedStreamPipesEntity), which has a 
> > propertety annotated with @RdfId.
> > 
> > Someone has an idea, what could be the problem? See Branch [STREAMPIPES-79]
> > 
> > 
> > Regards
> > Johannes
> > 
> > 
> 
> 
> 

Re: Preparing release

[Johannes Tex]

Hi Dominik,

yeah, thanks for solving the issues :) 

Johannes

On 2020/03/03 22:13:51, Dominik Riemer  wrote: 
> Hi Johannes,
> 
> thanks a lot, that a great overview.
> I already fixed some of the issues you mentioned, it seems we are on a good 
> way towards our first release :-)
> 
> Dominik
> 
> On 2020/02/28 18:05:18, Johannes Tex  wrote: 
> > Hi all,
> > 
> > a few days ago we started thinking about the first Apache release [1]. 
> > So I started to do functional tests: I created some issues that I think we 
> > should fix before the release, I marked the issues with release version 
> > '0.66.0'. You can find them all here: [2].
> > 
> > Regards
> > Johannes
> > 
> > [1] 
> > http://mail-archives.apache.org/mod_mbox/streampipes-dev/202002.mbox/%3C018b01d5e6fd%24842f62b0%248c8e2810%24%40apache.org%3E
> > [2] https://issues.apache.org/jira/projects/STREAMPIPES/versions/12347025
> > 
> > 
> > 
> 

Re: [DISCUSS] Coding guidelines for the UI

[Johannes Tex]

Hi Philipp,

today I activated tslint in my IDE, it is really helpful! Thanks!


Johannes

On 2020/03/05 22:00:19, Philipp Zehnder  wrote: 
> Hi all,
> 
> during the development of UI components I realized that we use many different 
> code styles.
>  
> The project contains a tslint.json file, and my IDE shows errors if there are 
> any style violations.
> My suggestion would be that everybody activates tslint to ensure we all use 
> the same formatting.
> 
> For me this makes the work much easier, because I am often not sure what the 
> cleanest way is in Typescript.
> I do not prefer any particular style as long as it is consistent, so I would 
> suggest that the tslint.json file is adjusted if someone disagrees with the 
> current formatting.
> What do you think about that? Is there a better alternative to tslint or has 
> anyone had a bad experience?
> 
> Philipp

[Help wanted] Jslod Serialization

[Johannes Tex]

Hello everyone,

the REST endpoint for receiving the information about the stored data in the 
data lake ("../datalake/info") is currently serializing the class 
'DataLakeMeasure' (org.apache.streampipes.rest.impl.datalake.DataLakeMeasure) 
in JSON, but it should be JSONLD.

I tried to change the serialization, but it does not work. I always get the 
error:
 "No identifier was found for 
[org.apache.streampipes.model.datalake.DataLakeMeasure@3165a907]!  The instance 
should implement Identifiable, have one or more properties annotated with 
@RdfId, or have an id function provided to the mapper.". 
The 'DataLakeMeasure' extends the 'UnnamedStreamPipesEntity' 
(org.apache.streampipes.model.base.UnnamedStreamPipesEntity), which has a 
propertety annotated with @RdfId.

Someone has an idea, what could be the problem? See Branch [STREAMPIPES-79]


Regards
Johannes

[jira] [Closed] (STREAMPIPES-94) Dashboard without name

[Johannes Tex (Jira)]

 [ 
https://issues.apache.org/jira/browse/STREAMPIPES-94?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johannes Tex closed STREAMPIPES-94.
---

> Dashboard without name
> --
>
> Key: STREAMPIPES-94
> URL: https://issues.apache.org/jira/browse/STREAMPIPES-94
> Project: StreamPipes
>  Issue Type: Bug
>  Components: UI
>        Reporter: Johannes Tex
>    Assignee: Johannes Tex
>Priority: Major
> Fix For: 0.66.0
>
>
> It should not be possible to create a dashboard without a name



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (STREAMPIPES-94) Dashboard without name

[Johannes Tex (Jira)]

 [ 
https://issues.apache.org/jira/browse/STREAMPIPES-94?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johannes Tex resolved STREAMPIPES-94.
-
Resolution: Fixed

> Dashboard without name
> --
>
> Key: STREAMPIPES-94
> URL: https://issues.apache.org/jira/browse/STREAMPIPES-94
> Project: StreamPipes
>  Issue Type: Bug
>  Components: UI
>        Reporter: Johannes Tex
>    Assignee: Johannes Tex
>Priority: Major
> Fix For: 0.66.0
>
>
> It should not be possible to create a dashboard without a name



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Preparing release

[Johannes Tex]

Hi all,

a few days ago we started thinking about the first Apache release [1]. 
So I started to do functional tests: I created some issues that I think we 
should fix before the release, I marked the issues with release version 
'0.66.0'. You can find them all here: [2].

Regards
Johannes

[1] 
http://mail-archives.apache.org/mod_mbox/streampipes-dev/202002.mbox/%3C018b01d5e6fd%24842f62b0%248c8e2810%24%40apache.org%3E
[2] https://issues.apache.org/jira/projects/STREAMPIPES/versions/12347025

[jira] [Created] (STREAMPIPES-99) Upload Adapter Template

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-99:
---

 Summary: Upload Adapter Template
 Key: STREAMPIPES-99
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-99
 Project: StreamPipes
  Issue Type: Bug
  Components: Connect, UI
Reporter: Johannes Tex
 Fix For: 0.66.0


Uploading an adapter template does not work: Uploading is successful, but the 
template is not visible.

Reproduce:
 # Create Adapter Template
 # Download Adapter Template
 # Delete Adapter Template
 # Upload Adapter Template



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (STREAMPIPES-97) Dashboard widget Edit

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-97:
---

 Summary: Dashboard widget Edit
 Key: STREAMPIPES-97
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-97
 Project: StreamPipes
  Issue Type: Bug
  Components: UI
Reporter: Johannes Tex
 Fix For: 0.66.0


When pressing the _Edit_ button of a Dashboard Widget only a blank window opens



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (STREAMPIPES-94) Dashboard without name

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-94:
---

 Summary: Dashboard without name
 Key: STREAMPIPES-94
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-94
 Project: StreamPipes
  Issue Type: Bug
  Components: UI
Reporter: Johannes Tex
 Fix For: 0.66.0


It should not be possible to create a dashboard without a name



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (STREAMPIPES-93) Notification Sink Placeholder

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-93:
---

 Summary: Notification Sink Placeholder
 Key: STREAMPIPES-93
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-93
 Project: StreamPipes
  Issue Type: Bug
  Components: Pipeline Elements
Reporter: Johannes Tex


The notification sink placeholder are not working. #propertyName# are not 
replaced by the value of the property.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (STREAMPIPES-92) Pipeline Log Button

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-92:
---

 Summary: Pipeline Log Button
 Key: STREAMPIPES-92
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-92
 Project: StreamPipes
  Issue Type: Bug
  Components: UI
Reporter: Johannes Tex
 Fix For: 0.66.0


The Pipeline Log button has no functionality



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (STREAMPIPES-91) Quick Edit Parameter Update

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-91:
---

 Summary: Quick Edit Parameter Update
 Key: STREAMPIPES-91
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-91
 Project: StreamPipes
  Issue Type: Bug
  Components: UI
Reporter: Johannes Tex


Element parameters are not saved when edit with _quick edit_, also __ with 
pressing the save button



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (STREAMPIPES-90) Cannot add Pipeline to Category

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-90:
---

 Summary: Cannot add Pipeline to Category
 Key: STREAMPIPES-90
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-90
 Project: StreamPipes
  Issue Type: Bug
  Components: UI
Reporter: Johannes Tex
 Fix For: 0.66.0
 Attachments: Bildschirmfoto 2020-02-28 um 17.58.40.png

Cannot add an existing pipeline to a pipeline catergory (see attachment)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (STREAMPIPES-89) Pipeline Update blank info box

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-89:
---

 Summary: Pipeline Update blank info box
 Key: STREAMPIPES-89
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-89
 Project: StreamPipes
  Issue Type: Bug
  Components: UI
Reporter: Johannes Tex
 Fix For: 0.66.0
 Attachments: Bildschirmfoto 2020-02-28 um 17.54.28.png

When you save an updated pipeline, an empty info box is displayed in the top 
right corner (see attachment).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (STREAMPIPES-88) Pipeline Editor Invalid connection

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-88:
---

 Summary: Pipeline Editor Invalid connection
 Key: STREAMPIPES-88
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-88
 Project: StreamPipes
  Issue Type: Bug
Reporter: Johannes Tex
 Attachments: Bildschirmfoto 2020-02-28 um 17.56.29.png

If elements cannot be connected to each other, no details are available that 
describe the reason for this. (see attachment)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (STREAMPIPES-87) Password field not full Widht

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-87:
---

 Summary: Password field not full Widht
 Key: STREAMPIPES-87
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-87
 Project: StreamPipes
  Issue Type: Bug
  Components: UI
Reporter: Johannes Tex


The passwordfield in Adapter configuration has not full widht (see attachment)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (STREAMPIPES-86) Pipeline cannot be saved

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-86:
---

 Summary: Pipeline cannot be saved
 Key: STREAMPIPES-86
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-86
 Project: StreamPipes
  Issue Type: Bug
  Components: UI
Reporter: Johannes Tex
 Fix For: 0.66.0


Reproduce:
 # Create Pileline
 # Add additional Element
 # Delete Element again

-> Pipeline cannot be saved



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (STREAMPIPES-85) Can't connect Pipeline elements

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-85:
---

 Summary: Can't connect Pipeline elements
 Key: STREAMPIPES-85
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-85
 Project: StreamPipes
  Issue Type: Bug
  Components: UI
Reporter: Johannes Tex
 Fix For: 0.66.0
 Attachments: Bildschirmfoto 2020-02-28 um 17.48.33.png

Reproduce:
 # Create Pileline
 # Leave Editor (e.g. go to Data explorer)
 # Go back to Editor 

-> sometimes elements cannot conntected anymore (see attachment)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (STREAMPIPES-84) Links are out of date

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-84:
---

 Summary: Links are out of date
 Key: STREAMPIPES-84
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-84
 Project: StreamPipes
  Issue Type: Bug
  Components: UI
Reporter: Johannes Tex
 Fix For: 0.66.0


* Links at "User Preferences" -> "Info" should be updated
 * Link "User Preferences" -> "Documentation" is not working



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (STREAMPIPES-83) User Feedback

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-83:
---

 Summary: User Feedback
 Key: STREAMPIPES-83
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-83
 Project: StreamPipes
  Issue Type: Bug
  Components: UI
Reporter: Johannes Tex
 Fix For: 0.66.0


Feedback button (top right) only opens a blank window



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: RE: Image Labeling

[Johannes Tex]

Hi

I created a Wiki page [1].
Everyone is invited to contribute :)

[1] https://cwiki.apache.org/confluence/display/STREAMPIPES/Generic+Data+Store

Johannes


On 2020/02/24 22:54:14, "Dominik Riemer"  wrote: 
> Hi Johannes,
> 
> +1 for having both REST and streaming interfaces to store images, we can 
> probably start with REST and add streaming interfaces later.
> I think the generic data store will be part of the general platform API 
> service, where we can integrate all endpoints that will be required by 
> external services (e.g., registered streams/processors/sinks, historical 
> data, images).
> What do you think, should we create a wiki page to collect all requirements 
> and design the endpoints we are going to need?
> 
> Dominik 
> 
> -Original Message-
> From: Johannes Tex  
> Sent: Sunday, February 23, 2020 11:35 AM
> To: dev@streampipes.apache.org
> Subject: Re: Image Labeling 
> 
> Hi,
> 
> I think a simple generic data store API that support the CRUD operations 
> would be a great.
> 
> I think all CRUD operations should be available as REST API and the CREATE 
> API maybe additionally with a messaging protocol (Kafka, MQTT), which is used 
> e.g. by the Data-Lake-Sink to store the images. Or do you think that 
> synchronous communication via REST is fast enough?
> In addition to reading entire files, the READ operation should have a stream 
> interface that can be used directly by the adapters, for example.
> 
> Johannes
> 
> 
> On 2020/02/21 18:49:58, Philipp Zehnder  wrote: 
> > Hi,
> > 
> > I also think we should store it either in a file, in the same directory as 
> > the image or in the CouchDB.
> > For now I am not sure what the better solution is. The only requirement is 
> > that once a user downloads the data, the labels should be provided in a 
> > Coco-JSON file, but this is possible with both options.
> > 
> > Since we have now multiple locations where we store data, we probably 
> > should start a discussion of how to Store application data within 
> > StreamPipes.
> > It might make sense to have an internal (or external) API for components 
> > and other service.
> > How do you think about that? What kind of features would such an API need?
> > 
> > Philipp
> > 
> > > On 19. Feb 2020, at 22:00, Johannes Tex  wrote:
> > > 
> > > Hi,
> > > 
> > > I starts with @Dominik question: The first Intention was to be part of 
> > > the Data-Explorer, with toggling between simple exploring and labelling. 
> > > @Philipp opened an Issue [STREAMPIPES-79] to refactoring the Data 
> > > explorer, maybe in this context we could extend the data explorer for 
> > > this two modes? 
> > > To display images, for example, we need almost the same mechanism like it 
> > > is necessary for the image labelling, except the Labeling itself. We also 
> > > need to extend the datalake API for images, which leads to @Philipp 
> > > question. 
> > > 
> > > The data lake API supports, at the moment, just data that can be 
> > > aggregated (numeric data). For the Image Labeling and viewing we need to 
> > > extend the API. My proposal would be to create a paging API for images to 
> > > the receive the next e.g. 10 images: It could be like this 
> > > "/datalake/ //". What do you think? While this 
> > > necessary extension we also can create the API to save the annotation.
> > > 
> > > I see three different options to save the annotations:
> > > * Influx -> save annotation direct with data point
> > >- when exporting need to create COCO file
> > >- need extra place to save (image) Labels/Categories
> > >- need to 'manupilate' data point, which is not possible in influx 
> > > (just delete and create new one)
> > > * File
> > > - need to handle a file
> > > * CouchDB
> > >- file generation is needed
> > > My proposal is to use the CouchDB to use the annotations. 
> > > 
> > > Johannes
> > > 
> > > 
> > > On 2020/02/17 21:12:38, Philipp Zehnder  wrote: 
> > >> Hi Johannes,
> > >> 
> > >> as for the API, do you think we can extend the dataset API, or should we 
> > >> create a separate REST API for image annotation?
> > >> 
> > >> Where do you plan to store the coco annotation information? In files or 
> > >> in a DB?
> > >> 
> > >> Philipp
> > >> 
> > >>> On 16. Feb 2020, at 19:51, Domin

Re: Image Labeling

[Johannes Tex]

Hi,

I starts with @Dominik question: The first Intention was to be part of the 
Data-Explorer, with toggling between simple exploring and labelling. @Philipp 
opened an Issue [STREAMPIPES-79] to refactoring the Data explorer, maybe in 
this context we could extend the data explorer for this two modes? 
To display images, for example, we need almost the same mechanism like it is 
necessary for the image labelling, except the Labeling itself. We also need to 
extend the datalake API for images, which leads to @Philipp question. 

The data lake API supports, at the moment, just data that can be aggregated 
(numeric data). For the Image Labeling and viewing we need to extend the API. 
My proposal would be to create a paging API for images to the receive the next 
e.g. 10 images: It could be like this "/datalake/ //". 
What do you think? While this necessary extension we also can create the API to 
save the annotation.

I see three different options to save the annotations:
* Influx -> save annotation direct with data point
- when exporting need to create COCO file
- need extra place to save (image) Labels/Categories
- need to 'manupilate' data point, which is not possible in influx (just 
delete and create new one)
* File
 - need to handle a file
* CouchDB
- file generation is needed
My proposal is to use the CouchDB to use the annotations. 

Johannes


On 2020/02/17 21:12:38, Philipp Zehnder  wrote: 
> Hi Johannes,
> 
> as for the API, do you think we can extend the dataset API, or should we 
> create a separate REST API for image annotation?
> 
> Where do you plan to store the coco annotation information? In files or in a 
> DB?
> 
> Philipp
> 
> > On 16. Feb 2020, at 19:51, Dominik Riemer  wrote:
> > 
> > Hi Johannes,
> > sounds good!
> > I think bounding boxes and polygons are totally fine for the first 
> > prototype.
> > 
> > How to you plan to integrate the labeling tool, will it be part of the data 
> > explorer or do you plan to add a new component?
> > 
> > Dominik
> > 
> > On 2020/02/14 16:30:17, Johannes Tex  wrote: 
> >> Hi,
> >> 
> >> Philip started to extend the datalake sink to store images 
> >> [STREAMPIPES-75]. 
> >> I started now to create an Image labeler that allows users to label images 
> >> in the datalake. [STREAMPIPES-78]. The Labels will be stored in the COCO 
> >> Annonation Format. [1] After labeling, the images can be used to train an 
> >> NN. 
> >> 
> >> The main features that the labeler should support
> >> - Labeling with Bound boxes
> >> - Labeling with Polygons
> >> 
> >> Do you have additional features that should also be supported?
> >> 
> >> Johannes
> >> 
> >> 
> >> [1] http://cocodataset.org/#format-data
> >> 
> >> 
> >> 
> 
> 
> 

Re: RE: STREAMPIPES-75: Extend data lake sink to store images

[Johannes Tex]

Hi,

I also think a service for file handling would be a good solution. 

At the moment we also use files for the Adapters that are stored in the Worker. 
Maybe this would be another use case for a file service?

Johannes 

On 2020/02/19 06:58:11, Dominik Riemer  wrote: 
> Hi Philipp,
> 
> yes, I think it makes sense to have a single service for handling files.
> When writing the CSVMetadataEnrichment component for Chris, I started to add 
> a simple file management to the backend and also extended the SDK with 
> methods to receive files from the backend (see 
> CsvMetadataEnrichmentController and FileServingResource in the backend).
> 
> We could extend this, isolate the file management to an individual 
> microservice and add a simple API in front of it that can be used by all 
> services that require to store or receive files (e.g., also for the included 
> assets of pipeline elements, which could be documentation, icons or ML 
> models).
> 
> Concerning HDFS, in my opinion this might be an option, but as we don't have 
> very large amounts of data by now to store, it would probably be a bit of 
> overkill here (one distributed system more to manage). 
> 
> Dominik
> 
> -Original Message-
> From: Philipp Zehnder  
> Sent: Tuesday, February 18, 2020 6:28 PM
> To: dev@streampipes.apache.org
> Subject: STREAMPIPES-75: Extend data lake sink to store images
> 
> Hi all,
> 
> I finished the implementation to store images in files instead of base 64 
> Strings in InfluxDB.
> 
> For the first version I mounted a local volume and added the images in a 
> folder in this volume. 
> I think this is a good starting point because the images are stored in a 
> local volume on the same host as the sink.
> Now the question is how can users access those images? I would suggest to 
> extend the data lake REST API for that.
> Therefore, the backend must mount the same volume as the internal sink 
> container with the data lake sink.
> 
> Does anyone of you have an alternative solution?
> 
> @Dominik, you implemented already an StreamPipes internal file storage. Could 
> we use that for the images as well or would the frequency be too high?
> 
> @all What about HDFS. We could set up HDFS, for files. Similar to InfluxDB as 
> a shared service between multiple containers
> 
> 
> Philipp
> 

Image Labeling

[Johannes Tex]

Hi,

Philip started to extend the datalake sink to store images [STREAMPIPES-75]. 
I started now to create an Image labeler that allows users to label images in 
the datalake. [STREAMPIPES-78]. The Labels will be stored in the COCO 
Annonation Format. [1] After labeling, the images can be used to train an NN. 

The main features that the labeler should support
- Labeling with Bound boxes
- Labeling with Polygons

Do you have additional features that should also be supported?

Johannes


[1] http://cocodataset.org/#format-data

[jira] [Created] (STREAMPIPES-78) Image Labeling

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-78:
---

 Summary: Image Labeling
 Key: STREAMPIPES-78
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-78
 Project: StreamPipes
  Issue Type: New Feature
  Components: Backend, UI
Reporter: Johannes Tex
Assignee: Johannes Tex


Development of an image labeling tool to label the stored images from the 
Datalake. The Labels will be stored in the _COCO Annonation_ Format. [1]
 * Bouding Box Labeling
 * Polygon Labeling

[1] [http://cocodataset.org/#format-data]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Refactor MQTT based adapters

[Johannes Tex]

Hi,

this is not the best solution but I think there is no other way at the moment. 
In my opinion it would be great if we could create a strings.en file specific 
for the MQTTUtil and all Adapters that use MQTTUtil import this file addition 
to the Adapter specific string files.

For now I created Assets for all MQTT Adapters and the Procotol (See Branch: 
netio).

P.S. I found a bug to get the protocol assets (e.g. the icons of protocols were 
not displayed when coming from the backend), but this is fixed now.

Johannes


On 2020/02/04 21:39:40, Philipp Zehnder  wrote: 
> Hi,
> 
> Ah ok. But then we have the problem I described in Issue [1] in the comments 
> field.
> For your solution we have to edit all the strings.en files that are using the 
> MQTT configurations. I think this solution is very error prone, but for now I 
> do not have a better solution so it would be great if you could commit your 
> changes.
> 
> What I do not understand is, why the TISensorTag adapter worked with the 
> Labels.from version of the MQTT Utils class?
> 
> 
> 
> 
> [1] 
> https://issues.apache.org/jira/projects/STREAMPIPES/issues/STREAMPIPES-69?filter=allopenissues=created+DESC%2C+priority+DESC%2C+updated+DESC
>  
> <https://issues.apache.org/jira/projects/STREAMPIPES/issues/STREAMPIPES-69?filter=allopenissues=created+DESC,+priority+DESC,+updated+DESC>
> 
> > On 4. Feb 2020, at 22:30, Johannes Tex  wrote:
> > 
> > Hi Philipp,
> > 
> > yes, on my computer it is working. 
> > I changed in the MQTT utils class to Labels.withId and added the string in 
> > the string.en file.
> > If you like i can commit it and you can try at your machine.
> > 
> > Johannes
> > 
> > On 2020/02/04 21:10:01, Philipp Zehnder  wrote: 
> >> Hi,
> >> 
> >> @Johannes: Does it work on your machine? It still does not work on my 
> >> computer. 
> >> The commit you mentioned was to fix a bug that occurred when the component 
> >> started. The problem is that the labels are not shown in the UI. See the 
> >> attached screenshot.
> >> 
> >> 
> >> 
> >>> On 4. Feb 2020, at 21:20, Dominik Riemer  wrote:
> >>> 
> >>> Great!
> >>> So I guess this was just Philipp's way to ask for a code review :D
> >>> 
> >>> -Original Message-
> >>> From: Johannes Tex  
> >>> Sent: Tuesday, February 4, 2020 9:16 PM
> >>> To: dev@streampipes.apache.org
> >>> Subject: Re: Refactor MQTT based adapters
> >>> 
> >>> Hi Philipp
> >>> 
> >>> I took a quick look and tried to use the NetioMQTTAdapter with 
> >>> "Labels.withId". It has worked as expected. I guess you already fixed it 
> >>> with your commit  "Add null check for alternative labels in label 
> >>> generator" :) 
> >>> 
> >>> Johannes
> >>> 
> >>> On 2020/02/04 16:00:07, Philipp Zehnder  wrote: 
> >>>> Hi all,
> >>>> 
> >>>> since we have multiple adapters that use MQTT I created a utils class 
> >>>> with all the code required for the configuration parameters.
> >>>> One of the new adapters is the NetioMQTTAdapter. The adapter itself 
> >>>> works, but the labels are not shown in the UI.
> >>>> 
> >>>> Maybe someone knows what the problem could be? 
> >>>> It works when I use the old labels with Labels.from. 
> >>>> When I use the new labels with Labels.withId it does not work. I think 
> >>>> the problem is with the alternative static properties, since it also 
> >>>> works when I remove them from the adapter description.
> >>>> Please write if anyone has a hint what the problem might be.
> >>>> 
> >>>> 
> >>>> Philipp
> >>>> 
> >>>> 
> >>>> 
> >>> 
> >> 
> >> 
> 
> 

Re: Refactor MQTT based adapters

[Johannes Tex]

Hi Philipp,

yes, on my computer it is working. 
I changed in the MQTT utils class to Labels.withId and added the string in the 
string.en file.
If you like i can commit it and you can try at your machine.

Johannes

On 2020/02/04 21:10:01, Philipp Zehnder  wrote: 
> Hi,
> 
> @Johannes: Does it work on your machine? It still does not work on my 
> computer. 
> The commit you mentioned was to fix a bug that occurred when the component 
> started. The problem is that the labels are not shown in the UI. See the 
> attached screenshot.
> 
> 
> 
> > On 4. Feb 2020, at 21:20, Dominik Riemer  wrote:
> > 
> > Great!
> > So I guess this was just Philipp's way to ask for a code review :D
> > 
> > -Original Message-
> > From: Johannes Tex  
> > Sent: Tuesday, February 4, 2020 9:16 PM
> > To: dev@streampipes.apache.org
> > Subject: Re: Refactor MQTT based adapters
> > 
> > Hi Philipp
> > 
> > I took a quick look and tried to use the NetioMQTTAdapter with 
> > "Labels.withId". It has worked as expected. I guess you already fixed it 
> > with your commit  "Add null check for alternative labels in label 
> > generator" :) 
> > 
> > Johannes
> > 
> > On 2020/02/04 16:00:07, Philipp Zehnder  wrote: 
> >> Hi all,
> >> 
> >> since we have multiple adapters that use MQTT I created a utils class with 
> >> all the code required for the configuration parameters.
> >> One of the new adapters is the NetioMQTTAdapter. The adapter itself works, 
> >> but the labels are not shown in the UI.
> >> 
> >> Maybe someone knows what the problem could be? 
> >> It works when I use the old labels with Labels.from. 
> >> When I use the new labels with Labels.withId it does not work. I think the 
> >> problem is with the alternative static properties, since it also works 
> >> when I remove them from the adapter description.
> >> Please write if anyone has a hint what the problem might be.
> >> 
> >> 
> >> Philipp
> >> 
> >> 
> >> 
> > 
> 
> 

Re: Refactor MQTT based adapters

[Johannes Tex]

Hi Philipp

I took a quick look and tried to use the NetioMQTTAdapter with "Labels.withId". 
It has worked as expected. I guess you already fixed it with your commit  "Add 
null check for alternative labels in label generator" :) 

Johannes

On 2020/02/04 16:00:07, Philipp Zehnder  wrote: 
> Hi all,
> 
> since we have multiple adapters that use MQTT I created a utils class with 
> all the code required for the configuration parameters.
> One of the new adapters is the NetioMQTTAdapter. The adapter itself works, 
> but the labels are not shown in the UI.
> 
> Maybe someone knows what the problem could be? 
> It works when I use the old labels with Labels.from. 
> When I use the new labels with Labels.withId it does not work. I think the 
> problem is with the alternative static properties, since it also works when I 
> remove them from the adapter description.
> Please write if anyone has a hint what the problem might be.
> 
> 
> Philipp
> 
> 
> 

[jira] [Closed] (STREAMPIPES-17) Expose Adapter Icons in Connect Container

[Johannes Tex (Jira)]

 [ 
https://issues.apache.org/jira/browse/STREAMPIPES-17?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johannes Tex closed STREAMPIPES-17.
---
Resolution: Implemented

> Expose Adapter Icons in Connect Container
> -
>
> Key: STREAMPIPES-17
> URL: https://issues.apache.org/jira/browse/STREAMPIPES-17
> Project: StreamPipes
>  Issue Type: New Feature
>  Components: Connect
>Reporter: Dominik Riemer
>    Assignee: Johannes Tex
>Priority: Major
>
> Icons and others assets should be managed similar to pipeline elements in an 
> asset folder.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[STREAMPIPES-17] Expose Adapter Icons in Connect Container

[Johannes Tex]

Hi,

connect adapter and procotols can now use the asset folder as it is done for 
the pipeline elements. To support this, we changed the IDs of the 
adapter/protocol to the package names, like the IDs of the pipeline elements. 
Note this when you create a new adapter/protocol. Also the 'local' string file 
is now supported by adapters/protocols. 

Assets are created for the ROS Adapter. For all others it still has to be done. 
Maybe it would be a nice task for a newbie to get familiar with the SDK?

Johannes

[jira] [Closed] (STREAMPIPES-22) Support for multiple properties

[Johannes Tex (Jira)]

 [ 
https://issues.apache.org/jira/browse/STREAMPIPES-22?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johannes Tex closed STREAMPIPES-22.
---
Resolution: Implemented

!image-2020-01-09-19-11-51-773.png!

> Support for multiple properties
> ---
>
> Key: STREAMPIPES-22
> URL: https://issues.apache.org/jira/browse/STREAMPIPES-22
> Project: StreamPipes
>  Issue Type: New Feature
>  Components: Connect
>        Reporter: Johannes Tex
>    Assignee: Johannes Tex
>Priority: Normal
> Attachments: image-2019-12-08-11-36-36-003.png
>
>
> According to dev Email: "[Add support for multiple properties in PLC4X S7 
> Adapter|https://lists.apache.org/thread.html/38a7621c664ce80ddb932a8b6c21fb623a2f7f771f7a3443fc03cfe4%40%3Cdev.streampipes.apache.org%3E];



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: AW: Add support for multiple properties in PLC4X S7 Adapter

[Johannes Tex]

Hi,

support for multiple properties (CollectionProperty) has been implemented. I 
also added the horizontal rendering property for StaticPropertyGroup and 
SelectionStaticProperty. Currently all SelectionStaticProperty will be renderd 
horizontal (Drop down menu), because we had problems with deserialization (UI) 
of the horizontal property if a SelectionStaticProperty is part of a 
StaticPropertyGroup. Maybe someone knows what the problem could be? (I opened 
an Issue STREAMPIPES-54)

However the CollectionProperty is already used by the ‘PLC4X S7’ Adapter. Your 
feedback is welcome :) 

Johannes

On 2019/11/21 07:15:45, "Johannes Tex"  wrote: 
> Hi,
> 
> I have started to implement the new feature!
> 
> At the StaticPropertyGroup Property I will add a new boolean property
> *horizontalRendering*. I also will do this for the OneOfStaticProperty
> (Single-Value-Selection), in this case horizontal rendering means that a
> dropdown is displayed instead of a vertical list of radio buttons.
> 
> 
> Johannes
> 
> -Ursprüngliche Nachricht-
> Von: Dominik Riemer [mailto:rie...@fzi.de] 
> Gesendet: Samstag, 16. November 2019 13:15
> An: dev@streampipes.apache.org
> Betreff: Re: Add support for multiple properties in PLC4X S7 Adapter
> 
> Hi Philipp,
> 
> 
> 
> that's looking great!
> 
> To implement your mockup, we need to extend the StaticPropertyGroup to allow
> horizontal rendering of group members. We can add this along with a some
> optimizations to the SDK's methods to create and extract static properties
> (Chris also mentioned that complex nested static property definitions can
> become quite complicated to model if you're not used to it).
> 
> 
> 
> Dominik
> 
> 
> 
> On 2019/11/16 09:48:24, Philipp Zehnder
> mailto:zehn...@fzi.de>> wrote:
> 
> > Hi,
> 
> >
> 
> >
> 
> > the first version of the of the PLC4X S7 adapter just supports to
> configure one item.
> 
> > A user provides the IP address, the runtimeName of the property, and the
> node type as shown in the image:
> 
> >
> 
> >
> 
> > In the next version it should be possible to select multiple items, like:
> 
> >
> 
> >
> 
> >
> 
> >
> 
> > To do that we have to extend the Typescript model to support
> CollectionStaticProperty and we also have to implement the UI component for
> that.
> 
> >
> 
> > Is it possible to have a StaticPropertyGroup in this collection that
> contains the Static properties for runtime name, node and data type?
> 
> >
> 
> > Does anyone have another solution how to display the configurations to the
> user or are there any other configurations required?
> 
> >
> 
> > Cheers,
> 
> > Philipp
> 
> >
> 
> >
> 
> >
> 
> >
> 
> >
> 
> >
> 
> >
> 
> >
> 
> >
> 
> >
> 
> >
> 
> >
> 
> >
> 
> 

[jira] [Created] (STREAMPIPES-54) Deserialization horizontal property of SelectionProperty if in Group

[Johannes Tex (Jira)]

Johannes Tex created STREAMPIPES-54:
---

 Summary: Deserialization horizontal property of SelectionProperty 
if in Group
 Key: STREAMPIPES-54
 URL: https://issues.apache.org/jira/browse/STREAMPIPES-54
 Project: StreamPipes
  Issue Type: Bug
  Components: UI
Reporter: Johannes Tex


The horizontal rendering property of the SelectionStaticProperty is not 
deserialized in the UI if it is part of the StaticPropertyGroup. Currently the 
SelectionStaticProperty is always rendered horizontally because it is set in 
the ui (see: SelectionStaticProperty.ts).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Confluence wiki for StreamPipes

[Johannes Tex]

+1

On 2019/12/02 16:23:57, Philipp Zehnder  wrote: 
> +1
> 
> > On 2. Dec 2019, at 16:53, wiener  wrote:
> > 
> > Sounds like a good idea.
> > 
> > +1
> > 
> > On 2019/12/02 13:32:03, "Dominik Riemer"  wrote: 
> >> Hi,> 
> >> 
> >>> 
> >> 
> >> following up the discussion thread [1], I'd like to call a vote to decide> 
> >> whether we want to have a Confluence wiki to manage some 
> >> developer-oriented> 
> >> knowledge (as mentioned in the DISCUSS thread).> 
> >> 
> >>> 
> >> 
> >> Please cast your vote:> 
> >> 
> >>> 
> >> 
> >>  [ ] +1, create a Confluence wiki for StreamPipes> 
> >> 
> >>  [ ] +0, I don't care either way> 
> >> 
> >>  [ ] -1, do not create a Confluence wiki, because.> 
> >> 
> >>> 
> >> 
> >> Voting will be open for at least 72 hours.> 
> >> 
> >>> 
> >> 
> >> Dominik> 
> >> 
> >>> 
> >> 
> >>> 
> >> 
> >> [1]> 
> >> 
> >>  
> >> 5f8e3f30495be5bde0@%3Cdev.streampipes.apache.org%3E>> 
> >> https://lists.apache.org/thread.html/93ce66501bc87636f319ef0dfedfedaf76d41e5>
> >>  
> >> f8e3f30495be5bde0@%3Cdev.streampipes.apache.org%3E> 
> >> 
> >> 
> > 
> > 
> 
> 
> 

AW: Add support for multiple properties in PLC4X S7 Adapter

[Johannes Tex]

Hi,

I have started to implement the new feature!

At the StaticPropertyGroup Property I will add a new boolean property
*horizontalRendering*. I also will do this for the OneOfStaticProperty
(Single-Value-Selection), in this case horizontal rendering means that a
dropdown is displayed instead of a vertical list of radio buttons.


Johannes

-Ursprüngliche Nachricht-
Von: Dominik Riemer [mailto:rie...@fzi.de] 
Gesendet: Samstag, 16. November 2019 13:15
An: dev@streampipes.apache.org
Betreff: Re: Add support for multiple properties in PLC4X S7 Adapter

Hi Philipp,



that's looking great!

To implement your mockup, we need to extend the StaticPropertyGroup to allow
horizontal rendering of group members. We can add this along with a some
optimizations to the SDK's methods to create and extract static properties
(Chris also mentioned that complex nested static property definitions can
become quite complicated to model if you're not used to it).



Dominik



On 2019/11/16 09:48:24, Philipp Zehnder
mailto:zehn...@fzi.de>> wrote:

> Hi,

>

>

> the first version of the of the PLC4X S7 adapter just supports to
configure one item.

> A user provides the IP address, the runtimeName of the property, and the
node type as shown in the image:

>

>

> In the next version it should be possible to select multiple items, like:

>

>

>

>

> To do that we have to extend the Typescript model to support
CollectionStaticProperty and we also have to implement the UI component for
that.

>

> Is it possible to have a StaticPropertyGroup in this collection that
contains the Static properties for runtime name, node and data type?

>

> Does anyone have another solution how to display the configurations to the
user or are there any other configurations required?

>

> Cheers,

> Philipp

>

>

>

>

>

>

>

>

>

>

>

>

>

AW: AW: Run and develop StreamPipes without configuring the IP address

[Johannes Tex]

Hi,

in regarding to the SP_DATA_LOCATION configuration of the connect-adapters:
It also can be removed and the path can be set automatically if the adapter
is running in debug mode. 

The only disadvantage is that the users cannot simply change the file
locations of the uploaded files. Are there scenarios where this would be
necessary?

Johannes

-Ursprüngliche Nachricht-
Von: Dominik Riemer [mailto:rie...@fzi.de] 
Gesendet: Samstag, 16. November 2019 13:26
An: dev@streampipes.apache.org
Betreff: Re: AW: Run and develop StreamPipes without configuring the IP
address

Hi,

cool, the config for pipeline elements looks good and is *much* easier to
understand.

For connect-adapters, can't we also remove SP_KAFKA_HOST (can be
automatically set in debug mode similar to pipeline elements)? What about
SP_DATA_LOCATION?

For the backend config, we need to remove Kafka-Rest anyways (Confluent
Community License) and AFAIK could also remove uri (RDF4J is now managed
internally) and Zookeeper_HOST (not longer needed).
Let's do some tests whether we can also just set a DEBUG flag when
developing on the backend that automatically sets KAFKA_HOST, COUCHDB_HOST
and JMS_HOST to localhost.

Dominik


On 2019/11/16 09:31:31, "Johannes Tex"  wrote:
> Hi,
>
>
>
> I think we don't need any more these configurations for the backend :
>
> -SP_DATALAKE_HOST
>
> -SP_DATALAKE_PORT
>
>
>
> It was used for the first version of the data-lake with Elasticsearch , we
> changed the data-lake database to Influx and removed the old data-lake API
> with Elasticsearch last week.
>
>
>
> Furthermore I think it would be useful to add the port configuration for
> Influx:
>
> SP_INFLUX_HOST=8086
>
>
>
>
>
> Regards
>
> Johannes
>
>
>
>
>
>
>
> Von: Philipp Zehnder [mailto:zehn...@fzi.de]
> Gesendet: Samstag, 16. November 2019 09:57
> An: dev@streampipes.apache.org
> Betreff: Run and develop StreamPipes without configuring the IP address
>
>
>
> Hi all,
>
>
>
> we removed the requirement to define the servers IP address when
StreamPipes
> is run completely in Docker.
>
> However, we still have to change several configs to ensure the new setting
> also works during development.
>
>
>
> This change effects the following projects:
>
>
>
> * streampipes
>
>   * Do we still need all of those configurations for the backend?
>
> SP_COUCHDB_HOST=localhost
> uri=http://localhost:8031/rdf4j-server
> SP_KAFKA_HOST=host.docker.internal
> SP_ZOOKEEPER_HOST=host.docker.internal
> SP_JMS_HOST=host.docker.internal
> SP_KAFKA_REST_HOST=host.docker.internal
> SP_KAFKA_REST_PORT=8073
> SP_BACKEND_HOST=host.docker.internal
> SP_ELASTICSEARCH_HOST=host.docker.internal
> SP_DATALAKE_HOST=host.docker.internal
> SP_DATALAKE_PORT=9200
> SP_INFLUX_HOST=host.docker.internal
>
>
>
> * streampipes-pipeline-elements
>
>   * Suggestion of an example configuration for processing containers:
>
>
>
> SP_PORT=6060
> SP_HOST=host.docker.internal
> SP_DEBUG=true
>
>
>
> * streampipes-connect-adapters
>
>   * Suggestion of an example configuration for connect adapters
>
>
>
> SP_KAFKA_HOST=localhost
>
> SP_CONNECT_CONTAINER_HOST=localhost
>
> SP_CONNECT_CONTAINER_MASTER_HOST=localhost
>
> SP_CONNECT_CONTAINER_WORKER_HOST=localhost
>
> SP_DATA_LOCATION=./test_data/
>
> SP_DEBUG=true
>
>
>
>
>
> Did I forget any configurations we have to change?
>
>
>
> I am still not completely certain when to use localhost and when to use
> host.docker.internal during development. Currently I added
> host.docker.internal to my /etc/hosts file.
>
> Is there a way to remove this entry?
>
>
>
> Additionally I suggest to remove all deployment folders (containing a
> docker-compose.yml file) in the projects and manage them centrally in the
> cli project.
>
>
>
> Cheers,
>
> Philipp
>
>

AW: Run and develop StreamPipes without configuring the IP address

[Johannes Tex]

Hi,



I think we don't need any more these configurations for the backend :

-SP_DATALAKE_HOST

-SP_DATALAKE_PORT



It was used for the first version of the data-lake with Elasticsearch , we
changed the data-lake database to Influx and removed the old data-lake API
with Elasticsearch last week.



Furthermore I think it would be useful to add the port configuration for
Influx:

SP_INFLUX_HOST=8086





Regards

Johannes







Von: Philipp Zehnder [mailto:zehn...@fzi.de]
Gesendet: Samstag, 16. November 2019 09:57
An: dev@streampipes.apache.org
Betreff: Run and develop StreamPipes without configuring the IP address



Hi all,



we removed the requirement to define the servers IP address when StreamPipes
is run completely in Docker.

However, we still have to change several configs to ensure the new setting
also works during development.



This change effects the following projects:



* streampipes

  * Do we still need all of those configurations for the backend?

SP_COUCHDB_HOST=localhost
uri=http://localhost:8031/rdf4j-server
SP_KAFKA_HOST=host.docker.internal
SP_ZOOKEEPER_HOST=host.docker.internal
SP_JMS_HOST=host.docker.internal
SP_KAFKA_REST_HOST=host.docker.internal
SP_KAFKA_REST_PORT=8073
SP_BACKEND_HOST=host.docker.internal
SP_ELASTICSEARCH_HOST=host.docker.internal
SP_DATALAKE_HOST=host.docker.internal
SP_DATALAKE_PORT=9200
SP_INFLUX_HOST=host.docker.internal



* streampipes-pipeline-elements

  * Suggestion of an example configuration for processing containers:



SP_PORT=6060
SP_HOST=host.docker.internal
SP_DEBUG=true



* streampipes-connect-adapters

  * Suggestion of an example configuration for connect adapters



SP_KAFKA_HOST=localhost

SP_CONNECT_CONTAINER_HOST=localhost

SP_CONNECT_CONTAINER_MASTER_HOST=localhost

SP_CONNECT_CONTAINER_WORKER_HOST=localhost

SP_DATA_LOCATION=./test_data/

SP_DEBUG=true





Did I forget any configurations we have to change?



I am still not completely certain when to use localhost and when to use
host.docker.internal during development. Currently I added
host.docker.internal to my /etc/hosts file.

Is there a way to remove this entry?



Additionally I suggest to remove all deployment folders (containing a
docker-compose.yml file) in the projects and manage them centrally in the
cli project.



Cheers,

Philipp