[GitHub] [incubator-streampipes-extensions] bossenti opened a new pull request #27: Fix Modbus Adapter

[GitBox]

bossenti opened a new pull request #27:
URL: https://github.com/apache/incubator-streampipes-extensions/pull/27


   @tenthe fulfilled ToDo



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Adding StreamPipes Python wrapper

[Philipp Zehnder]

I totally agree, from the user point of view (PE developer) we need a native 
python solution.
The question are, how can we realize that and how can we start.

I have one addition to your list:
* Support configuration parameters in the consul key-value store



> On 17. Jul 2020, at 09:17, Patrick Wiener  wrote:
> 
> Idk yet either. However, lastly all a processor really is, is a microservice 
> with an API, some boilerplate code and the logic iteself put in inside a 
> docker container.
> Thus we get a REST request from the backend including configurations to start 
> the given processor and a REST request from the backend to stop it.
> 
> So what would we need?
> 
> * a way for this service to register itself in consul
> * a way for it to describe itself to the backend (declareModel)
> * a way to consume/produce events (maybe in the first step with a fixed 
> protocol such as kafka)
> 
> anything else that I am missing?
> 
> So, yes we could potentially add is as it is as part of the extensions 
> project. However, in the long rung, there will be base implementations of the 
> above mentioned
> points that are more suitable as part of the core, rather than on the 
> „extension“ side - that’s my only concern right now as the current prototype 
> is really nothing more
> than a prototype.
> 
> Maybe we should think of a way of how we could not reimplement everything in 
> Python but rather extending the backend side in a way that makes it easier to 
> integrate
> it in general.
> 
> So I am open
> 
> Patrick
> 
> 
>> Am 17.07.2020 um 08:18 schrieb Philipp Zehnder :
>> 
>> Hi,
>> 
>> I still don't quite understand how we can integrate the Python code into the 
>> backend module. 
>> Because then we would have to put  the Python processor implementation in 
>> the backend as well, right? 
>> Or how can we use the Python dependencies in the extensions module?
>> 
>> Currently we use ExternalEventProcessor to trigger the invocation of a 
>> processor. Then a REST request is send from java to python.
>> Maybe we can add some of the java boilerplate code into the 
>> streampipes-wrapper-python module. But how about the python REST API?
>> 
>> Philipp
>> 
>>> On 16. Jul 2020, at 23:29, Dominik Riemer  wrote:
>>> 
>>> Hi,
>>> 
>>> I'm fully +1 for a complete, plain python wrapper that integrates both 
>>> runtime and controller interfaces! Also, given our microservice 
>>> architecture with standalone pipeline elements that communicate over 
>>> JSON/JSON-LD I don't think we need any code-level integration between 
>>> Python and Java. 
>>> 
>>> Concerning the code structure, I'd suggest to create a 
>>> streampipes-wrapper-python module in the core project, add the Python code 
>>> there and to create an example using the current Java 
>>> ExternalEventProcessor into the streampipes-examples project that explains 
>>> how to use the Python wrapper. By adding the Python code to the core 
>>> project, all wrappers would be located in the same repository, while the 
>>> extensions project solely provides specific pipeline elements and adapters.
>>> In the meantime, we could add the missing features to the Python wrapper. I 
>>> agree that it is some work, but it should mainly consist of parsing the 
>>> graphs (we could use JSON instead of JSON-LD here to simplify parsing), 
>>> extracting parameters and adding some Flask endpoints.
>>> 
>>> As I'm not that familiar with Python, are there any Python experts on the 
>>> list who want to help building the wrapper? I'd expect that finishing the 
>>> wrapper could probably be done within a few days if there is a Python 
>>> expert and someone who is familiar with the StreamPipes model - I'd be 
>>> happy to support the model side 
>>> 
>>> Dominik 
>>> 
>>> 
>>> -Original Message-
>>> From: Philipp Zehnder  
>>> Sent: Thursday, July 16, 2020 11:09 PM
>>> To: dev@streampipes.apache.org
>>> Subject: Re: Adding StreamPipes Python wrapper
>>> 
>>> Hi guys,
>>> 
>>> I am also in favor of integrating the current prototype of the python 
>>> wrapper for further development.
>>> I would also like to discuss how the proper integration might look like. 
>>> The cleanest way would indeed be to implement all the StreamPipes 
>>> interfaces and models in python, but I fear this is a lot of work and will 
>>> take quite some time.
>>> Is there a better way, or does anyone have experience integrating Python 
>>> code into Java?
>>> 
>>> As for the first integration I would suggest to create a module in the 
>>> extensions project and put all the code there. 
>>> We currently use the interfaces of the Java wrapper, right? So we do not 
>>> have any python specific endpoints.
>>> I think this would ease the usage for people in the community and already 
>>> try an early version of the wrapper.
>>> Alternatively, we can put it into the core in streampipes-wrapper-python as 
>>> you suggested, but then a user has to checkout the backend and the 
>>> extensions project to develop 

Re: Adding StreamPipes Python wrapper

[Patrick Wiener]

Idk yet either. However, lastly all a processor really is, is a microservice 
with an API, some boilerplate code and the logic iteself put in inside a docker 
container.
Thus we get a REST request from the backend including configurations to start 
the given processor and a REST request from the backend to stop it.

So what would we need?

* a way for this service to register itself in consul
* a way for it to describe itself to the backend (declareModel)
* a way to consume/produce events (maybe in the first step with a fixed 
protocol such as kafka)

anything else that I am missing?

So, yes we could potentially add is as it is as part of the extensions project. 
However, in the long rung, there will be base implementations of the above 
mentioned
points that are more suitable as part of the core, rather than on the 
„extension“ side - that’s my only concern right now as the current prototype is 
really nothing more
than a prototype.

Maybe we should think of a way of how we could not reimplement everything in 
Python but rather extending the backend side in a way that makes it easier to 
integrate
it in general.

So I am open

Patrick


> Am 17.07.2020 um 08:18 schrieb Philipp Zehnder :
> 
> Hi,
> 
> I still don't quite understand how we can integrate the Python code into the 
> backend module. 
> Because then we would have to put  the Python processor implementation in the 
> backend as well, right? 
> Or how can we use the Python dependencies in the extensions module?
> 
> Currently we use ExternalEventProcessor to trigger the invocation of a 
> processor. Then a REST request is send from java to python.
> Maybe we can add some of the java boilerplate code into the 
> streampipes-wrapper-python module. But how about the python REST API?
> 
> Philipp
> 
>> On 16. Jul 2020, at 23:29, Dominik Riemer  wrote:
>> 
>> Hi,
>> 
>> I'm fully +1 for a complete, plain python wrapper that integrates both 
>> runtime and controller interfaces! Also, given our microservice architecture 
>> with standalone pipeline elements that communicate over JSON/JSON-LD I don't 
>> think we need any code-level integration between Python and Java. 
>> 
>> Concerning the code structure, I'd suggest to create a 
>> streampipes-wrapper-python module in the core project, add the Python code 
>> there and to create an example using the current Java ExternalEventProcessor 
>> into the streampipes-examples project that explains how to use the Python 
>> wrapper. By adding the Python code to the core project, all wrappers would 
>> be located in the same repository, while the extensions project solely 
>> provides specific pipeline elements and adapters.
>> In the meantime, we could add the missing features to the Python wrapper. I 
>> agree that it is some work, but it should mainly consist of parsing the 
>> graphs (we could use JSON instead of JSON-LD here to simplify parsing), 
>> extracting parameters and adding some Flask endpoints.
>> 
>> As I'm not that familiar with Python, are there any Python experts on the 
>> list who want to help building the wrapper? I'd expect that finishing the 
>> wrapper could probably be done within a few days if there is a Python expert 
>> and someone who is familiar with the StreamPipes model - I'd be happy to 
>> support the model side 
>> 
>> Dominik 
>> 
>> 
>> -Original Message-
>> From: Philipp Zehnder  
>> Sent: Thursday, July 16, 2020 11:09 PM
>> To: dev@streampipes.apache.org
>> Subject: Re: Adding StreamPipes Python wrapper
>> 
>> Hi guys,
>> 
>> I am also in favor of integrating the current prototype of the python 
>> wrapper for further development.
>> I would also like to discuss how the proper integration might look like. 
>> The cleanest way would indeed be to implement all the StreamPipes interfaces 
>> and models in python, but I fear this is a lot of work and will take quite 
>> some time.
>> Is there a better way, or does anyone have experience integrating Python 
>> code into Java?
>> 
>> As for the first integration I would suggest to create a module in the 
>> extensions project and put all the code there. 
>> We currently use the interfaces of the Java wrapper, right? So we do not 
>> have any python specific endpoints.
>> I think this would ease the usage for people in the community and already 
>> try an early version of the wrapper.
>> Alternatively, we can put it into the core in streampipes-wrapper-python as 
>> you suggested, but then a user has to checkout the backend and the 
>> extensions project to develop a new processor.
>> Whats your opinion on that?
>> 
>> Philipp
>> 
>> 
>> 
>>> On 16. Jul 2020, at 20:41, Patrick Wiener  wrote:
>>> 
>>> Hi Grainier,
>>> 
>>> Definitely, it should make it super simple to integrate various well 
>>> known Python libs. The only real limitation is that they’ll also have 
>>> to work in an event-driven fashion.
>>> 
>>> I guess the most clean way would be to port the Java wrapper to Python 
>>> to finally have something such 

Re: Adding StreamPipes Python wrapper

[Philipp Zehnder]

Hi,

I still don't quite understand how we can integrate the Python code into the 
backend module. 
Because then we would have to put  the Python processor implementation in the 
backend as well, right? 
Or how can we use the Python dependencies in the extensions module?

Currently we use ExternalEventProcessor to trigger the invocation of a 
processor. Then a REST request is send from java to python.
Maybe we can add some of the java boilerplate code into the 
streampipes-wrapper-python module. But how about the python REST API?

Philipp

> On 16. Jul 2020, at 23:29, Dominik Riemer  wrote:
> 
> Hi,
> 
> I'm fully +1 for a complete, plain python wrapper that integrates both 
> runtime and controller interfaces! Also, given our microservice architecture 
> with standalone pipeline elements that communicate over JSON/JSON-LD I don't 
> think we need any code-level integration between Python and Java. 
> 
> Concerning the code structure, I'd suggest to create a 
> streampipes-wrapper-python module in the core project, add the Python code 
> there and to create an example using the current Java ExternalEventProcessor 
> into the streampipes-examples project that explains how to use the Python 
> wrapper. By adding the Python code to the core project, all wrappers would be 
> located in the same repository, while the extensions project solely provides 
> specific pipeline elements and adapters.
> In the meantime, we could add the missing features to the Python wrapper. I 
> agree that it is some work, but it should mainly consist of parsing the 
> graphs (we could use JSON instead of JSON-LD here to simplify parsing), 
> extracting parameters and adding some Flask endpoints.
> 
> As I'm not that familiar with Python, are there any Python experts on the 
> list who want to help building the wrapper? I'd expect that finishing the 
> wrapper could probably be done within a few days if there is a Python expert 
> and someone who is familiar with the StreamPipes model - I'd be happy to 
> support the model side 
> 
> Dominik 
> 
> 
> -Original Message-
> From: Philipp Zehnder  
> Sent: Thursday, July 16, 2020 11:09 PM
> To: dev@streampipes.apache.org
> Subject: Re: Adding StreamPipes Python wrapper
> 
> Hi guys,
> 
> I am also in favor of integrating the current prototype of the python wrapper 
> for further development.
> I would also like to discuss how the proper integration might look like. 
> The cleanest way would indeed be to implement all the StreamPipes interfaces 
> and models in python, but I fear this is a lot of work and will take quite 
> some time.
> Is there a better way, or does anyone have experience integrating Python code 
> into Java?
> 
> As for the first integration I would suggest to create a module in the 
> extensions project and put all the code there. 
> We currently use the interfaces of the Java wrapper, right? So we do not have 
> any python specific endpoints.
> I think this would ease the usage for people in the community and already try 
> an early version of the wrapper.
> Alternatively, we can put it into the core in streampipes-wrapper-python as 
> you suggested, but then a user has to checkout the backend and the extensions 
> project to develop a new processor.
> Whats your opinion on that?
> 
> Philipp
> 
> 
> 
>> On 16. Jul 2020, at 20:41, Patrick Wiener  wrote:
>> 
>> Hi Grainier,
>> 
>> Definitely, it should make it super simple to integrate various well 
>> known Python libs. The only real limitation is that they’ll also have 
>> to work in an event-driven fashion.
>> 
>> I guess the most clean way would be to port the Java wrapper to Python 
>> to finally have something such "pip install streampipes-python“. Right 
>> now in the prototype we have a special ExternalEventProcessor [1] that 
>> only calls in the
>> onInvocation() and onDetach() and forwards the request to a Flask 
>> endpoint in Python.
>> 
>> Do you have experience with running Python + Java projects „together“?
>> I saw Flink is using py4j [2]. 
>> 
>> What do you think about porting it all to Python? 
>> 
>> Patrick
>> 
>> [1] 
>> https://github.com/apache/incubator-streampipes/blob/dev/streampipes-w
>> rapper/src/main/java/org/apache/streampipes/wrapper/runtime/ExternalEv
>> entProcessor.java 
>> > wrapper/src/main/java/org/apache/streampipes/wrapper/runtime/ExternalE
>> ventProcessor.java> [2] https://www.py4j.org/ 
>> 
>> 
>>> Am 16.07.2020 um 14:42 schrieb Grainier Perera :
>>> 
>>> Hi Patrick,
>>> 
>>> This will be very useful. We can use this to expose the capabilities 
>>> of popular libraries such as scikit-learn, SciPy, etc... By the way, 
>>> How this works? Will it use java bridge, Jython or something similar?
>>> 
>>> Grainier Perera.
>>> 
>>> 
>>> On Thu, 16 Jul 2020 at 13:42, Patrick Wiener  wrote:
>>> 
 Hi guys,
 
 this mail is to inform you and discuss the addition of a new