I totally agree, from the user point of view (PE developer) we need a native python solution. The question are, how can we realize that and how can we start.
I have one addition to your list: * Support configuration parameters in the consul key-value store > On 17. Jul 2020, at 09:17, Patrick Wiener <[email protected]> wrote: > > Idk yet either. However, lastly all a processor really is, is a microservice > with an API, some boilerplate code and the logic iteself put in inside a > docker container. > Thus we get a REST request from the backend including configurations to start > the given processor and a REST request from the backend to stop it. > > So what would we need? > > * a way for this service to register itself in consul > * a way for it to describe itself to the backend (declareModel) > * a way to consume/produce events (maybe in the first step with a fixed > protocol such as kafka) > > anything else that I am missing? > > So, yes we could potentially add is as it is as part of the extensions > project. However, in the long rung, there will be base implementations of the > above mentioned > points that are more suitable as part of the core, rather than on the > „extension“ side - that’s my only concern right now as the current prototype > is really nothing more > than a prototype. > > Maybe we should think of a way of how we could not reimplement everything in > Python but rather extending the backend side in a way that makes it easier to > integrate > it in general. > > So I am open > > Patrick > > >> Am 17.07.2020 um 08:18 schrieb Philipp Zehnder <[email protected]>: >> >> Hi, >> >> I still don't quite understand how we can integrate the Python code into the >> backend module. >> Because then we would have to put the Python processor implementation in >> the backend as well, right? >> Or how can we use the Python dependencies in the extensions module? >> >> Currently we use ExternalEventProcessor to trigger the invocation of a >> processor. Then a REST request is send from java to python. >> Maybe we can add some of the java boilerplate code into the >> streampipes-wrapper-python module. But how about the python REST API? >> >> Philipp >> >>> On 16. Jul 2020, at 23:29, Dominik Riemer <[email protected]> wrote: >>> >>> Hi, >>> >>> I'm fully +1 for a complete, plain python wrapper that integrates both >>> runtime and controller interfaces! Also, given our microservice >>> architecture with standalone pipeline elements that communicate over >>> JSON/JSON-LD I don't think we need any code-level integration between >>> Python and Java. >>> >>> Concerning the code structure, I'd suggest to create a >>> streampipes-wrapper-python module in the core project, add the Python code >>> there and to create an example using the current Java >>> ExternalEventProcessor into the streampipes-examples project that explains >>> how to use the Python wrapper. By adding the Python code to the core >>> project, all wrappers would be located in the same repository, while the >>> extensions project solely provides specific pipeline elements and adapters. >>> In the meantime, we could add the missing features to the Python wrapper. I >>> agree that it is some work, but it should mainly consist of parsing the >>> graphs (we could use JSON instead of JSON-LD here to simplify parsing), >>> extracting parameters and adding some Flask endpoints. >>> >>> As I'm not that familiar with Python, are there any Python experts on the >>> list who want to help building the wrapper? I'd expect that finishing the >>> wrapper could probably be done within a few days if there is a Python >>> expert and someone who is familiar with the StreamPipes model - I'd be >>> happy to support the model side 😉 >>> >>> Dominik >>> >>> >>> -----Original Message----- >>> From: Philipp Zehnder <[email protected]> >>> Sent: Thursday, July 16, 2020 11:09 PM >>> To: [email protected] >>> Subject: Re: Adding StreamPipes Python wrapper >>> >>> Hi guys, >>> >>> I am also in favor of integrating the current prototype of the python >>> wrapper for further development. >>> I would also like to discuss how the proper integration might look like. >>> The cleanest way would indeed be to implement all the StreamPipes >>> interfaces and models in python, but I fear this is a lot of work and will >>> take quite some time. >>> Is there a better way, or does anyone have experience integrating Python >>> code into Java? >>> >>> As for the first integration I would suggest to create a module in the >>> extensions project and put all the code there. >>> We currently use the interfaces of the Java wrapper, right? So we do not >>> have any python specific endpoints. >>> I think this would ease the usage for people in the community and already >>> try an early version of the wrapper. >>> Alternatively, we can put it into the core in streampipes-wrapper-python as >>> you suggested, but then a user has to checkout the backend and the >>> extensions project to develop a new processor. >>> Whats your opinion on that? >>> >>> Philipp >>> >>> >>> >>>> On 16. Jul 2020, at 20:41, Patrick Wiener <[email protected]> wrote: >>>> >>>> Hi Grainier, >>>> >>>> Definitely, it should make it super simple to integrate various well >>>> known Python libs. The only real limitation is that they’ll also have >>>> to work in an event-driven fashion. >>>> >>>> I guess the most clean way would be to port the Java wrapper to Python >>>> to finally have something such "pip install streampipes-python“. Right >>>> now in the prototype we have a special ExternalEventProcessor [1] that >>>> only calls in the >>>> onInvocation() and onDetach() and forwards the request to a Flask >>>> endpoint in Python. >>>> >>>> Do you have experience with running Python + Java projects „together“? >>>> I saw Flink is using py4j [2]. >>>> >>>> What do you think about porting it all to Python? >>>> >>>> Patrick >>>> >>>> [1] >>>> https://github.com/apache/incubator-streampipes/blob/dev/streampipes-w >>>> rapper/src/main/java/org/apache/streampipes/wrapper/runtime/ExternalEv >>>> entProcessor.java >>>> <https://github.com/apache/incubator-streampipes/blob/dev/streampipes- >>>> wrapper/src/main/java/org/apache/streampipes/wrapper/runtime/ExternalE >>>> ventProcessor.java> [2] https://www.py4j.org/ <https://www.py4j.org/> >>>> >>>> >>>>> Am 16.07.2020 um 14:42 schrieb Grainier Perera <[email protected]>: >>>>> >>>>> Hi Patrick, >>>>> >>>>> This will be very useful. We can use this to expose the capabilities >>>>> of popular libraries such as scikit-learn, SciPy, etc... By the way, >>>>> How this works? Will it use java bridge, Jython or something similar? >>>>> >>>>> Grainier Perera. >>>>> >>>>> >>>>> On Thu, 16 Jul 2020 at 13:42, Patrick Wiener <[email protected]> wrote: >>>>> >>>>>> Hi guys, >>>>>> >>>>>> this mail is to inform you and discuss the addition of a new wrapper >>>>>> for >>>>>> StreamPipes: StreamPipes Python Wrapper >>>>>> >>>>>> Current wrappers such as standalone (JVM) or distributed (Flink) >>>>>> already allow us to develop new processors in the given runtime >>>>>> environment. I suppose to add the Python wrapper to this family. >>>>>> >>>>>> Why Python wrapper? >>>>>> >>>>>> * Python is a widely used language especially in the domain of data >>>>>> science >>>>>> * Python is more concise and thus better to read >>>>>> * We provide more options for standalone algorithms: It allows >>>>>> newcomers unfamiliar with Java to faster implement their algorithmns >>>>>> >>>>>> Current implementation: >>>>>> >>>>>> Currently it only works when implementing the declareModel() as part >>>>>> of the controller in Java and sending the invocation request to >>>>>> Python on the receiver side. Thus, it is necessary to run both Java >>>>>> + Python in one container . While it works, this should of course >>>>>> not be the standard way to do it. >>>>>> >>>>>> As said, I already started a very very basic implementation of it >>>>>> that I would add it to the core project under >>>>>> streampipes-wrapper-python or do you have any other thoughts? >>>>>> >>>>>> I am happy to discuss this topic with you and hope that some of you >>>>>> are eager to help working on the Python wrapper. >>>>>> >>>>>> What are your thoughts? >>>>>> >>>>>> Patrick >>>>>> >>>>>> >>>>>> >>>>>> >>>> >>> >>> >> >
