Hi, I'm fully +1 for a complete, plain python wrapper that integrates both runtime and controller interfaces! Also, given our microservice architecture with standalone pipeline elements that communicate over JSON/JSON-LD I don't think we need any code-level integration between Python and Java.
Concerning the code structure, I'd suggest to create a streampipes-wrapper-python module in the core project, add the Python code there and to create an example using the current Java ExternalEventProcessor into the streampipes-examples project that explains how to use the Python wrapper. By adding the Python code to the core project, all wrappers would be located in the same repository, while the extensions project solely provides specific pipeline elements and adapters. In the meantime, we could add the missing features to the Python wrapper. I agree that it is some work, but it should mainly consist of parsing the graphs (we could use JSON instead of JSON-LD here to simplify parsing), extracting parameters and adding some Flask endpoints. As I'm not that familiar with Python, are there any Python experts on the list who want to help building the wrapper? I'd expect that finishing the wrapper could probably be done within a few days if there is a Python expert and someone who is familiar with the StreamPipes model - I'd be happy to support the model side 😉 Dominik -----Original Message----- From: Philipp Zehnder <[email protected]> Sent: Thursday, July 16, 2020 11:09 PM To: [email protected] Subject: Re: Adding StreamPipes Python wrapper Hi guys, I am also in favor of integrating the current prototype of the python wrapper for further development. I would also like to discuss how the proper integration might look like. The cleanest way would indeed be to implement all the StreamPipes interfaces and models in python, but I fear this is a lot of work and will take quite some time. Is there a better way, or does anyone have experience integrating Python code into Java? As for the first integration I would suggest to create a module in the extensions project and put all the code there. We currently use the interfaces of the Java wrapper, right? So we do not have any python specific endpoints. I think this would ease the usage for people in the community and already try an early version of the wrapper. Alternatively, we can put it into the core in streampipes-wrapper-python as you suggested, but then a user has to checkout the backend and the extensions project to develop a new processor. Whats your opinion on that? Philipp > On 16. Jul 2020, at 20:41, Patrick Wiener <[email protected]> wrote: > > Hi Grainier, > > Definitely, it should make it super simple to integrate various well > known Python libs. The only real limitation is that they’ll also have > to work in an event-driven fashion. > > I guess the most clean way would be to port the Java wrapper to Python > to finally have something such "pip install streampipes-python“. Right > now in the prototype we have a special ExternalEventProcessor [1] that > only calls in the > onInvocation() and onDetach() and forwards the request to a Flask > endpoint in Python. > > Do you have experience with running Python + Java projects „together“? > I saw Flink is using py4j [2]. > > What do you think about porting it all to Python? > > Patrick > > [1] > https://github.com/apache/incubator-streampipes/blob/dev/streampipes-w > rapper/src/main/java/org/apache/streampipes/wrapper/runtime/ExternalEv > entProcessor.java > <https://github.com/apache/incubator-streampipes/blob/dev/streampipes- > wrapper/src/main/java/org/apache/streampipes/wrapper/runtime/ExternalE > ventProcessor.java> [2] https://www.py4j.org/ <https://www.py4j.org/> > > >> Am 16.07.2020 um 14:42 schrieb Grainier Perera <[email protected]>: >> >> Hi Patrick, >> >> This will be very useful. We can use this to expose the capabilities >> of popular libraries such as scikit-learn, SciPy, etc... By the way, >> How this works? Will it use java bridge, Jython or something similar? >> >> Grainier Perera. >> >> >> On Thu, 16 Jul 2020 at 13:42, Patrick Wiener <[email protected]> wrote: >> >>> Hi guys, >>> >>> this mail is to inform you and discuss the addition of a new wrapper >>> for >>> StreamPipes: StreamPipes Python Wrapper >>> >>> Current wrappers such as standalone (JVM) or distributed (Flink) >>> already allow us to develop new processors in the given runtime >>> environment. I suppose to add the Python wrapper to this family. >>> >>> Why Python wrapper? >>> >>> * Python is a widely used language especially in the domain of data >>> science >>> * Python is more concise and thus better to read >>> * We provide more options for standalone algorithms: It allows >>> newcomers unfamiliar with Java to faster implement their algorithmns >>> >>> Current implementation: >>> >>> Currently it only works when implementing the declareModel() as part >>> of the controller in Java and sending the invocation request to >>> Python on the receiver side. Thus, it is necessary to run both Java >>> + Python in one container . While it works, this should of course >>> not be the standard way to do it. >>> >>> As said, I already started a very very basic implementation of it >>> that I would add it to the core project under >>> streampipes-wrapper-python or do you have any other thoughts? >>> >>> I am happy to discuss this topic with you and hope that some of you >>> are eager to help working on the Python wrapper. >>> >>> What are your thoughts? >>> >>> Patrick >>> >>> >>> >>> >
