Hi guys, I am also in favor of integrating the current prototype of the python wrapper for further development. I would also like to discuss how the proper integration might look like. The cleanest way would indeed be to implement all the StreamPipes interfaces and models in python, but I fear this is a lot of work and will take quite some time. Is there a better way, or does anyone have experience integrating Python code into Java?
As for the first integration I would suggest to create a module in the extensions project and put all the code there. We currently use the interfaces of the Java wrapper, right? So we do not have any python specific endpoints. I think this would ease the usage for people in the community and already try an early version of the wrapper. Alternatively, we can put it into the core in streampipes-wrapper-python as you suggested, but then a user has to checkout the backend and the extensions project to develop a new processor. Whats your opinion on that? Philipp > On 16. Jul 2020, at 20:41, Patrick Wiener <[email protected]> wrote: > > Hi Grainier, > > Definitely, it should make it super simple to integrate various well known > Python libs. The only real limitation is that they’ll also have to work in an > event-driven fashion. > > I guess the most clean way would be to port the Java wrapper to Python to > finally have something such "pip install streampipes-python“. Right now in > the > prototype we have a special ExternalEventProcessor [1] that only calls in the > onInvocation() and onDetach() and forwards the request to a Flask endpoint in > Python. > > Do you have experience with running Python + Java projects „together“? > I saw Flink is using py4j [2]. > > What do you think about porting it all to Python? > > Patrick > > [1] > https://github.com/apache/incubator-streampipes/blob/dev/streampipes-wrapper/src/main/java/org/apache/streampipes/wrapper/runtime/ExternalEventProcessor.java > > <https://github.com/apache/incubator-streampipes/blob/dev/streampipes-wrapper/src/main/java/org/apache/streampipes/wrapper/runtime/ExternalEventProcessor.java> > [2] https://www.py4j.org/ <https://www.py4j.org/> > > >> Am 16.07.2020 um 14:42 schrieb Grainier Perera <[email protected]>: >> >> Hi Patrick, >> >> This will be very useful. We can use this to expose the capabilities of >> popular libraries such as scikit-learn, SciPy, etc... By the way, How this >> works? Will it use java bridge, Jython or something similar? >> >> Grainier Perera. >> >> >> On Thu, 16 Jul 2020 at 13:42, Patrick Wiener <[email protected]> wrote: >> >>> Hi guys, >>> >>> this mail is to inform you and discuss the addition of a new wrapper for >>> StreamPipes: StreamPipes Python Wrapper >>> >>> Current wrappers such as standalone (JVM) or distributed (Flink) already >>> allow us to develop new processors in the given >>> runtime environment. I suppose to add the Python wrapper to this family. >>> >>> Why Python wrapper? >>> >>> * Python is a widely used language especially in the domain of data science >>> * Python is more concise and thus better to read >>> * We provide more options for standalone algorithms: It allows newcomers >>> unfamiliar with Java to faster implement their algorithmns >>> >>> Current implementation: >>> >>> Currently it only works when implementing the declareModel() as part of >>> the controller in Java and sending the invocation request >>> to Python on the receiver side. Thus, it is necessary to run both Java + >>> Python in one container . While it works, this should >>> of course not be the standard way to do it. >>> >>> As said, I already started a very very basic implementation of it that I >>> would add it to the core project under streampipes-wrapper-python >>> or do you have any other thoughts? >>> >>> I am happy to discuss this topic with you and hope that some of you are >>> eager to help working on the Python wrapper. >>> >>> What are your thoughts? >>> >>> Patrick >>> >>> >>> >>> >
