Hi Grainier, Definitely, it should make it super simple to integrate various well known Python libs. The only real limitation is that they’ll also have to work in an event-driven fashion.
I guess the most clean way would be to port the Java wrapper to Python to finally have something such "pip install streampipes-python“. Right now in the prototype we have a special ExternalEventProcessor [1] that only calls in the onInvocation() and onDetach() and forwards the request to a Flask endpoint in Python. Do you have experience with running Python + Java projects „together“? I saw Flink is using py4j [2]. What do you think about porting it all to Python? Patrick [1] https://github.com/apache/incubator-streampipes/blob/dev/streampipes-wrapper/src/main/java/org/apache/streampipes/wrapper/runtime/ExternalEventProcessor.java <https://github.com/apache/incubator-streampipes/blob/dev/streampipes-wrapper/src/main/java/org/apache/streampipes/wrapper/runtime/ExternalEventProcessor.java> [2] https://www.py4j.org/ <https://www.py4j.org/> > Am 16.07.2020 um 14:42 schrieb Grainier Perera <[email protected]>: > > Hi Patrick, > > This will be very useful. We can use this to expose the capabilities of > popular libraries such as scikit-learn, SciPy, etc... By the way, How this > works? Will it use java bridge, Jython or something similar? > > Grainier Perera. > > > On Thu, 16 Jul 2020 at 13:42, Patrick Wiener <[email protected]> wrote: > >> Hi guys, >> >> this mail is to inform you and discuss the addition of a new wrapper for >> StreamPipes: StreamPipes Python Wrapper >> >> Current wrappers such as standalone (JVM) or distributed (Flink) already >> allow us to develop new processors in the given >> runtime environment. I suppose to add the Python wrapper to this family. >> >> Why Python wrapper? >> >> * Python is a widely used language especially in the domain of data science >> * Python is more concise and thus better to read >> * We provide more options for standalone algorithms: It allows newcomers >> unfamiliar with Java to faster implement their algorithmns >> >> Current implementation: >> >> Currently it only works when implementing the declareModel() as part of >> the controller in Java and sending the invocation request >> to Python on the receiver side. Thus, it is necessary to run both Java + >> Python in one container . While it works, this should >> of course not be the standard way to do it. >> >> As said, I already started a very very basic implementation of it that I >> would add it to the core project under streampipes-wrapper-python >> or do you have any other thoughts? >> >> I am happy to discuss this topic with you and hope that some of you are >> eager to help working on the Python wrapper. >> >> What are your thoughts? >> >> Patrick >> >> >> >>
