Maciej, How about a little more useful response of "we'll help you find the right audience for this discussion and collaborate with you to make the case."?
- David On Thu, Mar 24, 2016 at 11:32 AM, Maciej Fijalkowski <fij...@gmail.com> wrote: > Ok fine, but we're not the receipents of such a message. > > Please lobby PSF for having a JIT, we all support that :-) > > On Thu, Mar 24, 2016 at 5:23 PM, John Camara <john.m.cam...@gmail.com> wrote: >> Hi Fijal, >> >> I understand where your coming from and not trying to convince you to work >> on it. Just mainly trying to point out a need that may not be obvious to >> this community. I don't spend much time on big data and analytics so I >> don't have a lot of time to devote to this task. That could change in the >> future so you never know I may end up getting involved with this. >> >> At the end of the day I think it is the PSF, which needs to do an honest >> assessment of the current state of Python and in programming in general, so >> that they can help direct the future of Python. I think with an honest >> assessment it should be clear that it is absolutely necessary that a dynamic >> language have a JIT. Otherwise, a language like Node would not be growing so >> quickly on the server side. An honest assessment would conclude that Python >> needs to play a major role in big data and analytics as we don't want this >> to be another area where Python misses the boat. As with all languages >> other than JavaScript we missed playing an important role on web front end. >> More recently we missed out on mobile. I don't think it is good for us to >> miss out on big data. It would be a shame since we had such a strong >> scientific community which initially gave us a huge advantage over other >> communities. Missing out on big data might also be the driver that moves >> the scientific community in a different direction which would be a big loss >> to Python. >> >> I personally don't see any particular companies or industries that are >> willing to fund the tasks needed to solve these issues. It's not to say >> there are no more funds for Python projects its just likely no one company >> will be willing to fund these kinds of projects on their own. It really >> needs the PSF to coordinate these efforts but they seamed to be more focus >> on trying to make Python 3 a success instead of improving the overall health >> of the community. >> >> I believe that Python is in pretty good shape in being able to solve these >> issues but it just needs some funding and focus to get there. >> >> Hopefully the workshop will be successful and help create some focus. >> >> John >> >> On Thu, Mar 24, 2016 at 8:56 AM, Maciej Fijalkowski <fij...@gmail.com> >> wrote: >>> >>> Hi John >>> >>> Thanks for explaining the current situation of the ecosystem. I'm not >>> quite sure what your intention is. PyPy (and CPython) is very easy to >>> embed through any C-level API, especially with the latest additions to >>> cffi embedding. If someone feels like doing the work to share stuff >>> that way (as I presume a lot of data presented in JVM can be >>> represented as some pointer and shape how to access it), then he's >>> obviously more than free to do so, I'm even willing to help with that. >>> Now this seems like a medium-to-big size project that additionally >>> will require quite a bit of community will to endorse. Are you willing >>> to volunteer to work on such a project and dedicate a lot of time to >>> it? If not, then there is no way you can convince us to volunteer our >>> own time to do it - it's just too big and quite a bit far out of our >>> usual areas of interest. If there is some commercial interest (and I >>> think there might be) in pushing python and especially pypy further in >>> that area, we might want to have a better story for numpy first, but >>> then feel free to send those corporate interest people my way, we can >>> maybe organize something. If you want us to do community service to >>> push Python solutions in the area I have very little clue about >>> however, I would like to politely decline. >>> >>> Cheers, >>> fijal >>> >>> On Thu, Mar 24, 2016 at 2:22 PM, John Camara <john.m.cam...@gmail.com> >>> wrote: >>> > Besides JPype and PyJNIus there is also https://www.py4j.org/. I >>> > haven't >>> > heard of JPype being used in any recent projects so I assuming it is >>> > outdated by now. PyJNIus gets used but I tend to only see it used on >>> > Android projects. The Py4J project gets used often in >>> > numerical/scientific >>> > projects mainly due to it use in PySpark. The problem with all these >>> > libraries is that they don't have a way to share large amounts of memory >>> > between the JVM and Python VMs and so large chunks of data have to be >>> > copied/serialized when going between the 2 VMs. >>> > >>> > Spark is the de facto standard in clustering computing at this point in >>> > time. At a high level Spark executes code that is distributed >>> > throughout a >>> > cluster so that the code being executed is as close as possible to where >>> > the >>> > data lives so as to minimize transferring of large amounts of data. The >>> > code that needs to be executed are packaged up into units called >>> > Resilient >>> > Distributed Dataset (RDD). RDDs are lazy evaluated and are essential >>> > graphs >>> > of the operations that need to be performed on the data. They are >>> > capable >>> > of reading data from many types of sources, outputting to multiple types >>> > of >>> > sources, containing the code that needs to be executed, and are also >>> > responsible to caching or keeping results in memory for future RDDs that >>> > maybe executed. >>> > >>> > If you write all your code in Java or Scala, its execution will be >>> > performed >>> > in JVMs distributed in the cluster. On the other hand, Spark does not >>> > limit >>> > its use to only Java based languages so Python can be used. In the case >>> > of >>> > Python the PySpark library is used. When Python is used, the PySpark >>> > library can be used to define the RDDs that will be executed under the >>> > JVM. >>> > In this scenario, only if required, the final results of the >>> > calculations >>> > will end up being passed to Python. I say only if necessary as its >>> > possible >>> > the end results may just be left in memory or to create an output such >>> > as an >>> > hdfs file in hadoop and does not need to be transferred to Python. Under >>> > this scenario the code is written in Python but effectively all the >>> > "real" >>> > work is performed under the JVM. >>> > >>> > Often someone writing Python is also going to want to perform some of >>> > the >>> > operations under Python. This can be done as the RDDs that are created >>> > can >>> > contain both operations that get performed under the JVM as well as >>> > Python >>> > (and of course other languages are supported). When Python is involved >>> > Spark will start up Python VMs on the required nodes so that the Python >>> > portions of the work can be performed. The Python VMs can either be >>> > CPython, PyPy or even a mix of both CPython and PyPy. The downside to >>> > using >>> > non Java languages is the overhead of passing data between the JVM and >>> > the >>> > Python VM as the memory is not shared between the processes but instead >>> > copied/serialized between them. >>> > >>> > Because this data is copied between the 2 VMs, anyone who writes Python >>> > code >>> > for this environment always has to be conscious of the data being copied >>> > between the processes so as to not let the amount of the extra overhead >>> > become a large burden. Quite often the goal will be to first perform >>> > the >>> > bulk of the operations under the JVM and then hopefully only a smaller >>> > subset of the data will have to be processed under Python. If this can >>> > be >>> > done then the overhead can be minimized and then there is essential no >>> > down >>> > sides to using Python in the pipeline of operations. >>> > >>> > If your unfortunate and need to perform some of the processing early in >>> > the >>> > pipline under Python and worse yet if there is a need to go back and >>> > forth >>> > many times between Python and Java the overhead of coping huge amounts >>> > of >>> > data can significantly slow things down which essentially puts Python at >>> > a >>> > disadvantage to Java. >>> > >>> > If it was possible to change the model of execution such that it was >>> > possible to embed the Python VM in the JVM or vice versa and that the >>> > memory >>> > could be shared between the 2 VMs the downside of using Python in this >>> > environment would be eliminated or at the very least minimized to the >>> > point >>> > where it is no longer an issue. Thus the need for a jffi library. >>> > >>> > There is a strong desire by many to use dynamic languages in these >>> > clustered >>> > environments and Python is likely in the best position to become the >>> > language of choice due to its ability to work with C based libraries and >>> > of >>> > course its syntax. The issues that hold Python back at this point is >>> > the >>> > serialization overhead, not so great state of packaging, and not having >>> > both >>> > the speed of the JIT and complete access to numpy/scipy ecosystem. >>> > >>> > Luckily for Python at this point there is no other dynamic language that >>> > is >>> > a clear winner today. But if too much time passes before these issues >>> > are >>> > solved I'm sure another language will step up to the plate. At this >>> > point >>> > my expectations is that Node could likely make a move. It already has >>> > the >>> > speed due to the Java Script JITs, it already has a great story for >>> > packaging and deployment, and its growth is exploding on the server side >>> > due >>> > to all the money being poured into it. What it strongly lacks today is >>> > the >>> > connection to C/legacy code, numerical/scientific modules and of course >>> > it >>> > also does not have a solution to the data copying overhead it also has >>> > with >>> > the JVM. >>> > >>> > Any way, this is just my 2 cents on what is currently holding Python >>> > back >>> > from taking off in this space. >>> > >>> > On Thu, Mar 24, 2016 at 2:32 AM, Hakan Ardo <hakan.a...@gmail.com> >>> > wrote: >>> >> >>> >> >>> >> On Mar 23, 2016 21:49, "Armin Rigo" <ar...@tunes.org> wrote: >>> >> > >>> >> > Hi John, >>> >> > >>> >> > On 23 March 2016 at 19:16, John Camara <john.m.cam...@gmail.com> >>> >> > wrote: >>> >> > > I would like to suggest one more topic for the workshop. I see a >>> >> > > big >>> >> > > need >>> >> > > for a library (jffi) similar to cffi but that provides a bridge to >>> >> > > Java >>> >> > > instead of C code. The ability to seamlessly work with native Java >>> >> > > data/code >>> >> > > would offer a huge improvement (...) >>> >> > >>> >> > Isn't it what JPype does? Can you describe how it isn't suitable for >>> >> > your needs? >>> >> >>> >> There is also PyJNIus: >>> >> >>> >> https://pyjnius.readthedocs.org/en/latest/ >>> > >>> > >>> > >>> > _______________________________________________ >>> > pypy-dev mailing list >>> > pypy-dev@python.org >>> > https://mail.python.org/mailman/listinfo/pypy-dev >>> > >> >> > _______________________________________________ > pypy-dev mailing list > pypy-dev@python.org > https://mail.python.org/mailman/listinfo/pypy-dev _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
