Fwd: Does pyspark worker really use pipe?

Hello, As pyspark internals wiki said, pyspark worker use pipe to communicate, not socket. https://cwiki.apache.org/confluence/display/SPARK/PySpark+Internals I have checked the pyspark/worker.py code: if __name__ == '__main__': # Read a local port to connect to from stdin java_port =

Re: Spark history server running on Mongo

Yes, you are absolutely right, though UI does not change often, and it potentially allows to iterate faster, IMHO, which is why started working on this. For me, it felt like this functionality could easily be outsourced to a separate project. And, as you pointed out, I did add some small fixes to

Re: Spark history server running on Mongo

On Tue, Jul 18, 2017 at 7:21 PM, Ivan Sadikov wrote: > Repository that I linked to does not require rebuilding Spark and could be > used with current distribution, which is preferable in my case. Fair enough, although that means that you're re-implementing the Spark UI,

Re: [ANNOUNCE] Announcing Apache Spark 2.2.0

Oh yeah, new Spark version, new regression bugs :) https://issues.apache.org/jira/browse/SPARK-21470 M. 2017-07-17 22:01 GMT+02:00 Sam Elamin : > Well done! This is amazing news :) Congrats and really cant wait to > spread the structured streaming love! > > On Mon,