SparkGraph review process

2019-10-04 Thread Mats Rydberg
Hello dear Spark community We are the developers behind the SparkGraph SPIP, which is a project created out of our work on openCypher Morpheus ( https://github.com/opencypher/morpheus). During this year we have collaborated with mainly Xiangrui Meng of Databricks to define and develop a new

Re: [SS] How to create a streaming DataFrame (for a custom Source in Spark 2.4.4 / MicroBatch / DSv1)?

2019-10-04 Thread Jungtaek Lim
I remembered the actual case from developer who implements custom data source. https://lists.apache.org/thread.html/c1a210510b48bb1fea89828c8e2f5db8c27eba635e0079a97b0c7faf@%3Cdev.spark.apache.org%3E Quoting here: We started implementing DSv2 in the 2.4 branch, but quickly discovered that the

ApacheCon Europe 2019 talks which are relevant to Apache Spark

2019-10-04 Thread myrle
Dear Apache Spark committers, In a little over 2 weeks time, ApacheCon Europe is taking place in Berlin. Join us from October 22 to 24 for an exciting program and lovely get-together of the Apache Community. We are also planning a hackathon.  If your project is interested in participating,

[build system] jenkins restarted

2019-10-04 Thread Shane Knapp
it was wedged and i had to perform a quick restart. sorry about the interruption of service! shane -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu - To unsubscribe e-mail:

Re: SparkGraph review process

2019-10-04 Thread Xiangrui Meng
Hi all, I want to clarify my role first to avoid misunderstanding. I'm an individual contributor here. My work on the graph SPIP as well as other Spark features I contributed to are not associated with my employer. It became quite challenging for me to keep track of the graph SPIP work due to

[build system] maven snapshot builds moved to ubuntu workers

2019-10-04 Thread Shane Knapp
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-maven-snapshots/ https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.4-maven-snapshots/ i created dry-run test builds and everything looked great. please file a JIRA is anything published by these jobs looks fishy. shane -- Shane