Re: CVE-2020-9480: Apache Spark RCE vulnerability in auth-enabled standalone master

2020-08-03 Thread Sean Owen
I'm resending this CVE from several months ago to user@ and dev@, as we understand that a tool to exploit it may be released soon. The most straightforward mitigation for those that are affected (using the standalone master, where spark.authenticate is necessary) is to update to 2.4.6 or 3.0.0+. F

Re: spark-on-k8s is still experimental?

2020-08-03 Thread Holden Karau
There was discussion around removing the statement and declaring it GA but I believe it was decided to leave it in until an external shuffle service is supported on K8s. On Mon, Aug 3, 2020 at 2:45 AM JackyLee wrote: > +1. It has been worked well in our company and we has used it to support > on

Re: [PySpark] Revisiting PySpark type annotations

2020-08-03 Thread Driesprong, Fokko
Cool stuff! Moving it to the ASF would be a great first step. I think you might want to check the IP Clearance template: http://incubator.apache.org/ip-clearance/ip-clearance-template.html This is the one being used when donating the Airflow Kubernetes operator from Google to the ASF: http://mail

Re: spark-on-k8s is still experimental?

2020-08-03 Thread JackyLee
+1. It has been worked well in our company and we has used it to support online services since March in this year. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To unsubscribe e-mail: dev-unsubscr..

Re: spark-on-k8s is still experimental?

2020-08-03 Thread Sean Owen
Likewise, I'm not super familiar with this integration, but, it's been out for several minor and one new major version, and doubt that it is any more 'experimental' now that it ever will be. Unless someone who knows more suggests that it's still really a WIP, or that some aspects are still fairly s

Re: [PySpark] Revisiting PySpark type annotations

2020-08-03 Thread Hyukjin Kwon
Okay, seems like we can create a separate repo as apache/spark? e.g.) https://issues.apache.org/jira/browse/INFRA-20470 We can also think about porting the files as are. I will try to have a short sync with the author Maciej, and share what we discussed offline. 2020년 7월 22일 (수) 오후 10:43, Maciej

spark-on-k8s is still experimental?

2020-08-03 Thread Takeshi Yamamuro
Hi, all A Spark user I know asked me this question. I checked the the spark-on-k8s document and it says; ``` **The Kubernetes scheduler is currently experimental. In future versions, there may be behavioral changes around configuration, container images and entrypoints.** ``` https://github.com/ap