Re: Spark version verification

Hi Mich,> What are the correlations among these links and the ability to establish a spark build version   Check the documentation list here, http://spark.apache.org/documentation.html . And the `latest` always points to the list head, for example http://spark.apache.org/docs/lat

Re: Spark version verification

Hi! Thanks Sean and Kent! By reading your answers I have also learnt something new. @Mich Talebzadeh : see the commit content by prefixing it with *https://github.com/apache/spark/commit/ *. So in your case https://github.com/apache/spark/commit/1d550c4e9

Re: Spark version verification

Hi Kent, Thanks for the links. You have to excuse my ignorance, what are the correlations among these links and the ability to establish a spark build version? view my Linkedin profile *Disclaimer:* Use it at your own risk. Any an

Re: Spark version verification

Please refer to http://spark.apache.org/docs/latest/api/sql/index.html#version  Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase

Re: Spark version verification

Many thanks spark-sql> SELECT version(); 3.1.1 1d550c4e90275ab418b9161925049239227f3dc9 What does 1d550c4e90275ab418b9161925049239227f3dc9 signify please? view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and a

Re: Spark version verification

I believe you can "SELECT version()" in Spark SQL to see the build version. On Sun, Mar 21, 2021 at 4:41 AM Mich Talebzadeh wrote: > Thanks for the detailed info. > > I was hoping that one can find a simpler answer to the Spark version than > doing forensic examination on base code so to speak.

Re: Spark version verification

Thanks for the detailed info. I was hoping that one can find a simpler answer to the Spark version than doing forensic examination on base code so to speak. The primer for this verification is that on GCP dataprocs originally built on 3.11-rc2, there was an issue with running Spark Structured Str

Re: Spark version verification

Hi! I would check out the Spark source then diff those two RCs (first just take look to the list of the changed files): $ git diff v3.1.1-rc1..v3.1.1-rc2 --stat ... The shell scripts in the release can be checked very easily: $ git diff v3.1.1-rc1..v3.1.1-rc2 --stat | grep ".sh " bin/docker-im

Spark version verification

Hi What would be a signature in Spark version or binaries that confirms the release is built on Spark built on 3.1.1 as opposed to 3.1.1-RC-1 or RC-2? Thanks Mich view my Linkedin profile *Disclaimer:* Use it at your own risk. An