spark-postgres A library for reading data from and transferring data to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x faster.spark-func-extrasA library that brings excellent and useful functions from various modern database management systems to Apache Spark.
Many thanksspark-sql> SELECT version();3.1.1 1d550c4e90275ab418b9161925049239227f3dc9What does 1d550c4e90275ab418b9161925049239227f3dc9 signify please?
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On Sun, 21 Mar 2021 at 15:14, Sean Owen <sro...@gmail.com> wrote:I believe you can "SELECT version()" in Spark SQL to see the build version.On Sun, Mar 21, 2021 at 4:41 AM Mich Talebzadeh <mich.talebza...@gmail.com> wrote:Thanks for the detailed info.I was hoping that one can find a simpler answer to the Spark version than doing forensic examination on base code so to speak.The primer for this verification is that on GCP dataprocs originally built on 3.11-rc2, there was an issue with running Spark Structured Streaming (SSS) which I reported to this forum before.After a while and me reporting to Google, they have now upgraded the base to Spark 3.1.1 itself. I am not privy to how they did the upgrade itself.In the meantime we installed 3.1.1 on-premise and ran it with the same Python code for SSS. It worked fine.However, when I run the same code on GCP dataproc upgraded to 3.1.1, occasionally I see this error21/03/18 16:53:38 ERROR org.apache.spark.scheduler.AsyncEventQueue: Listener EventLoggingListener threw an exception
java.util.ConcurrentModificationException
at java.util.Hashtable$Enumerator.next(Hashtable.java:1387)
This may be for other reasons or the consequence of upgrading from 3.1.1-rc2 to 3.11?
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On Sat, 20 Mar 2021 at 22:41, Attila Zsolt Piros <piros.attila.zs...@gmail.com> wrote:Hi!
I would check out the Spark source then diff those two RCs (first just take look to the list of the changed files):
$ git diff v3.1.1-rc1..v3.1.1-rc2 --stat
...
The shell scripts in the release can be checked very easily:
$ git diff v3.1.1-rc1..v3.1.1-rc2 --stat | grep ".sh "
bin/docker-image-tool.sh | 6 +-
dev/create-release/release-build.sh | 2 +-
We are lucky as docker-image-tool.sh is part of the released version.
Is it from v3.1.1-rc2 or v3.1.1-rc1?
Of course this only works if docker-image-tool.sh is not changed from the v3.1.1-rc2 back to v3.1.1-rc1.
So let's continue with the python (and latter with R) files:
$ git diff v3.1.1-rc1..v3.1.1-rc2 --stat | grep ".py "
python/pyspark/sql/avro/functions.py | 4 +-
python/pyspark/sql/dataframe.py | 1 +
python/pyspark/sql/functions.py | 285 +++++------
.../pyspark/sql/tests/test_pandas_cogrouped_map.py | 12 +
python/pyspark/sql/tests/test_pandas_map.py | 8 +
...
After you have enough proof you can stop (to decide what is enough here should be decided by you).
Finally you can use javap / scalap on the classes from the jars and check some code changes which is more harder to be analyzed than a simple text file.
Best Regards,
AttilaOn Thu, Mar 18, 2021 at 4:09 PM Mich Talebzadeh <mich.talebza...@gmail.com> wrote:HiWhat would be a signature in Spark version or binaries that confirms the release is built on Spark built on 3.1.1 as opposed to 3.1.1-RC-1 or RC-2?ThanksMich
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
