Thanks! I was wondering about that ClientE2ETestSuite failure today, so I'm
glad to know that it's also being experienced by others.
On a similar note, I am experiencing the following error when running the
Python tests with Python 3.7:
+ ./python/run-tests --python-executables=python3
Running PySpark tests. Output is in
/home/ec2-user/spark/python/unit-tests.log
Will test against the following Python executables: ['python3']
Will test the following Python modules: ['pyspark-connect', 'pyspark-core',
'pyspark-errors', 'pyspark-ml', 'pyspark-mllib', 'pyspark-pandas',
'pyspark-pandas-slow', 'pyspark-resource', 'pyspark-sql',
'pyspark-streaming']
python3 python_implementation is CPython
python3 version is: Python 3.7.16
Starting test(python3): pyspark.ml.tests.test_feature (temp output:
/home/ec2-user/spark/python/target/8ca9ab1a-05cc-4845-bf89-30d9001510bc/python3__pyspark.ml.tests.test_feature__kg6sseie.log)
Starting test(python3): pyspark.ml.tests.test_base (temp output:
/home/ec2-user/spark/python/target/f2264f3b-6b26-4e61-9452-8d6ddd7eb002/python3__pyspark.ml.tests.test_base__0902zf9_.log)
Starting test(python3): pyspark.ml.tests.test_algorithms (temp output:
/home/ec2-user/spark/python/target/d1dc4e07-e58c-4c03-abe5-09d8fab22e6a/python3__pyspark.ml.tests.test_algorithms__lh3wb2u8.log)
Starting test(python3): pyspark.ml.tests.test_evaluation (temp output:
/home/ec2-user/spark/python/target/3f42dc79-c945-4cf2-a1eb-83e72b40a9ee/python3__pyspark.ml.tests.test_evaluation__89idc7fa.log)
Finished test(python3): pyspark.ml.tests.test_base (16s)
Starting test(python3): pyspark.ml.tests.test_functions (temp output:
/home/ec2-user/spark/python/target/5a3b90f0-216b-4edd-9d15-6619d3e03300/python3__pyspark.ml.tests.test_functions__g5u1290s.log)
Traceback (most recent call last):
File "/usr/lib64/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib64/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ec2-user/spark/python/pyspark/ml/tests/test_functions.py",
line 21, in <module>
from pyspark.ml.functions import predict_batch_udf
File "/home/ec2-user/spark/python/pyspark/ml/functions.py", line 38, in
<module>
from typing import Any, Callable, Iterator, List, Mapping, Protocol,
TYPE_CHECKING, Tuple, Union
ImportError: cannot import name 'Protocol' from 'typing'
(/usr/lib64/python3.7/typing.py)
Had test failures in pyspark.ml.tests.test_functions with python3; see logs.
I know we should move on to a newer version of Python, but isn't Python 3.7
still officially supported?
Thank you,
Jonathan Kelly
On Wed, Feb 22, 2023 at 1:47 PM Herman van Hovell
<her...@databricks.com.invalid> wrote:
> Hi All,
>
> Thanks for testing the 3.4.0 RC! I apologize for the maven testing
> failures for the Spark Connect Scala Client. We will try to get those
> sorted as soon as possible.
>
> This is an artifact of having multiple build systems, and only running CI
> for one (SBT). That, however, is a debate for another day :)...
>
> Cheers,
> Herman
>
> On Wed, Feb 22, 2023 at 5:32 PM Bjørn Jørgensen <bjornjorgen...@gmail.com>
> wrote:
>
>> ./build/mvn clean package
>>
>> I'm using ubuntu rolling, python 3.11 openjdk 17
>>
>> CompatibilitySuite:
>> - compatibility MiMa tests *** FAILED ***
>> java.lang.AssertionError: assertion failed: Failed to find the jar
>> inside folder: /home/bjorn/spark-3.4.0/connector/connect/client/jvm/target
>> at scala.Predef$.assert(Predef.scala:223)
>> at
>> org.apache.spark.sql.connect.client.util.IntegrationTestUtils$.findJar(IntegrationTestUtils.scala:67)
>> at
>> org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar$lzycompute(CompatibilitySuite.scala:57)
>> at
>> org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar(CompatibilitySuite.scala:53)
>> at
>> org.apache.spark.sql.connect.client.CompatibilitySuite.$anonfun$new$1(CompatibilitySuite.scala:69)
>> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
>> at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
>> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>> at org.scalatest.Transformer.apply(Transformer.scala:22)
>> at org.scalatest.Transformer.apply(Transformer.scala:20)
>> ...
>> - compatibility API tests: Dataset *** FAILED ***
>> java.lang.AssertionError: assertion failed: Failed to find the jar
>> inside folder: /home/bjorn/spark-3.4.0/connector/connect/client/jvm/target
>> at scala.Predef$.assert(Predef.scala:223)
>> at
>> org.apache.spark.sql.connect.client.util.IntegrationTestUtils$.findJar(IntegrationTestUtils.scala:67)
>> at
>> org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar$lzycompute(CompatibilitySuite.scala:57)
>> at
>> org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar(CompatibilitySuite.scala:53)
>> at
>> org.apache.spark.sql.connect.client.CompatibilitySuite.$anonfun$new$7(CompatibilitySuite.scala:110)
>> at
>> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
>> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
>> at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
>> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>> at org.scalatest.Transformer.apply(Transformer.scala:22)
>> ...
>> SparkConnectClientSuite:
>> - Placeholder test: Create SparkConnectClient
>> - Test connection
>> - Test connection string
>> - Check URI: sc://host, isCorrect: true
>> - Check URI: sc://localhost/, isCorrect: true
>> - Check URI: sc://localhost:1234/, isCorrect: true
>> - Check URI: sc://localhost/;, isCorrect: true
>> - Check URI: sc://host:123, isCorrect: true
>> - Check URI: sc://host:123/;user_id=a94, isCorrect: true
>> - Check URI: scc://host:12, isCorrect: false
>> - Check URI: http://host, isCorrect: false
>> - Check URI: sc:/host:1234/path, isCorrect: false
>> - Check URI: sc://host/path, isCorrect: false
>> - Check URI: sc://host/;parm1;param2, isCorrect: false
>> - Check URI: sc://host:123;user_id=a94, isCorrect: false
>> - Check URI: sc:///user_id=123, isCorrect: false
>> - Check URI: sc://host:-4, isCorrect: false
>> - Check URI: sc://:123/, isCorrect: false
>> - Non user-id parameters throw unsupported errors
>> DatasetSuite:
>> - limit
>> - select
>> - filter
>> - write
>> UserDefinedFunctionSuite:
>> - udf and encoder serialization
>> Run completed in 21 seconds, 944 milliseconds.
>> Total number of tests run: 389
>> Suites: completed 10, aborted 0
>> Tests: succeeded 386, failed 3, canceled 0, ignored 0, pending 0
>> *** 3 TESTS FAILED ***
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Reactor Summary for Spark Project Parent POM 3.4.0:
>> [INFO]
>> [INFO] Spark Project Parent POM ........................... SUCCESS [
>> 47.096 s]
>> [INFO] Spark Project Tags ................................. SUCCESS [
>> 14.759 s]
>> [INFO] Spark Project Sketch ............................... SUCCESS [
>> 21.628 s]
>> [INFO] Spark Project Local DB ............................. SUCCESS [
>> 20.311 s]
>> [INFO] Spark Project Networking ........................... SUCCESS
>> [01:07 min]
>> [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [
>> 15.921 s]
>> [INFO] Spark Project Unsafe ............................... SUCCESS [
>> 16.020 s]
>> [INFO] Spark Project Launcher ............................. SUCCESS [
>> 10.873 s]
>> [INFO] Spark Project Core ................................. SUCCESS
>> [37:10 min]
>> [INFO] Spark Project ML Local Library ..................... SUCCESS [
>> 40.841 s]
>> [INFO] Spark Project GraphX ............................... SUCCESS
>> [02:39 min]
>> [INFO] Spark Project Streaming ............................ SUCCESS
>> [05:53 min]
>> [INFO] Spark Project Catalyst ............................. SUCCESS
>> [11:22 min]
>> [INFO] Spark Project SQL .................................. SUCCESS [
>> 02:27 h]
>> [INFO] Spark Project ML Library ........................... SUCCESS
>> [22:45 min]
>> [INFO] Spark Project Tools ................................ SUCCESS [
>> 7.263 s]
>> [INFO] Spark Project Hive ................................. SUCCESS [
>> 01:21 h]
>> [INFO] Spark Project REPL ................................. SUCCESS
>> [02:07 min]
>> [INFO] Spark Project Assembly ............................. SUCCESS [
>> 11.704 s]
>> [INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [
>> 26.748 s]
>> [INFO] Spark Integration for Kafka 0.10 ................... SUCCESS
>> [01:44 min]
>> [INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS
>> [33:27 min]
>> [INFO] Spark Project Examples ............................. SUCCESS
>> [01:17 min]
>> [INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [
>> 27.292 s]
>> [INFO] Spark Avro ......................................... SUCCESS
>> [02:18 min]
>> [INFO] Spark Project Connect Common ....................... SUCCESS [
>> 43.728 s]
>> [INFO] Spark Project Connect Server ....................... SUCCESS
>> [01:19 min]
>> [INFO] Spark Project Connect Client ....................... FAILURE [
>> 53.524 s]
>> [INFO] Spark Protobuf ..................................... SKIPPED
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD FAILURE
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 05:58 h
>> [INFO] Finished at: 2023-02-22T22:28:38+01:00
>> [INFO]
>> ------------------------------------------------------------------------
>> [ERROR] Failed to execute goal
>> org.scalatest:scalatest-maven-plugin:2.2.0:test (test) on project
>> spark-connect-client-jvm_2.12: There are test failures -> [Help 1]
>>
>> ons. 22. feb. 2023 kl. 21:41 skrev Mridul Muralidharan <mri...@gmail.com
>> >:
>>
>>>
>>> Signatures, digests, etc check out fine - thanks for updating them !
>>> Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes
>>>
>>>
>>> The test ClientE2ETestSuite.simple udf failed [1] in "Connect Client "
>>> module ... yet to test "Spark Protobuf" module due to the failure.
>>>
>>>
>>> Regards,
>>> Mridul
>>>
>>> [1]
>>>
>>> - simple udf *** FAILED ***
>>>
>>> io.grpc.StatusRuntimeException: INTERNAL:
>>> org.apache.spark.sql.ClientE2ETestSuite
>>>
>>> at io.grpc.Status.asRuntimeException(Status.java:535)
>>>
>>> at
>>> io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:660)
>>>
>>> at org.apache.spark.sql.connect.client.SparkResult.org
>>> $apache$spark$sql$connect$client$SparkResult$$processResponses(SparkResult.scala:50)
>>>
>>> at
>>> org.apache.spark.sql.connect.client.SparkResult.length(SparkResult.scala:95)
>>>
>>> at
>>> org.apache.spark.sql.connect.client.SparkResult.toArray(SparkResult.scala:112)
>>>
>>> at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2037)
>>>
>>> at org.apache.spark.sql.Dataset.withResult(Dataset.scala:2267)
>>>
>>> at org.apache.spark.sql.Dataset.collect(Dataset.scala:2036)
>>>
>>> at
>>> org.apache.spark.sql.ClientE2ETestSuite.$anonfun$new$5(ClientE2ETestSuite.scala:65)
>>>
>>> at
>>> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
>>>
>>> ...
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Feb 22, 2023 at 2:07 AM Mridul Muralidharan <mri...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> Thanks Xinrong !
>>>> The signature verifications are fine now ... will continue with testing
>>>> the release.
>>>>
>>>>
>>>> Regards,
>>>> Mridul
>>>>
>>>>
>>>> On Wed, Feb 22, 2023 at 1:27 AM Xinrong Meng <xinrong.apa...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Mridul,
>>>>>
>>>>> Would you please try that again? It should work now.
>>>>>
>>>>> On Wed, Feb 22, 2023 at 2:04 PM Mridul Muralidharan <mri...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Hi Xinrong,
>>>>>>
>>>>>> Was it signed with the same key as present in KEYS [1] ?
>>>>>> I am seeing errors with gpg when validating. For example:
>>>>>>
>>>>>>
>>>>>> $ gpg --verify pyspark-3.4.0.tar.gz.asc
>>>>>>
>>>>>> gpg: assuming signed data in 'pyspark-3.4.0.tar.gz'
>>>>>>
>>>>>> gpg: Signature made Tue 21 Feb 2023 05:56:05 AM CST
>>>>>>
>>>>>> gpg: using RSA key
>>>>>> CC68B3D16FE33A766705160BA7E57908C7A4E1B1
>>>>>>
>>>>>> gpg: issuer "xinr...@apache.org"
>>>>>>
>>>>>> gpg: Can't check signature: No public key
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Mridul
>>>>>>
>>>>>> [1] https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>
>>>>>>
>>>>>> On Tue, Feb 21, 2023 at 10:36 PM Xinrong Meng <
>>>>>> xinrong.apa...@gmail.com> wrote:
>>>>>>
>>>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>>>> version 3.4.0.
>>>>>>>
>>>>>>> The vote is open until 11:59pm Pacific time *February 27th* and
>>>>>>> passes if a majority +1 PMC votes are cast, with a minimum of 3 +1
>>>>>>> votes.
>>>>>>>
>>>>>>> [ ] +1 Release this package as Apache Spark 3.4.0
>>>>>>> [ ] -1 Do not release this package because ...
>>>>>>>
>>>>>>> To learn more about Apache Spark, please see
>>>>>>> http://spark.apache.org/
>>>>>>>
>>>>>>> The tag to be voted on is *v3.4.0-rc1* (commit
>>>>>>> e2484f626bb338274665a49078b528365ea18c3b):
>>>>>>> https://github.com/apache/spark/tree/v3.4.0-rc1
>>>>>>>
>>>>>>> The release files, including signatures, digests, etc. can be found
>>>>>>> at:
>>>>>>> https://dist.apache.org/repos/dist/dev/spark/v3.4.0-rc1-bin/
>>>>>>>
>>>>>>> Signatures used for Spark RCs can be found in this file:
>>>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>>
>>>>>>> The staging repository for this release can be found at:
>>>>>>>
>>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1435
>>>>>>>
>>>>>>> The documentation corresponding to this release can be found at:
>>>>>>> https://dist.apache.org/repos/dist/dev/spark/v3.4.0-rc1-docs/
>>>>>>>
>>>>>>> The list of bug fixes going into 3.4.0 can be found at the following
>>>>>>> URL:
>>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/12351465
>>>>>>>
>>>>>>> This release is using the release script of the tag v3.4.0-rc1.
>>>>>>>
>>>>>>>
>>>>>>> FAQ
>>>>>>>
>>>>>>> =========================
>>>>>>> How can I help test this release?
>>>>>>> =========================
>>>>>>> If you are a Spark user, you can help us test this release by taking
>>>>>>> an existing Spark workload and running on this release candidate,
>>>>>>> then
>>>>>>> reporting any regressions.
>>>>>>>
>>>>>>> If you're working in PySpark you can set up a virtual env and install
>>>>>>> the current RC and see if anything important breaks, in the
>>>>>>> Java/Scala
>>>>>>> you can add the staging repository to your projects resolvers and
>>>>>>> test
>>>>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>>>>> you don't end up building with a out of date RC going forward).
>>>>>>>
>>>>>>> ===========================================
>>>>>>> What should happen to JIRA tickets still targeting 3.4.0?
>>>>>>> ===========================================
>>>>>>> The current list of open tickets targeted at 3.4.0 can be found at:
>>>>>>> https://issues.apache.org/jira/projects/SPARK and search for
>>>>>>> "Target Version/s" = 3.4.0
>>>>>>>
>>>>>>> Committers should look at those and triage. Extremely important bug
>>>>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>>>>> be worked on immediately. Everything else please retarget to an
>>>>>>> appropriate release.
>>>>>>>
>>>>>>> ==================
>>>>>>> But my bug isn't fixed?
>>>>>>> ==================
>>>>>>> In order to make timely releases, we will typically not hold the
>>>>>>> release unless the bug in question is a regression from the previous
>>>>>>> release. That being said, if there is something which is a regression
>>>>>>> that has not been correctly targeted please ping me or a committer to
>>>>>>> help target the issue.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Xinrong Meng
>>>>>>>
>>>>>>
>>
>> --
>> Bjørn Jørgensen
>> Vestre Aspehaug 4, 6010 Ålesund
>> Norge
>>
>> +47 480 94 297
>>
>
