unsubscribe

[김병찬]

제목unsubscribe

Re: Building Spark to run PySpark Tests?

[Sean Owen]

It's not clear what error you're facing from this info (ConnectionError
could mean lots of things), so would be hard to generalize answers. How
much mem do you have on your Mac?
-Xmx2g sounds low, but also probably doesn't matter much.
Spark builds work on my Mac, FWIW.

On Thu, Jan 19, 2023 at 10:15 AM Adam Chhina  wrote:

> Hmm, would there be a list of common env issues that would interfere with
> builds? Looking up the error message, it seemed like often the issue was
> OOM by the JVM process. I’m not sure if that’s what’s happening here, since
> during the build and setting up the tests the config should have allocated
> enough memory?
>
> I’ve been just trying to follow the build docs, and so far I’m running as
> such:
>
> > git clone --branch v3.2.3 https://github.com/apache/spark.git
> > cd spark
> > export MAVEN_OPTS="-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g” // was
> unset, but set to be safe
> > export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES // I saw in the
> developer tools that some pyspark tests were having issues on macOS
> > export JAVA_HOME=`/usr/libexec/java_home -v 11`
> > ./build/mvn -DskipTests clean package -Phive
> > ./python/run-tests --python-executables --testnames
> ‘pyspark.tests.test_broadcast'
>
> > java -version
>
> openjdk version "11.0.17" 2022-10-18
>
> OpenJDK Runtime Environment Homebrew (build 11.0.17+0)
>
> OpenJDK 64-Bit Server VM Homebrew (build 11.0.17+0, mixed mode)
>
>
> > OS
>
> Ventura 13.1 (22C65)
>
>
> Best,
>
>
> Adam Chhina
>
> On Jan 18, 2023, at 6:50 PM, Sean Owen  wrote:
>
> Release _branches_ are tested as commits arrive to the branch, yes. That's
> what you see at https://github.com/apache/spark/actions
> Released versions are fixed, they don't change, and were also manually
> tested before release, so no they are not re-tested; there is no need.
>
> You presumably have some local env issue, because the source of Spark
> 3.2.3 was passing CI/CD at time of release as well as manual tests of the
> PMC.
>
>
> On Wed, Jan 18, 2023 at 5:24 PM Adam Chhina  wrote:
>
>> Hi Sean,
>>
>> That’s fair in regards to 3.3.x being the current release branch. I’m not
>> familiar with the testing schedule, but I had assumed all currently
>> supported release versions would have some nightly/weekly tests ran; is
>> that not the case? I only ask, as when I when I’m seeing these test
>> failures, I assumed these were either known/unknown from some recurring
>> testing pipeline.
>>
>> Also, unfortunately using v3.2.3 also had the same test failures.
>>
>> > git clone --branch v3.2.3 https://github.com/apache/spark.git
>>
>> I’ve posted the traceback below for one of the ran tests. At the end it
>> mentioned to check the logs - `see logs`. However I wasn’t sure whether
>> that just meant the traceback or some more detailed logs elsewhere? I
>> wasn’t able to see any files that looked relevant running `find . -name
>> “*logs*”` afterwards. Sorry if I’m missing something obvious.
>>
>> ```
>> test_broadcast_no_encryption (pyspark.tests.test_broadcast.BroadcastTest)
>> ... ERROR
>> test_broadcast_value_against_gc
>> (pyspark.tests.test_broadcast.BroadcastTest) ... ERROR
>> test_broadcast_value_driver_encryption
>> (pyspark.tests.test_broadcast.BroadcastTest) ... ERROR
>> test_broadcast_value_driver_no_encryption
>> (pyspark.tests.test_broadcast.BroadcastTest) ... ERROR
>> test_broadcast_with_encryption
>> (pyspark.tests.test_broadcast.BroadcastTest) ... ERROR
>>
>> ==
>> ERROR: test_broadcast_with_encryption
>> (pyspark.tests.test_broadcast.BroadcastTest)
>> --
>> Traceback (most recent call last):
>>   File "$path/spark/python/pyspark/tests/test_broadcast.py", line 67, in
>> test_broadcast_with_encryption
>> self._test_multiple_broadcasts(("spark.io.encryption.enabled",
>> "true"))
>>   File "$path/spark/python/pyspark/tests/test_broadcast.py", line 58, in
>> _test_multiple_broadcasts
>> conf = SparkConf()
>>   File "$path/spark/python/pyspark/conf.py", line 120, in __init__
>> self._jconf = _jvm.SparkConf(loadDefaults)
>>   File
>> "$path/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line
>> 1709, in __getattr__
>> answer = self._gateway_client.send_command(
>>   File
>> "$path/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line
>> 1036, in send_command
>> connection = self._get_connection()
>>   File
>> "$path/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", line
>> 284, in _get_connection
>> connection = self._create_new_connection()
>>   File
>> "$path/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", line
>> 291, in _create_new_connection
>> connection.connect_to_java_server()
>>   File
>> "$path/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", line
>> 438, in connect_to_java_server
>> self.socket.connect((self.java_address, self.java_por

Re: Building Spark to run PySpark Tests?

[Adam Chhina]

Hmm, would there be a list of common env issues that would interfere with 
builds? Looking up the error message, it seemed like often the issue was OOM by 
the JVM process. I’m not sure if that’s what’s happening here, since during the 
build and setting up the tests the config should have allocated enough memory?

I’ve been just trying to follow the build docs, and so far I’m running as such:

> git clone --branch v3.2.3 https://github.com/apache/spark.git 
> cd spark
> export MAVEN_OPTS="-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g” // was unset, 
> but set to be safe
> export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES // I saw in the developer 
> tools that some pyspark tests were having issues on macOS
> export JAVA_HOME=`/usr/libexec/java_home -v 11`
> ./build/mvn -DskipTests clean package -Phive
> ./python/run-tests --python-executables --testnames 
> ‘pyspark.tests.test_broadcast'

> java -version
openjdk version "11.0.17" 2022-10-18
OpenJDK Runtime Environment Homebrew (build 11.0.17+0)
OpenJDK 64-Bit Server VM Homebrew (build 11.0.17+0, mixed mode)

> OS
Ventura 13.1 (22C65)

Best,

Adam Chhina

> On Jan 18, 2023, at 6:50 PM, Sean Owen  wrote:
> 
> Release _branches_ are tested as commits arrive to the branch, yes. That's 
> what you see at https://github.com/apache/spark/actions
> Released versions are fixed, they don't change, and were also manually tested 
> before release, so no they are not re-tested; there is no need.
> 
> You presumably have some local env issue, because the source of Spark 3.2.3 
> was passing CI/CD at time of release as well as manual tests of the PMC.
> 
> 
> On Wed, Jan 18, 2023 at 5:24 PM Adam Chhina  > wrote:
>> Hi Sean,
>> 
>> That’s fair in regards to 3.3.x being the current release branch. I’m not 
>> familiar with the testing schedule, but I had assumed all currently 
>> supported release versions would have some nightly/weekly tests ran; is that 
>> not the case? I only ask, as when I when I’m seeing these test failures, I 
>> assumed these were either known/unknown from some recurring testing pipeline.
>> 
>> Also, unfortunately using v3.2.3 also had the same test failures.
>> 
>> > git clone --branch v3.2.3 https://github.com/apache/spark.git
>> 
>> I’ve posted the traceback below for one of the ran tests. At the end it 
>> mentioned to check the logs - `see logs`. However I wasn’t sure whether that 
>> just meant the traceback or some more detailed logs elsewhere? I wasn’t able 
>> to see any files that looked relevant running `find . -name “*logs*”` 
>> afterwards. Sorry if I’m missing something obvious.
>> 
>> ```
>> test_broadcast_no_encryption (pyspark.tests.test_broadcast.BroadcastTest) 
>> ... ERROR
>> test_broadcast_value_against_gc (pyspark.tests.test_broadcast.BroadcastTest) 
>> ... ERROR
>> test_broadcast_value_driver_encryption 
>> (pyspark.tests.test_broadcast.BroadcastTest) ... ERROR
>> test_broadcast_value_driver_no_encryption 
>> (pyspark.tests.test_broadcast.BroadcastTest) ... ERROR
>> test_broadcast_with_encryption (pyspark.tests.test_broadcast.BroadcastTest) 
>> ... ERROR
>> 
>> ==
>> ERROR: test_broadcast_with_encryption 
>> (pyspark.tests.test_broadcast.BroadcastTest)
>> --
>> Traceback (most recent call last):
>>   File "$path/spark/python/pyspark/tests/test_broadcast.py", line 67, in 
>> test_broadcast_with_encryption
>> self._test_multiple_broadcasts(("spark.io.encryption.enabled", "true"))
>>   File "$path/spark/python/pyspark/tests/test_broadcast.py", line 58, in 
>> _test_multiple_broadcasts
>> conf = SparkConf()
>>   File "$path/spark/python/pyspark/conf.py", line 120, in __init__
>> self._jconf = _jvm.SparkConf(loadDefaults)
>>   File "$path/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", 
>> line 1709, in __getattr__
>> answer = self._gateway_client.send_command(
>>   File "$path/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", 
>> line 1036, in send_command
>> connection = self._get_connection()
>>   File "$path/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", 
>> line 284, in _get_connection
>> connection = self._create_new_connection()
>>   File "$path/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", 
>> line 291, in _create_new_connection
>> connection.connect_to_java_server()
>>   File "$path/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", 
>> line 438, in connect_to_java_server
>> self.socket.connect((self.java_address, self.java_port))
>> ConnectionRefusedError: [Errno 61] Connection refused
>> 
>> --
>> Ran 7 tests in 12.950s
>> 
>> FAILED (errors=7)
>> sys:1: ResourceWarning: unclosed file <_io.BufferedWriter name=4>
>> 
>> Had test failures in pyspark.tests.test_broadcast with 
>> /usr/local/bin/pyt

Re: Can you create an apache jira account for me? Thanks very much!

[Sean Owen]

I can help offline. Send me your preferred JIRA user name.

On Thu, Jan 19, 2023 at 7:12 AM Wei Yan  wrote:

> When I tried to sign up through this site:
> https://issues.apache.org/jira/secure/Signup!default.jspa
> I got an error message:"Sorry, you can't sign up to this Jira site at the
> moment as it's private."
> and I got a suggestion:"If you think you should be able to sign up then
> you should let the Jira administrator know".
> So I think I need some help.
>
>
>

Can you create an apache jira account for me? Thanks very much!

[Wei Yan]

When I tried to sign up through this site:
https://issues.apache.org/jira/secure/Signup!default.jspa
I got an error message:"Sorry, you can't sign up to this Jira site at the
moment as it's private."
and I got a suggestion:"If you think you should be able to sign up then you
should let the Jira administrator know".
So I think I need some help.