Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]

2023-11-28 Thread via GitHub


HyukjinKwon closed pull request #44065: [SPARK-46149][ML][PYTHON][TESTS] Skip 
'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12
URL: https://github.com/apache/spark/pull/44065


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]

2023-11-28 Thread via GitHub


HyukjinKwon commented on PR #44065:
URL: https://github.com/apache/spark/pull/44065#issuecomment-1831144154

   Merged to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]

2023-11-28 Thread via GitHub


HyukjinKwon commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408649869


##
python/pyspark/tests/test_worker.py:
##
@@ -230,7 +230,7 @@ def conf(cls):
 _conf.set("spark.python.worker.faulthandler.enabled", "true")
 return _conf
 
-@unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with 
Python 3.12")
+@unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with 
Python 3.12")

Review Comment:
   Yeah, I think so actually. For that, let me take a separate look for 
individual (with reading 
https://docs.python.org/3/reference/expressions.html#value-comparisons 
carefully), and fix them if there's anything wrong (or that looks weird).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]

2023-11-28 Thread via GitHub


dongjoon-hyun commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408626232


##
python/pyspark/tests/test_worker.py:
##
@@ -230,7 +230,7 @@ def conf(cls):
 _conf.set("spark.python.worker.faulthandler.enabled", "true")
 return _conf
 
-@unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with 
Python 3.12")
+@unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with 
Python 3.12")

Review Comment:
   So, is it impossible to be **equal** because one is 3-item tuple and the 
other is 2-item tuple?
   ```
   >>> (3, 12, 0) == (3, 12)
   False
   
   >>> (3, 12, 0) <= (3, 12)
   False
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]

2023-11-28 Thread via GitHub


HyukjinKwon commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408617846


##
python/pyspark/tests/test_worker.py:
##
@@ -230,7 +230,7 @@ def conf(cls):
 _conf.set("spark.python.worker.faulthandler.enabled", "true")
 return _conf
 
-@unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with 
Python 3.12")
+@unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with 
Python 3.12")

Review Comment:
   sure - lemme check them



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]

2023-11-28 Thread via GitHub


HyukjinKwon commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408617846


##
python/pyspark/tests/test_worker.py:
##
@@ -230,7 +230,7 @@ def conf(cls):
 _conf.set("spark.python.worker.faulthandler.enabled", "true")
 return _conf
 
-@unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with 
Python 3.12")
+@unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with 
Python 3.12")

Review Comment:
   sure



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]

2023-11-28 Thread via GitHub


dongjoon-hyun commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408617098


##
python/pyspark/tests/test_worker.py:
##
@@ -230,7 +230,7 @@ def conf(cls):
 _conf.set("spark.python.worker.faulthandler.enabled", "true")
 return _conf
 
-@unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with 
Python 3.12")
+@unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with 
Python 3.12")

Review Comment:
   To @HyukjinKwon , do we need to reevaluate every instances like this? It 
looks a little counter-intuitive.
   
   ```
   $ git grep 'sys.version_info >'
   python/pyspark/pandas/__init__.py:if sys.version_info >= (3, 7):
   python/pyspark/pandas/frame.py:can_return_named_tuples = 
sys.version_info >= (3, 7) or len(self.columns) + index < 255
   python/pyspark/pandas/tests/computation/test_apply_func.py:if 
sys.version_info >= (3, 8):
   python/pyspark/pandas/tests/test_typedef.py:if sys.version_info 
>= (3, 8):
   python/pyspark/pandas/typedef/typehints.py:if sys.version_info >= (3, 8):
   python/pyspark/sql/connect/_typing.py:if sys.version_info >= (3, 8):
   python/pyspark/sql/connect/proto/base_pb2.pyi:if sys.version_info >= (3, 10):
   python/pyspark/sql/connect/proto/catalog_pb2.pyi:if sys.version_info >= (3, 
8):
   python/pyspark/sql/connect/proto/commands_pb2.pyi:if sys.version_info >= (3, 
10):
   python/pyspark/sql/connect/proto/common_pb2.pyi:if sys.version_info >= (3, 
8):
   python/pyspark/sql/connect/proto/example_plugins_pb2.pyi:if sys.version_info 
>= (3, 8):
   python/pyspark/sql/connect/proto/expressions_pb2.pyi:if sys.version_info >= 
(3, 10):
   python/pyspark/sql/connect/proto/relations_pb2.pyi:if sys.version_info >= 
(3, 10):
   python/pyspark/sql/connect/proto/types_pb2.pyi:if sys.version_info >= (3, 8):
   python/pyspark/tests/test_worker.py:@unittest.skipIf(sys.version_info > 
(3, 11), "SPARK-46130: Flaky with Python 3.12")
   ```
   
   ```
   $ git grep 'sys.version_info <'
   core/src/main/scala/org/apache/spark/TestUtils.scala:val pythonSnippet = 
s"import sys; sys.exit(sys.version_info < ($major, $minor, $reversion))"
   dev/run-tests:PYTHON_VERSION_CHECK=$(python3 -c 'import sys; 
print(sys.version_info < (3, 8, 0))')
   dev/run-tests-jenkins:PYTHON_VERSION_CHECK=$(python3 -c 'import sys; 
print(sys.version_info < (3, 8, 0))')
   python/pyspark/cloudpickle/compat.py:if sys.version_info < (3, 8):
   python/pyspark/pandas/typedef/typehints.py:if sys.version_info < (3, 11):
   python/pyspark/shuffle.py:if sys.version_info < (3, 11):
   python/pyspark/sql/tests/pandas/test_pandas_udf_typehints.py:
@unittest.skipIf(sys.version_info < (3, 9), "Type hinting generics require 
Python 3.9.")
   
python/pyspark/sql/tests/pandas/test_pandas_udf_typehints_with_future_annotations.py:
sys.version_info < (3, 9),
   python/run-tests:PYTHON_VERSION_CHECK=$(python3 -c 'import sys; 
print(sys.version_info < (3, 8, 0))')
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]

2023-11-28 Thread via GitHub


dongjoon-hyun commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408617098


##
python/pyspark/tests/test_worker.py:
##
@@ -230,7 +230,7 @@ def conf(cls):
 _conf.set("spark.python.worker.faulthandler.enabled", "true")
 return _conf
 
-@unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with 
Python 3.12")
+@unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with 
Python 3.12")

Review Comment:
   To @HyukjinKwon , do we need to reevaluate every instances like this? It 
looks a little counter-intuitive.
   
   ```
   $ git grep 'sys.version_info >'
   python/pyspark/pandas/__init__.py:if sys.version_info >= (3, 7):
   python/pyspark/pandas/frame.py:can_return_named_tuples = 
sys.version_info >= (3, 7) or len(self.columns) + index < 255
   python/pyspark/pandas/tests/computation/test_apply_func.py:if 
sys.version_info >= (3, 8):
   python/pyspark/pandas/tests/test_typedef.py:if sys.version_info 
>= (3, 8):
   python/pyspark/pandas/typedef/typehints.py:if sys.version_info >= (3, 8):
   python/pyspark/sql/connect/_typing.py:if sys.version_info >= (3, 8):
   python/pyspark/sql/connect/proto/base_pb2.pyi:if sys.version_info >= (3, 10):
   python/pyspark/sql/connect/proto/catalog_pb2.pyi:if sys.version_info >= (3, 
8):
   python/pyspark/sql/connect/proto/commands_pb2.pyi:if sys.version_info >= (3, 
10):
   python/pyspark/sql/connect/proto/common_pb2.pyi:if sys.version_info >= (3, 
8):
   python/pyspark/sql/connect/proto/example_plugins_pb2.pyi:if sys.version_info 
>= (3, 8):
   python/pyspark/sql/connect/proto/expressions_pb2.pyi:if sys.version_info >= 
(3, 10):
   python/pyspark/sql/connect/proto/relations_pb2.pyi:if sys.version_info >= 
(3, 10):
   python/pyspark/sql/connect/proto/types_pb2.pyi:if sys.version_info >= (3, 8):
   python/pyspark/tests/test_worker.py:@unittest.skipIf(sys.version_info > 
(3, 11), "SPARK-46130: Flaky with Python 3.12")
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]

2023-11-28 Thread via GitHub


dongjoon-hyun commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408617098


##
python/pyspark/tests/test_worker.py:
##
@@ -230,7 +230,7 @@ def conf(cls):
 _conf.set("spark.python.worker.faulthandler.enabled", "true")
 return _conf
 
-@unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with 
Python 3.12")
+@unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with 
Python 3.12")

Review Comment:
   To @HyukjinKwon , do we need to reevaluate every instances?
   
   ```
   $ git grep 'sys.version_info >'
   python/pyspark/pandas/__init__.py:if sys.version_info >= (3, 7):
   python/pyspark/pandas/frame.py:can_return_named_tuples = 
sys.version_info >= (3, 7) or len(self.columns) + index < 255
   python/pyspark/pandas/tests/computation/test_apply_func.py:if 
sys.version_info >= (3, 8):
   python/pyspark/pandas/tests/test_typedef.py:if sys.version_info 
>= (3, 8):
   python/pyspark/pandas/typedef/typehints.py:if sys.version_info >= (3, 8):
   python/pyspark/sql/connect/_typing.py:if sys.version_info >= (3, 8):
   python/pyspark/sql/connect/proto/base_pb2.pyi:if sys.version_info >= (3, 10):
   python/pyspark/sql/connect/proto/catalog_pb2.pyi:if sys.version_info >= (3, 
8):
   python/pyspark/sql/connect/proto/commands_pb2.pyi:if sys.version_info >= (3, 
10):
   python/pyspark/sql/connect/proto/common_pb2.pyi:if sys.version_info >= (3, 
8):
   python/pyspark/sql/connect/proto/example_plugins_pb2.pyi:if sys.version_info 
>= (3, 8):
   python/pyspark/sql/connect/proto/expressions_pb2.pyi:if sys.version_info >= 
(3, 10):
   python/pyspark/sql/connect/proto/relations_pb2.pyi:if sys.version_info >= 
(3, 10):
   python/pyspark/sql/connect/proto/types_pb2.pyi:if sys.version_info >= (3, 8):
   python/pyspark/tests/test_worker.py:@unittest.skipIf(sys.version_info > 
(3, 11), "SPARK-46130: Flaky with Python 3.12")
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]

2023-11-28 Thread via GitHub


dongjoon-hyun commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408615284


##
python/pyspark/tests/test_worker.py:
##
@@ -230,7 +230,7 @@ def conf(cls):
 _conf.set("spark.python.worker.faulthandler.enabled", "true")
 return _conf
 
-@unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with 
Python 3.12")
+@unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with 
Python 3.12")

Review Comment:
   Oh, interesting.



##
python/pyspark/ml/torch/tests/test_distributor.py:
##
@@ -398,6 +398,9 @@ def test_local_file_with_pytorch(self) -> None:
 test_file_path, learning_rate_str
 )
 
+@unittest.skipIf(
+sys.version_info > (3, 12), "SPARK-46078: Fails with dev torch with 
Python 3.12"

Review Comment:
   Is this correct? The new commit seems to be unable to skip 3.12.0.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]

2023-11-28 Thread via GitHub


dongjoon-hyun commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408614954


##
python/pyspark/ml/torch/tests/test_distributor.py:
##
@@ -398,6 +398,9 @@ def test_local_file_with_pytorch(self) -> None:
 test_file_path, learning_rate_str
 )
 
+@unittest.skipIf(
+sys.version_info > (3, 12), "SPARK-46078: Fails with dev torch with 
Python 3.12"

Review Comment:
   Is this correct? The new commit seems to be unable to skip 3.12.0.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]

2023-11-28 Thread via GitHub


HyukjinKwon commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408614514


##
python/pyspark/tests/test_worker.py:
##
@@ -230,7 +230,7 @@ def conf(cls):
 _conf.set("spark.python.worker.faulthandler.enabled", "true")
 return _conf
 
-@unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with 
Python 3.12")
+@unittest.skipIf(sys.version_info > (3, 12), "SPARK-46130: Flaky with 
Python 3.12")

Review Comment:
   It was my bad. I am fixing it together:
   
   ```
   Python 3.11.5 (main, Sep 11 2023, 08:19:27) [Clang 14.0.6 ] on darwin
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import sys
   >>> sys.version_info > (3, 12)
   False
   >>> sys.version_info > (3, 11)
   True
   ```
   
   ```
   >>> (3, 12, 0) > (3, 12)
   True
   >>> (3, 11, 0) > (3, 12)
   False
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]

2023-11-28 Thread via GitHub


HyukjinKwon commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408613472


##
python/pyspark/ml/torch/tests/test_distributor.py:
##
@@ -398,6 +398,9 @@ def test_local_file_with_pytorch(self) -> None:
 test_file_path, learning_rate_str
 )
 
+@unittest.skipIf(
+sys.version_info > (3, 11), "SPARK-46078: Fails with dev torch with 
Python 3.12"

Review Comment:
   ```suggestion
   sys.version_info > (3, 12), "SPARK-46078: Fails with dev torch with 
Python 3.12"
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]

2023-11-28 Thread via GitHub


HyukjinKwon commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408613472


##
python/pyspark/ml/torch/tests/test_distributor.py:
##
@@ -398,6 +398,9 @@ def test_local_file_with_pytorch(self) -> None:
 test_file_path, learning_rate_str
 )
 
+@unittest.skipIf(
+sys.version_info > (3, 11), "SPARK-46078: Fails with dev torch with 
Python 3.12"

Review Comment:
   ```suggestion
   sys.version_info > (3, 12), "SPARK-46078: Fails with dev torch with 
Python 3.12"
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]

2023-11-28 Thread via GitHub


HyukjinKwon commented on code in PR #44065:
URL: https://github.com/apache/spark/pull/44065#discussion_r1408612421


##
python/pyspark/ml/torch/tests/test_distributor.py:
##
@@ -398,6 +398,9 @@ def test_local_file_with_pytorch(self) -> None:
 test_file_path, learning_rate_str
 )
 
+@unittest.skipIf(
+sys.version_info > (3, 11), "SPARK-46078: Fails with dev torch with 
Python 3.12"

Review Comment:
   ```suggestion
   sys.version_info < (3, 12), "SPARK-46078: Fails with dev torch with 
Python 3.12"
   ```



##
python/pyspark/ml/torch/tests/test_distributor.py:
##
@@ -398,6 +398,9 @@ def test_local_file_with_pytorch(self) -> None:
 test_file_path, learning_rate_str
 )
 
+@unittest.skipIf(
+sys.version_info > (3, 11), "SPARK-46078: Fails with dev torch with 
Python 3.12"

Review Comment:
   ```suggestion
   sys.version_info < (3, 12), "SPARK-46078: Fails with dev torch with 
Python 3.12"
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]

2023-11-28 Thread via GitHub


dongjoon-hyun commented on PR #44065:
URL: https://github.com/apache/spark/pull/44065#issuecomment-1831055718

   Thank you for taking care of them, @HyukjinKwon !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



Re: [PR] [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12 [spark]

2023-11-28 Thread via GitHub


HyukjinKwon commented on PR #44065:
URL: https://github.com/apache/spark/pull/44065#issuecomment-1831051264

   Build: 
https://github.com/HyukjinKwon/spark/actions/runs/7026809016/job/19120174720


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org