[
https://issues.apache.org/jira/browse/SPARK-60142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yen-Hsiang, Wang updated SPARK-60142:
-------------------------------------
Priority: Minor (was: Major)
> Add unit tests for vector field type in
> _SparkPartitionTorchDataset._get_field_converter
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-60142
> URL: https://issues.apache.org/jira/browse/SPARK-60142
> Project: Spark
> Issue Type: Test
> Components: ML
> Affects Versions: 4.4.0
> Reporter: Yen-Hsiang, Wang
> Priority: Minor
>
> Problem
> _SparkPartitionTorchDataset._get_field_converter in
> python/pyspark/ml/torch/data.py handles three branches: vector, numeric/array
> types, and
> unsupported types (raises ValueError). The vector branch — which converts
> both dense and sparse MLlib vectors from their Arrow/pandas dict
> representation — is currently not covered by any test.
> The only existing test
> (TorchDistributorDataLoaderUnitTests.test_data_loader) is an integration test
> that exercises the full TorchDistributor
> pipeline and has been skipped (@unittest.skip) since SPARK-50864 due to
> slowness, leaving the vector converter logic without any automated
> coverage.
> What
> Add lightweight unit tests directly against
> _SparkPartitionTorchDataset._get_field_converter for the vector field type,
> covering:
> - Dense vector (type == 1)
> - Sparse vector (type == 0) with non-empty indices
> - Sparse vector with empty indices (edge case)
> Since _get_field_converter is a @staticmethod and the vector converter only
> depends on NumPy, no SparkSession is needed — the tests run fast
> and in isolation.
> Why
> The vector branch contains non-trivial logic (sparse → dense expansion via
> index assignment). A bug there would silently produce wrong training
> data without any test catching it.
> ---
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
