This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 27e2411b99f8 [SPARK-58766][PYTHON][TEST][FOLLOWUP] Render NaT-sentinel
time64 scalars in golden files
27e2411b99f8 is described below
commit 27e2411b99f8eaee3d563bcdb445bb91630f8d0f
Author: Spenser Sun <[email protected]>
AuthorDate: Tue Aug 25 13:00:09 2026 +0800
[SPARK-58766][PYTHON][TEST][FOLLOWUP] Render NaT-sentinel time64 scalars in
golden files
### What changes were proposed in this pull request?
Add a `ValueError` branch to `_scalar_str` in
`python/pyspark/testing/goldenutils.py` so a `time64[ns]` scalar holding
INT64_MIN renders as the marker `NaT collision`, beside the existing `temporal
overflow` and raw-bytes fallbacks, and regenerate the four affected
`from_pandas` type-scalar golden cells (`int64:standard` and `Int64:standard` x
`time64[ns]`, safe and unsafe).
Rendering a `time64[ns]` scalar goes through pandas, because Python's
`datetime.time` has only microsecond resolution. pandas reserves INT64_MIN as
its NaT sentinel, so it refuses that one value with `ValueError: NaTType does
not support time`. The branch is gated on both the type and the value, so any
other `ValueError` still propagates.
### Why are the changes needed?
`PyArrowArrayFromPandasTypeScalarTests` currently fails on master. The four
cells above read `ERRValueError`, which by convention means the conversion
raised — but the conversion succeeds and only the rendering failed. The broad
`try/except` in the cell helper recorded the rendering failure as if it were a
conversion error, so narrowing that `try/except` in #58091 turned the
mislabelling into a test error.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
`python/pyspark/tests/upstream/pyarrow/` passes (63 passed, 1 skipped), and
a sweep over pyarrow 18-25 x pandas 2/3 is 16/16 with no `overrides` entries.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
Closes #58266 from Spenserrrr/golden-nat-sentinel.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 08e97ded9430aa7dc931936ebbfe5293404e6cba)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/testing/goldenutils.py | 16 +++++++--
..._pyarrow_array_from_pandas_type_scalar_safe.csv | 4 +--
...n_pyarrow_array_from_pandas_type_scalar_safe.md | 42 +++++++++++-----------
...yarrow_array_from_pandas_type_scalar_unsafe.csv | 4 +--
...pyarrow_array_from_pandas_type_scalar_unsafe.md | 42 +++++++++++-----------
5 files changed, 59 insertions(+), 49 deletions(-)
diff --git a/python/pyspark/testing/goldenutils.py
b/python/pyspark/testing/goldenutils.py
index f9ba1f66fd98..6d60c4a8be92 100644
--- a/python/pyspark/testing/goldenutils.py
+++ b/python/pyspark/testing/goldenutils.py
@@ -316,13 +316,16 @@ class GoldenFileTestMixin:
"""
Render one PyArrow scalar for a golden cell via PyArrow's own
``str(scalar)``.
- Some values are valid in Arrow yet unrenderable via ``str``:
+ Some values that Arrow stores are unrenderable via ``str``:
- a temporal value past Python's ``datetime`` range (e.g. date32 past
year 9999)
raises ``OverflowError`` -> the marker ``temporal overflow``;
- an unsafe ``binary``->``string`` cast relabels bytes without UTF-8
validation,
so non-UTF-8 bytes raise ``UnicodeDecodeError`` -> the raw bytes, so
the golden
- still tracks what Arrow stored for them.
- An unexpected error (non-temporal overflow, non-string decode error)
propagates.
+ still tracks what Arrow stored for them;
+ - a nanosecond ``time64`` holding INT64_MIN collides with pandas' NaT
sentinel and
+ raises ``ValueError`` -> the marker ``NaT collision``.
+ An unexpected error (non-temporal overflow, non-string decode error,
or a
+ ValueError from any other value) propagates.
"""
try:
return str(scalar).replace("\x00", "\\0")
@@ -341,6 +344,13 @@ class GoldenFileTestMixin:
if pa.types.is_string(scalar.type) or
pa.types.is_large_string(scalar.type):
return repr(scalar.as_buffer().to_pybytes())
raise
+ except ValueError:
+ # A nanosecond time64 renders through pandas (Python's ``time`` is
microsecond
+ # resolution), and pandas reads INT64_MIN as its NaT sentinel, so
it refuses
+ # that one value. Any other ValueError is unexpected, so re-raise.
+ if pa.types.is_time(scalar.type) and scalar.value == pd.NaT.value:
+ return "NaT collision"
+ raise
@classmethod
def repr_arrow_value(
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.csv
index a054d3a83712..0ef13693498d 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.csv
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.csv
@@ -1,5 +1,5 @@
source \ target int8 int64 float32 timestamp[us] date32[day]
duration[us] time64[ns] string binary
-int64:standard ERR@ArrowInvalid [0, 1, -1, 9223372036854775807,
-9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00,
1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow,
temporal overflow]@timestamp[us] ERR@ArrowNotImplementedError
[0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us]
ERR@ValueError ERR@ArrowTypeError [b'', b'\x01',
b'\xff\xff\xff\xff\xff\xff\xff\xff', b'\xff\x [...]
+int64:standard ERR@ArrowInvalid [0, 1, -1, 9223372036854775807,
-9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00,
1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow,
temporal overflow]@timestamp[us] ERR@ArrowNotImplementedError
[0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] [00:00:00,
00:00:00, 23:59:59.999999, 23:47:16.854775, NaT collision]@time64[ns]
ERR@ArrowType [...]
float64:standard ERR@ArrowInvalid ERR@ArrowInvalid [0.0,
1.5, -1.5]@float32 ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowTypeError [b'', b'', b'']@binary
bool:standard [1, 0, 1]@int8 [1, 0, 1]@int64 [1.0, 0.0, 1.0]@float32
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowTypeError [b'\x01', b'', b'\x01']@binary
object:string ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError [hello, world, ]@string
[b'hello', b'world', b'']@binary
@@ -8,7 +8,7 @@ date:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeEr
object:datetime ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid [2024-06-15 18:30:00]@timestamp[us]
[2024-06-15]@date32[day] ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError
object:timedelta ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError [1 day,
2:00:00]@duration[us] ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError
time:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError [18:30:45]@time64[ns] ERR@ArrowTypeError
ERR@ArrowTypeError
-Int64:standard ERR@ArrowInvalid [0, 1, 9223372036854775807,
-9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00,
1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us]
ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us]
ERR@ValueError ERR@ArrowTypeError [b'', b'\x01',
b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
+Int64:standard ERR@ArrowInvalid [0, 1, 9223372036854775807,
-9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00,
1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us]
ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us]
[00:00:00, 00:00:00, 23:47:16.854775, NaT collision]@time64[ns]
ERR@ArrowTypeError [b'', b'\x01', b'\xff\xff\xff\xff\xff\xff\xff\x7f',
b'']@binary
string[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [hello,
world]@string [b'hello', b'world']@binary
large_binary[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError [hello, world]@string [b'hello',
b'world']@binary
int64:overflow ERR@ArrowInvalid [300, 1]@int64 [300.0, 1.0]@float32
[1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us]
ERR@ArrowNotImplementedError [0:00:00.000300, 0:00:00.000001]@duration[us]
[00:00:00, 00:00:00]@time64[ns] ERR@ArrowTypeError [b',\x01',
b'\x01']@binary
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.md
index 65476ad0e9e9..2157f49343ac 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.md
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.md
@@ -1,21 +1,21 @@
-| source \ target | int8 | int64
| float32 |
timestamp[us]
| date32[day]
| duration[us]
| time64[ns]
[...]
-|--------------------------------|------------------------------|-------------------------------------------------------------|------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|-------------------------------
[...]
-| int64:standard | ERR@ArrowInvalid | [0, 1, -1,
9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid
| [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999,
temporal overflow, temporal overflow]@timestamp[us] |
ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, -1 day,
23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days,
19:59:05.224192]@duration[us] | ERR@ValueError [...]
-| float64:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | [0.0, 1.5,
-1.5]@float32 | ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
-| bool:standard | [1, 0, 1]@int8 | [1, 0,
1]@int64 | [1.0, 0.0, 1.0]@float32
| ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
-| object:string | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
|
ERR@ArrowTypeError | ERR@ArrowTypeError
| ERR@ArrowTypeError [...]
-| object:bytes | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
|
ERR@ArrowTypeError | ERR@ArrowTypeError
| ERR@ArrowTypeError [...]
-| date:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
|
[2024-06-15]@date32[day] | ERR@ArrowTypeError
| ERR@ArrowTypeError [...]
-| object:datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| [2024-06-15 18:30:00]@timestamp[us]
|
[2024-06-15]@date32[day] | ERR@ArrowTypeError
| ERR@ArrowTypeError [...]
-| object:timedelta | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
|
ERR@ArrowTypeError | [1 day, 2:00:00]@duration[us]
| ERR@ArrowTypeError [...]
-| time:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
|
ERR@ArrowTypeError | ERR@ArrowTypeError
| [18:30:45]@time64[ns] [...]
-| Int64:standard | ERR@ArrowInvalid | [0, 1,
9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid
| [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, temporal overflow,
temporal overflow]@timestamp[us] |
ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, 106751991
days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us]
| ERR@ValueError [...]
-| string[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowInvalid
|
ERR@ArrowInvalid | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
-| large_binary[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
-| int64:overflow | ERR@ArrowInvalid | [300,
1]@int64 | [300.0, 1.0]@float32
| [1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us]
|
ERR@ArrowNotImplementedError | [0:00:00.000300,
0:00:00.000001]@duration[us]
| [00:00:00, 00:00:00]@time64[ns [...]
-| float64:fractional | ERR@ArrowInvalid |
ERR@ArrowInvalid | [1.5,
2.5]@float32 | ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
-| float64:infinity | ERR@ArrowInvalid |
ERR@ArrowInvalid | [inf,
1.0]@float32 | ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
-| float64:precision | ERR@ArrowInvalid |
ERR@ArrowInvalid |
[1.1234568357467651]@float32 | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
-| datetime64[ns]:sub-us | ERR@ArrowNotImplementedError |
[1704067200000000123]@int64 |
ERR@ArrowNotImplementedError | ERR@ArrowInvalid
| [2024-01-01]@date32[day] | ERR@ArrowNotImplementedError
| [00:00:00]@time64[ns] [...]
-| object:date-then-datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01,
2024-01-01]@date32[day] | ERR@ArrowTypeError
|
ERR@ArrowTypeError [...]
-| object:datetime-then-date | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01,
2024-01-01]@date32[day] | ERR@ArrowTypeError
|
ERR@ArrowTypeError [...]
\ No newline at end of file
+| source \ target | int8 | int64
| float32 |
timestamp[us]
| date32[day]
| duration[us]
| time64[ns]
[...]
+|--------------------------------|------------------------------|-------------------------------------------------------------|------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|-------------------------------
[...]
+| int64:standard | ERR@ArrowInvalid | [0, 1, -1,
9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid
| [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999,
temporal overflow, temporal overflow]@timestamp[us] |
ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, -1 day,
23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days,
19:59:05.224192]@duration[us] | [00:00:00, 00:00:00, 23:59:59. [...]
+| float64:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | [0.0, 1.5,
-1.5]@float32 | ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
+| bool:standard | [1, 0, 1]@int8 | [1, 0,
1]@int64 | [1.0, 0.0, 1.0]@float32
| ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
+| object:string | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
|
ERR@ArrowTypeError | ERR@ArrowTypeError
| ERR@ArrowTypeError [...]
+| object:bytes | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
|
ERR@ArrowTypeError | ERR@ArrowTypeError
| ERR@ArrowTypeError [...]
+| date:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
|
[2024-06-15]@date32[day] | ERR@ArrowTypeError
| ERR@ArrowTypeError [...]
+| object:datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| [2024-06-15 18:30:00]@timestamp[us]
|
[2024-06-15]@date32[day] | ERR@ArrowTypeError
| ERR@ArrowTypeError [...]
+| object:timedelta | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
|
ERR@ArrowTypeError | [1 day, 2:00:00]@duration[us]
| ERR@ArrowTypeError [...]
+| time:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
|
ERR@ArrowTypeError | ERR@ArrowTypeError
| [18:30:45]@time64[ns] [...]
+| Int64:standard | ERR@ArrowInvalid | [0, 1,
9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid
| [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, temporal overflow,
temporal overflow]@timestamp[us] |
ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, 106751991
days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us]
| [00:00:00, 00:00:00, 23:47:16. [...]
+| string[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowInvalid
|
ERR@ArrowInvalid | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
+| large_binary[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
+| int64:overflow | ERR@ArrowInvalid | [300,
1]@int64 | [300.0, 1.0]@float32
| [1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us]
|
ERR@ArrowNotImplementedError | [0:00:00.000300,
0:00:00.000001]@duration[us]
| [00:00:00, 00:00:00]@time64[ns [...]
+| float64:fractional | ERR@ArrowInvalid |
ERR@ArrowInvalid | [1.5,
2.5]@float32 | ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
+| float64:infinity | ERR@ArrowInvalid |
ERR@ArrowInvalid | [inf,
1.0]@float32 | ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
+| float64:precision | ERR@ArrowInvalid |
ERR@ArrowInvalid |
[1.1234568357467651]@float32 | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
+| datetime64[ns]:sub-us | ERR@ArrowNotImplementedError |
[1704067200000000123]@int64 |
ERR@ArrowNotImplementedError | ERR@ArrowInvalid
| [2024-01-01]@date32[day] | ERR@ArrowNotImplementedError
| [00:00:00]@time64[ns] [...]
+| object:date-then-datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01,
2024-01-01]@date32[day] | ERR@ArrowTypeError
|
ERR@ArrowTypeError [...]
+| object:datetime-then-date | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01,
2024-01-01]@date32[day] | ERR@ArrowTypeError
|
ERR@ArrowTypeError [...]
\ No newline at end of file
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.csv
index b04f68c6e346..32a24d6a661c 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.csv
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.csv
@@ -1,5 +1,5 @@
source \ target int8 int64 float32 timestamp[us] date32[day]
duration[us] time64[ns] string binary
-int64:standard [0, 1, -1, -1, 0]@int8 [0, 1, -1, 9223372036854775807,
-9223372036854775808]@int64 [0.0, 1.0, -1.0, 9.223372036854776e+18,
-9.223372036854776e+18]@float32 [1970-01-01 00:00:00, 1970-01-01
00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow, temporal
overflow]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00,
0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days, 4:00:54.775807,
-106751992 days, 19:59:05.224192]@duration[us] ERR@ValueError
ERR@ArrowTypeError [...]
+int64:standard [0, 1, -1, -1, 0]@int8 [0, 1, -1, 9223372036854775807,
-9223372036854775808]@int64 [0.0, 1.0, -1.0, 9.223372036854776e+18,
-9.223372036854776e+18]@float32 [1970-01-01 00:00:00, 1970-01-01
00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow, temporal
overflow]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00,
0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days, 4:00:54.775807,
-106751992 days, 19:59:05.224192]@duration[us] [00:00:00, 00:00:00,
23:59:59.999 [...]
float64:standard [0, 1, -1]@int8 [0, 1, -1]@int64 [0.0, 1.5,
-1.5]@float32 ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowTypeError [b'', b'', b'']@binary
bool:standard [1, 0, 1]@int8 [1, 0, 1]@int64 [1.0, 0.0, 1.0]@float32
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowTypeError [b'\x01', b'', b'\x01']@binary
object:string ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError [hello, world, ]@string
[b'hello', b'world', b'']@binary
@@ -8,7 +8,7 @@ date:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeEr
object:datetime ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid [2024-06-15 18:30:00]@timestamp[us]
[2024-06-15]@date32[day] ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError
object:timedelta ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError [1 day,
2:00:00]@duration[us] ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError
time:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError [18:30:45]@time64[ns] ERR@ArrowTypeError
ERR@ArrowTypeError
-Int64:standard ERR@ArrowInvalid [0, 1, 9223372036854775807,
-9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00,
1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us]
ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us]
ERR@ValueError ERR@ArrowTypeError [b'', b'\x01',
b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
+Int64:standard ERR@ArrowInvalid [0, 1, 9223372036854775807,
-9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00,
1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us]
ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us]
[00:00:00, 00:00:00, 23:47:16.854775, NaT collision]@time64[ns]
ERR@ArrowTypeError [b'', b'\x01', b'\xff\xff\xff\xff\xff\xff\xff\x7f',
b'']@binary
string[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [hello,
world]@string [b'hello', b'world']@binary
large_binary[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError [hello, world]@string [b'hello',
b'world']@binary
int64:overflow [44, 1]@int8 [300, 1]@int64 [300.0, 1.0]@float32
[1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us]
ERR@ArrowNotImplementedError [0:00:00.000300, 0:00:00.000001]@duration[us]
[00:00:00, 00:00:00]@time64[ns] ERR@ArrowTypeError [b',\x01',
b'\x01']@binary
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.md
index 05ddfff08f59..a6f12040dc1d 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.md
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.md
@@ -1,21 +1,21 @@
-| source \ target | int8 | int64
| float32
| timestamp[us]
| date32[day] | duration[us]
[...]
-|--------------------------------|------------------------------|-------------------------------------------------------------|-------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------|------------------------------------------------------------------------------------------------------------------------
[...]
-| int64:standard | [0, 1, -1, -1, 0]@int8 | [0, 1, -1,
9223372036854775807, -9223372036854775808]@int64 | [0.0, 1.0, -1.0,
9.223372036854776e+18, -9.223372036854776e+18]@float32 | [1970-01-01 00:00:00,
1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow,
temporal overflow]@timestamp[us] | ERR@ArrowNotImplementedError |
[0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@du [...]
-| float64:standard | [0, 1, -1]@int8 | [0, 1,
-1]@int64 | [0.0, 1.5, -1.5]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
[...]
-| bool:standard | [1, 0, 1]@int8 | [1, 0,
1]@int64 | [1.0, 0.0, 1.0]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
[...]
-| object:string | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError |
ERR@ArrowTypeError
[...]
-| object:bytes | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError |
ERR@ArrowTypeError
[...]
-| date:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-06-15]@date32[day] |
ERR@ArrowTypeError
[...]
-| object:datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| [2024-06-15
18:30:00]@timestamp[us]
| [2024-06-15]@date32[day] |
ERR@ArrowTypeError
[...]
-| object:timedelta | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError | [1 day,
2:00:00]@duration[us]
[...]
-| time:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError |
ERR@ArrowTypeError
[...]
-| Int64:standard | ERR@ArrowInvalid | [0, 1,
9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid
| [1970-01-01 00:00:00,
1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us]
| ERR@ArrowNotImplementedError | [0:00:00,
0:00:00.000001, 106751991 days, 4:00:54.775807, -106751992 days,
19:59:05.224192]@duration[us] [...]
-| string[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowInvalid
| ERR@ArrowInvalid |
ERR@ArrowNotImplementedError
[...]
-| large_binary[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
|
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
[...]
-| int64:overflow | [44, 1]@int8 | [300,
1]@int64 | [300.0, 1.0]@float32
| [1970-01-01 00:00:00.000300,
1970-01-01 00:00:00.000001]@timestamp[us]
| ERR@ArrowNotImplementedError | [0:00:00.000300,
0:00:00.000001]@duration[us]
[...]
-| float64:fractional | [1, 2]@int8 | [1, 2]@int64
| [1.5, 2.5]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
[...]
-| float64:infinity | [0, 1]@int8 |
[-9223372036854775808, 1]@int64 | [inf,
1.0]@float32 |
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
[...]
-| float64:precision | [1]@int8 | [1]@int64
| [1.1234568357467651]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
[...]
-| datetime64[ns]:sub-us | ERR@ArrowNotImplementedError |
[1704067200000000123]@int64 |
ERR@ArrowNotImplementedError |
[2024-01-01 00:00:00]@timestamp[us]
| [2024-01-01]@date32[day]
| ERR@ArrowNotImplementedError
[...]
-| object:date-then-datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01, 2024-01-01]@date32[day] |
ERR@ArrowTypeError
[...]
-| object:datetime-then-date | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01, 2024-01-01]@date32[day] |
ERR@ArrowTypeError
[...]
\ No newline at end of file
+| source \ target | int8 | int64
| float32
| timestamp[us]
| date32[day] | duration[us]
[...]
+|--------------------------------|------------------------------|-------------------------------------------------------------|-------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------|------------------------------------------------------------------------------------------------------------------------
[...]
+| int64:standard | [0, 1, -1, -1, 0]@int8 | [0, 1, -1,
9223372036854775807, -9223372036854775808]@int64 | [0.0, 1.0, -1.0,
9.223372036854776e+18, -9.223372036854776e+18]@float32 | [1970-01-01 00:00:00,
1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow,
temporal overflow]@timestamp[us] | ERR@ArrowNotImplementedError |
[0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@du [...]
+| float64:standard | [0, 1, -1]@int8 | [0, 1,
-1]@int64 | [0.0, 1.5, -1.5]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
[...]
+| bool:standard | [1, 0, 1]@int8 | [1, 0,
1]@int64 | [1.0, 0.0, 1.0]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
[...]
+| object:string | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError |
ERR@ArrowTypeError
[...]
+| object:bytes | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError |
ERR@ArrowTypeError
[...]
+| date:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-06-15]@date32[day] |
ERR@ArrowTypeError
[...]
+| object:datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| [2024-06-15
18:30:00]@timestamp[us]
| [2024-06-15]@date32[day] |
ERR@ArrowTypeError
[...]
+| object:timedelta | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError | [1 day,
2:00:00]@duration[us]
[...]
+| time:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError |
ERR@ArrowTypeError
[...]
+| Int64:standard | ERR@ArrowInvalid | [0, 1,
9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid
| [1970-01-01 00:00:00,
1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us]
| ERR@ArrowNotImplementedError | [0:00:00,
0:00:00.000001, 106751991 days, 4:00:54.775807, -106751992 days,
19:59:05.224192]@duration[us] [...]
+| string[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowInvalid
| ERR@ArrowInvalid |
ERR@ArrowNotImplementedError
[...]
+| large_binary[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
|
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
[...]
+| int64:overflow | [44, 1]@int8 | [300,
1]@int64 | [300.0, 1.0]@float32
| [1970-01-01 00:00:00.000300,
1970-01-01 00:00:00.000001]@timestamp[us]
| ERR@ArrowNotImplementedError | [0:00:00.000300,
0:00:00.000001]@duration[us]
[...]
+| float64:fractional | [1, 2]@int8 | [1, 2]@int64
| [1.5, 2.5]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
[...]
+| float64:infinity | [0, 1]@int8 |
[-9223372036854775808, 1]@int64 | [inf,
1.0]@float32 |
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
[...]
+| float64:precision | [1]@int8 | [1]@int64
| [1.1234568357467651]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
[...]
+| datetime64[ns]:sub-us | ERR@ArrowNotImplementedError |
[1704067200000000123]@int64 |
ERR@ArrowNotImplementedError |
[2024-01-01 00:00:00]@timestamp[us]
| [2024-01-01]@date32[day]
| ERR@ArrowNotImplementedError
[...]
+| object:date-then-datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01, 2024-01-01]@date32[day] |
ERR@ArrowTypeError
[...]
+| object:datetime-then-date | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01, 2024-01-01]@date32[day] |
ERR@ArrowTypeError
[...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
