This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new d4fb4e774d87 [SPARK-58765][PYTHON][TEST][FOLLOWUP] Regenerate the
type-scalar from_pandas golden for the temporal-overflow repr
d4fb4e774d87 is described below
commit d4fb4e774d87272a99d6c53e52220db27a8033a8
Author: Spenser Sun <[email protected]>
AuthorDate: Wed Aug 19 23:59:08 2026 +0000
[SPARK-58765][PYTHON][TEST][FOLLOWUP] Regenerate the type-scalar
from_pandas golden for the temporal-overflow repr
### What changes were proposed in this pull request?
Regenerate the `pa.Array.from_pandas` type-scalar golden files so the
`int64:standard` / `Int64:standard` -> `timestamp[us]` cells render `temporal
overflow` instead of `ERROverflowError`. Golden files only; no code change.
### Why are the changes needed?
The type-scalar golden (SPARK-58765, #57996) was generated before
SPARK-58766 (#58000), which revises `GoldenFileTestMixin._scalar_str` to render
out-of-Python-range temporal scalars as `temporal overflow` rather than letting
the per-scalar `str(...)` raise `OverflowError`. #58000 landed on master one
commit before #57996 merged, so #57996's golden files are stale compared with
the new renderer. Those two cells still have `ERROverflowError`, while the
correct one should be `temporal o [...]
`PyArrowArrayFromPandasTypeScalarTests` right now fails on post-merge CI on
that commit. Also, this test is registered under the `pyspark-core` module, so
this red will affect scheduled Python builds and any PR whose CI runs that
module.
### Does this PR introduce any user-facing change?
No (test-only).
### How was this patch tested?
Regenerated with `SPARK_GENERATE_GOLDEN_FILES=1`; the diff is exactly the
two `timestamp[us]` cells (safe and unsafe goldens). Swept pyarrow 18-25 x
pandas 2/3: 16/16 pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #58140 from Spenserrrr/fix-type-scalar-golden-temporal-overflow.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Yicong-Huang <[email protected]>
(cherry picked from commit 326eec2d53c7cb3f2c55a6a9121af20b518297e7)
Signed-off-by: Yicong-Huang <[email protected]>
---
..._pyarrow_array_from_pandas_type_scalar_safe.csv | 4 +--
...n_pyarrow_array_from_pandas_type_scalar_safe.md | 42 +++++++++++-----------
...yarrow_array_from_pandas_type_scalar_unsafe.csv | 4 +--
...pyarrow_array_from_pandas_type_scalar_unsafe.md | 42 +++++++++++-----------
4 files changed, 46 insertions(+), 46 deletions(-)
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.csv
index 655dce556473..a054d3a83712 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.csv
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.csv
@@ -1,5 +1,5 @@
source \ target int8 int64 float32 timestamp[us] date32[day]
duration[us] time64[ns] string binary
-int64:standard ERR@ArrowInvalid [0, 1, -1, 9223372036854775807,
-9223372036854775808]@int64 ERR@ArrowInvalid ERR@OverflowError
ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, -1 day,
23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days,
19:59:05.224192]@duration[us] ERR@ValueError ERR@ArrowTypeError
[b'', b'\x01', b'\xff\xff\xff\xff\xff\xff\xff\xff',
b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
+int64:standard ERR@ArrowInvalid [0, 1, -1, 9223372036854775807,
-9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00,
1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow,
temporal overflow]@timestamp[us] ERR@ArrowNotImplementedError
[0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us]
ERR@ValueError ERR@ArrowTypeError [b'', b'\x01',
b'\xff\xff\xff\xff\xff\xff\xff\xff', b'\xff\x [...]
float64:standard ERR@ArrowInvalid ERR@ArrowInvalid [0.0,
1.5, -1.5]@float32 ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowTypeError [b'', b'', b'']@binary
bool:standard [1, 0, 1]@int8 [1, 0, 1]@int64 [1.0, 0.0, 1.0]@float32
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowTypeError [b'\x01', b'', b'\x01']@binary
object:string ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError [hello, world, ]@string
[b'hello', b'world', b'']@binary
@@ -8,7 +8,7 @@ date:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeEr
object:datetime ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid [2024-06-15 18:30:00]@timestamp[us]
[2024-06-15]@date32[day] ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError
object:timedelta ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError [1 day,
2:00:00]@duration[us] ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError
time:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError [18:30:45]@time64[ns] ERR@ArrowTypeError
ERR@ArrowTypeError
-Int64:standard ERR@ArrowInvalid [0, 1, 9223372036854775807,
-9223372036854775808]@int64 ERR@ArrowInvalid ERR@OverflowError
ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us]
ERR@ValueError ERR@ArrowTypeError [b'', b'\x01',
b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
+Int64:standard ERR@ArrowInvalid [0, 1, 9223372036854775807,
-9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00,
1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us]
ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us]
ERR@ValueError ERR@ArrowTypeError [b'', b'\x01',
b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
string[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [hello,
world]@string [b'hello', b'world']@binary
large_binary[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError [hello, world]@string [b'hello',
b'world']@binary
int64:overflow ERR@ArrowInvalid [300, 1]@int64 [300.0, 1.0]@float32
[1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us]
ERR@ArrowNotImplementedError [0:00:00.000300, 0:00:00.000001]@duration[us]
[00:00:00, 00:00:00]@time64[ns] ERR@ArrowTypeError [b',\x01',
b'\x01']@binary
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.md
index 89f7ce26e4da..65476ad0e9e9 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.md
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.md
@@ -1,21 +1,21 @@
-| source \ target | int8 | int64
| float32 |
timestamp[us] |
date32[day] | duration[us]
| time64[ns] | string | binary
[...]
-|--------------------------------|------------------------------|-------------------------------------------------------------|------------------------------|------------------------------------------------------------------------|--------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|---------------------------------|-------------------------|------------------------------
[...]
-| int64:standard | ERR@ArrowInvalid | [0, 1, -1,
9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid
| ERR@OverflowError |
ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, -1 day,
23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days,
19:59:05.224192]@duration[us] | ERR@ValueError |
ERR@ArrowTypeError | [b'', b'\x01', b'\xff\xff\xff [...]
-| float64:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | [0.0, 1.5,
-1.5]@float32 | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowTypeError | [b'', b'',
b'']@binary [...]
-| bool:standard | [1, 0, 1]@int8 | [1, 0,
1]@int64 | [1.0, 0.0, 1.0]@float32
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError | ERR@ArrowTypeError |
[b'\x01', b'', b'\x01']@binar [...]
-| object:string | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError | ERR@ArrowTypeError
| ERR@ArrowTypeError | [hello, world, ]@string |
[b'hello', b'world', b'']@bin [...]
-| object:bytes | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError | ERR@ArrowTypeError
| ERR@ArrowTypeError | [hello, world]@string |
[b'hello', b'world']@binary [...]
-| date:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-06-15]@date32[day] | ERR@ArrowTypeError
| ERR@ArrowTypeError | ERR@ArrowTypeError |
ERR@ArrowTypeError [...]
-| object:datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| [2024-06-15 18:30:00]@timestamp[us]
| [2024-06-15]@date32[day] | ERR@ArrowTypeError
| ERR@ArrowTypeError | ERR@ArrowTypeError |
ERR@ArrowTypeError [...]
-| object:timedelta | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError | [1 day, 2:00:00]@duration[us]
| ERR@ArrowTypeError | ERR@ArrowTypeError |
ERR@ArrowTypeError [...]
-| time:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError | ERR@ArrowTypeError
| [18:30:45]@time64[ns] | ERR@ArrowTypeError |
ERR@ArrowTypeError [...]
-| Int64:standard | ERR@ArrowInvalid | [0, 1,
9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid
| ERR@OverflowError |
ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, 106751991
days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us]
| ERR@ValueError | ERR@ArrowTypeError |
[b'', b'\x01', b'\xff\xff\xff [...]
-| string[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowInvalid
| ERR@ArrowInvalid | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError | [hello, world]@string |
[b'hello', b'world']@binary [...]
-| large_binary[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError | [hello, world]@string |
[b'hello', b'world']@binary [...]
-| int64:overflow | ERR@ArrowInvalid | [300,
1]@int64 | [300.0, 1.0]@float32
| [1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us] |
ERR@ArrowNotImplementedError | [0:00:00.000300,
0:00:00.000001]@duration[us]
| [00:00:00, 00:00:00]@time64[ns] |
ERR@ArrowTypeError | [b',\x01', b'\x01']@binary [...]
-| float64:fractional | ERR@ArrowInvalid |
ERR@ArrowInvalid | [1.5,
2.5]@float32 | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowTypeError | [b'', b'']@binary
[...]
-| float64:infinity | ERR@ArrowInvalid |
ERR@ArrowInvalid | [inf,
1.0]@float32 | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowTypeError | [b'', b'']@binary
[...]
-| float64:precision | ERR@ArrowInvalid |
ERR@ArrowInvalid |
[1.1234568357467651]@float32 | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowTypeError |
[b'\x98nt\xd3\xad\xf9\xf1?']@ [...]
-| datetime64[ns]:sub-us | ERR@ArrowNotImplementedError |
[1704067200000000123]@int64 |
ERR@ArrowNotImplementedError | ERR@ArrowInvalid
| [2024-01-01]@date32[day] |
ERR@ArrowNotImplementedError
| [00:00:00]@time64[ns]
| ERR@ArrowTypeError | [b'{']@binary [...]
-| object:date-then-datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01, 2024-01-01]@date32[day] | ERR@ArrowTypeError
| ERR@ArrowTypeError | ERR@ArrowTypeError |
ERR@ArrowTypeError [...]
-| object:datetime-then-date | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01, 2024-01-01]@date32[day] | ERR@ArrowTypeError
| ERR@ArrowTypeError | ERR@ArrowTypeError |
ERR@ArrowTypeError [...]
\ No newline at end of file
+| source \ target | int8 | int64
| float32 |
timestamp[us]
| date32[day]
| duration[us]
| time64[ns]
[...]
+|--------------------------------|------------------------------|-------------------------------------------------------------|------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|-------------------------------
[...]
+| int64:standard | ERR@ArrowInvalid | [0, 1, -1,
9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid
| [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999,
temporal overflow, temporal overflow]@timestamp[us] |
ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, -1 day,
23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days,
19:59:05.224192]@duration[us] | ERR@ValueError [...]
+| float64:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | [0.0, 1.5,
-1.5]@float32 | ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
+| bool:standard | [1, 0, 1]@int8 | [1, 0,
1]@int64 | [1.0, 0.0, 1.0]@float32
| ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
+| object:string | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
|
ERR@ArrowTypeError | ERR@ArrowTypeError
| ERR@ArrowTypeError [...]
+| object:bytes | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
|
ERR@ArrowTypeError | ERR@ArrowTypeError
| ERR@ArrowTypeError [...]
+| date:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
|
[2024-06-15]@date32[day] | ERR@ArrowTypeError
| ERR@ArrowTypeError [...]
+| object:datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| [2024-06-15 18:30:00]@timestamp[us]
|
[2024-06-15]@date32[day] | ERR@ArrowTypeError
| ERR@ArrowTypeError [...]
+| object:timedelta | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
|
ERR@ArrowTypeError | [1 day, 2:00:00]@duration[us]
| ERR@ArrowTypeError [...]
+| time:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
|
ERR@ArrowTypeError | ERR@ArrowTypeError
| [18:30:45]@time64[ns] [...]
+| Int64:standard | ERR@ArrowInvalid | [0, 1,
9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid
| [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, temporal overflow,
temporal overflow]@timestamp[us] |
ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, 106751991
days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us]
| ERR@ValueError [...]
+| string[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowInvalid
|
ERR@ArrowInvalid | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
+| large_binary[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
+| int64:overflow | ERR@ArrowInvalid | [300,
1]@int64 | [300.0, 1.0]@float32
| [1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us]
|
ERR@ArrowNotImplementedError | [0:00:00.000300,
0:00:00.000001]@duration[us]
| [00:00:00, 00:00:00]@time64[ns [...]
+| float64:fractional | ERR@ArrowInvalid |
ERR@ArrowInvalid | [1.5,
2.5]@float32 | ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
+| float64:infinity | ERR@ArrowInvalid |
ERR@ArrowInvalid | [inf,
1.0]@float32 | ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
+| float64:precision | ERR@ArrowInvalid |
ERR@ArrowInvalid |
[1.1234568357467651]@float32 | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError [...]
+| datetime64[ns]:sub-us | ERR@ArrowNotImplementedError |
[1704067200000000123]@int64 |
ERR@ArrowNotImplementedError | ERR@ArrowInvalid
| [2024-01-01]@date32[day] | ERR@ArrowNotImplementedError
| [00:00:00]@time64[ns] [...]
+| object:date-then-datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01,
2024-01-01]@date32[day] | ERR@ArrowTypeError
|
ERR@ArrowTypeError [...]
+| object:datetime-then-date | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01,
2024-01-01]@date32[day] | ERR@ArrowTypeError
|
ERR@ArrowTypeError [...]
\ No newline at end of file
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.csv
index e1690c2003b5..b04f68c6e346 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.csv
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.csv
@@ -1,5 +1,5 @@
source \ target int8 int64 float32 timestamp[us] date32[day]
duration[us] time64[ns] string binary
-int64:standard [0, 1, -1, -1, 0]@int8 [0, 1, -1, 9223372036854775807,
-9223372036854775808]@int64 [0.0, 1.0, -1.0, 9.223372036854776e+18,
-9.223372036854776e+18]@float32 ERR@OverflowError
ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, -1 day,
23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days,
19:59:05.224192]@duration[us] ERR@ValueError ERR@ArrowTypeError
[b'', b'\x01', b'\xff\xff\xff\xff\xff\xff\xff\xff',
b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
+int64:standard [0, 1, -1, -1, 0]@int8 [0, 1, -1, 9223372036854775807,
-9223372036854775808]@int64 [0.0, 1.0, -1.0, 9.223372036854776e+18,
-9.223372036854776e+18]@float32 [1970-01-01 00:00:00, 1970-01-01
00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow, temporal
overflow]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00,
0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days, 4:00:54.775807,
-106751992 days, 19:59:05.224192]@duration[us] ERR@ValueError
ERR@ArrowTypeError [...]
float64:standard [0, 1, -1]@int8 [0, 1, -1]@int64 [0.0, 1.5,
-1.5]@float32 ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowTypeError [b'', b'', b'']@binary
bool:standard [1, 0, 1]@int8 [1, 0, 1]@int64 [1.0, 0.0, 1.0]@float32
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowTypeError [b'\x01', b'', b'\x01']@binary
object:string ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError [hello, world, ]@string
[b'hello', b'world', b'']@binary
@@ -8,7 +8,7 @@ date:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeEr
object:datetime ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid [2024-06-15 18:30:00]@timestamp[us]
[2024-06-15]@date32[day] ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError
object:timedelta ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError [1 day,
2:00:00]@duration[us] ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError
time:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError [18:30:45]@time64[ns] ERR@ArrowTypeError
ERR@ArrowTypeError
-Int64:standard ERR@ArrowInvalid [0, 1, 9223372036854775807,
-9223372036854775808]@int64 ERR@ArrowInvalid ERR@OverflowError
ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us]
ERR@ValueError ERR@ArrowTypeError [b'', b'\x01',
b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
+Int64:standard ERR@ArrowInvalid [0, 1, 9223372036854775807,
-9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00,
1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us]
ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us]
ERR@ValueError ERR@ArrowTypeError [b'', b'\x01',
b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
string[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [hello,
world]@string [b'hello', b'world']@binary
large_binary[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError [hello, world]@string [b'hello',
b'world']@binary
int64:overflow [44, 1]@int8 [300, 1]@int64 [300.0, 1.0]@float32
[1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us]
ERR@ArrowNotImplementedError [0:00:00.000300, 0:00:00.000001]@duration[us]
[00:00:00, 00:00:00]@time64[ns] ERR@ArrowTypeError [b',\x01',
b'\x01']@binary
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.md
index ce6d7dcd2c61..05ddfff08f59 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.md
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.md
@@ -1,21 +1,21 @@
-| source \ target | int8 | int64
| float32
| timestamp[us]
| date32[day] |
duration[us]
| time64[ns]
| string [...]
-|--------------------------------|------------------------------|-------------------------------------------------------------|-------------------------------------------------------------------------|------------------------------------------------------------------------|--------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|---------------------------------|-------------
[...]
-| int64:standard | [0, 1, -1, -1, 0]@int8 | [0, 1, -1,
9223372036854775807, -9223372036854775808]@int64 | [0.0, 1.0, -1.0,
9.223372036854776e+18, -9.223372036854776e+18]@float32 | ERR@OverflowError
| ERR@ArrowNotImplementedError
| [0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] | ERR@ValueError
| ERR@ArrowTyp [...]
-| float64:standard | [0, 1, -1]@int8 | [0, 1,
-1]@int64 | [0.0, 1.5, -1.5]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowTyp [...]
-| bool:standard | [1, 0, 1]@int8 | [1, 0,
1]@int64 | [1.0, 0.0, 1.0]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowTyp [...]
-| object:string | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| [hello, worl [...]
-| object:bytes | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| [hello, worl [...]
-| date:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-06-15]@date32[day]
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| ERR@ArrowTyp [...]
-| object:datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| [2024-06-15
18:30:00]@timestamp[us] |
[2024-06-15]@date32[day] | ERR@ArrowTypeError
| ERR@ArrowTypeError | ERR@ArrowTyp [...]
-| object:timedelta | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| [1 day, 2:00:00]@duration[us]
| ERR@ArrowTypeError
| ERR@ArrowTyp [...]
-| time:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| ERR@ArrowTypeError
|
[18:30:45]@time64[ns] | ERR@ArrowTyp [...]
-| Int64:standard | ERR@ArrowInvalid | [0, 1,
9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid
| ERR@OverflowError
| ERR@ArrowNotImplementedError
| [0:00:00, 0:00:00.000001, 106751991 days, 4:00:54.775807, -106751992 days,
19:59:05.224192]@duration[us] | ERR@ValueError
| ERR@ArrowTyp [...]
-| string[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowInvalid
| ERR@ArrowInvalid
| ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | [hello, worl [...]
-| large_binary[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
|
ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError | [hello, worl [...]
-| int64:overflow | [44, 1]@int8 | [300,
1]@int64 | [300.0, 1.0]@float32
| [1970-01-01 00:00:00.000300,
1970-01-01 00:00:00.000001]@timestamp[us] | ERR@ArrowNotImplementedError
| [0:00:00.000300, 0:00:00.000001]@duration[us]
| [00:00:00,
00:00:00]@time64[ns] | ERR@ArrowTyp [...]
-| float64:fractional | [1, 2]@int8 | [1, 2]@int64
| [1.5, 2.5]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowTyp [...]
-| float64:infinity | [0, 1]@int8 |
[-9223372036854775808, 1]@int64 | [inf,
1.0]@float32 |
ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError | ERR@ArrowTyp [...]
-| float64:precision | [1]@int8 | [1]@int64
| [1.1234568357467651]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowTyp [...]
-| datetime64[ns]:sub-us | ERR@ArrowNotImplementedError |
[1704067200000000123]@int64 |
ERR@ArrowNotImplementedError |
[2024-01-01 00:00:00]@timestamp[us] |
[2024-01-01]@date32[day] | ERR@ArrowNotImplementedError
| [00:00:00]@time64[ns] | ERR@ArrowTyp [...]
-| object:date-then-datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01,
2024-01-01]@date32[day] | ERR@ArrowTypeError
|
ERR@ArrowTypeError | ERR@ArrowTyp [...]
-| object:datetime-then-date | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01,
2024-01-01]@date32[day] | ERR@ArrowTypeError
|
ERR@ArrowTypeError | ERR@ArrowTyp [...]
\ No newline at end of file
+| source \ target | int8 | int64
| float32
| timestamp[us]
| date32[day] | duration[us]
[...]
+|--------------------------------|------------------------------|-------------------------------------------------------------|-------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------|------------------------------------------------------------------------------------------------------------------------
[...]
+| int64:standard | [0, 1, -1, -1, 0]@int8 | [0, 1, -1,
9223372036854775807, -9223372036854775808]@int64 | [0.0, 1.0, -1.0,
9.223372036854776e+18, -9.223372036854776e+18]@float32 | [1970-01-01 00:00:00,
1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow,
temporal overflow]@timestamp[us] | ERR@ArrowNotImplementedError |
[0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@du [...]
+| float64:standard | [0, 1, -1]@int8 | [0, 1,
-1]@int64 | [0.0, 1.5, -1.5]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
[...]
+| bool:standard | [1, 0, 1]@int8 | [1, 0,
1]@int64 | [1.0, 0.0, 1.0]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
[...]
+| object:string | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError |
ERR@ArrowTypeError
[...]
+| object:bytes | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError |
ERR@ArrowTypeError
[...]
+| date:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-06-15]@date32[day] |
ERR@ArrowTypeError
[...]
+| object:datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| [2024-06-15
18:30:00]@timestamp[us]
| [2024-06-15]@date32[day] |
ERR@ArrowTypeError
[...]
+| object:timedelta | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError | [1 day,
2:00:00]@duration[us]
[...]
+| time:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError |
ERR@ArrowTypeError
[...]
+| Int64:standard | ERR@ArrowInvalid | [0, 1,
9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid
| [1970-01-01 00:00:00,
1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us]
| ERR@ArrowNotImplementedError | [0:00:00,
0:00:00.000001, 106751991 days, 4:00:54.775807, -106751992 days,
19:59:05.224192]@duration[us] [...]
+| string[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowInvalid
| ERR@ArrowInvalid |
ERR@ArrowNotImplementedError
[...]
+| large_binary[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
|
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
[...]
+| int64:overflow | [44, 1]@int8 | [300,
1]@int64 | [300.0, 1.0]@float32
| [1970-01-01 00:00:00.000300,
1970-01-01 00:00:00.000001]@timestamp[us]
| ERR@ArrowNotImplementedError | [0:00:00.000300,
0:00:00.000001]@duration[us]
[...]
+| float64:fractional | [1, 2]@int8 | [1, 2]@int64
| [1.5, 2.5]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
[...]
+| float64:infinity | [0, 1]@int8 |
[-9223372036854775808, 1]@int64 | [inf,
1.0]@float32 |
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
[...]
+| float64:precision | [1]@int8 | [1]@int64
| [1.1234568357467651]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
[...]
+| datetime64[ns]:sub-us | ERR@ArrowNotImplementedError |
[1704067200000000123]@int64 |
ERR@ArrowNotImplementedError |
[2024-01-01 00:00:00]@timestamp[us]
| [2024-01-01]@date32[day]
| ERR@ArrowNotImplementedError
[...]
+| object:date-then-datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01, 2024-01-01]@date32[day] |
ERR@ArrowTypeError
[...]
+| object:datetime-then-date | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01, 2024-01-01]@date32[day] |
ERR@ArrowTypeError
[...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
