Hyukjin Kwon created SPARK-62012:
------------------------------------

             Summary: Send local rows and pandas string columns of PySpark's 
large Arrow types under spark.sql.execution.arrow.useLargeVarTypes, and compare 
the cache threshold with pyarrow's nbytes
                 Key: SPARK-62012
                 URL: https://issues.apache.org/jira/browse/SPARK-62012
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Problem
PySpark 4.3.0's createDataFrame reads 
spark.sql.execution.arrow.useLargeVarTypes and, when it is
"true", builds its Arrow table with large types:
- the strings and binaries of the schema, also in arrays, maps, structs and the 
SQL type of a
  user-defined type, are large_string and large_binary
  (LocalDataToArrowConversion.convert, conversion.py:994-1005; to_arrow_type,
  pandas/types.py:137-139);
- the binaries of a variant, a geometry and a geography stay binary.
It then compares `_table.nbytes` with 
spark.sql.session.localRelationCacheThreshold to choose
between an inline and a cached relation (connect/session.py:782). pyarrow's 
nbytes counts `len`
offsets of a string, binary, list or map array, not the `len + 1` of its buffer.

The drop-in and the Rust API read the setting, but two paths ignored it:
- local rows: the core's Arrow conversion always made Utf8 and Binary
  (crates/spark-connect/src/session.rs, datatype_to_arrow);
- the binding's IPC of pandas object-string columns 
(crates/pyspark-rs/src/pandas_input.rs,
  _pandas_string_ipc), which writes Utf8.

The size that every path compared with the threshold was arrow-rs's 
get_slice_memory_size, which
counts `len + 1` offsets: 4 bytes more per string column than PySpark's measure
(crates/spark-connect/src/arrow_input.rs, batch_data_size).

Repro
    spark.conf.set("spark.sql.session.localRelationSizeLimit", "1000")
    spark.conf.set("spark.sql.session.localRelationCacheThreshold", "100000")
    spark.conf.set("spark.sql.execution.arrow.useLargeVarTypes", "true")
    schema = StructType([StructField("s", StringType())])
    spark.createDataFrame([("abcd",)] * 10000, schema).count()
    spark.createDataFrame(pd.DataFrame({"s": ["abcd"] * 10000}), schema).count()
    spark.conf.unset("spark.sql.execution.arrow.useLargeVarTypes")
    spark.conf.set("spark.sql.session.localRelationCacheThreshold", "8001")
    spark.createDataFrame([("abcd",)] * 1000, schema).count()

Actual (drop-in):
- the first two: 10000, sent inline, as strings of 80000 bytes;
- the third: AnalysisException [LOCAL_RELATION_SIZE_LIMIT_EXCEEDED], as it 
measured 8004
  bytes.

Expected (pyspark-client 4.3.0):
- the first two: AnalysisException [LOCAL_RELATION_SIZE_LIMIT_EXCEEDED], as 
large strings of
  120000 bytes are cached;
- the third: 1000, inline, as its table is 8000 bytes.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to