JyunSheng Kao created SPARK-61343:
-------------------------------------

             Summary: Remove type: [arg-type] in pyspark.sql.connect.session
                 Key: SPARK-61343
                 URL: https://issues.apache.org/jira/browse/SPARK-61343
             Project: Spark
          Issue Type: Improvement
          Components: Connect, PySpark
    Affects Versions: 4.4.0
            Reporter: JyunSheng Kao


In {{def createDataFrame(...)}} ({*}python/pyspark/sql/connect/session.py{*}), 
{{configs}} comes from {{{}get_config_dict{}}}.

The comment (line 810) clarifies why values are never None or 
{{SQL_CONF_NOT_FOUND}} is raised in line 574.

{code:java}
line 573 # Get all related configs in a batch
line 574        configs = self._client.get_config_dict(


line 810 # get_config_dict throws [SQL_CONF_NOT_FOUND] if the key is not found.
line 811        cache_threshold = int(
{code}

So, it's safe to remove {{# type: ignore[arg-type]}} for these config load
 * spark.sql.session.localRelationCacheThreshold
 * spark.sql.session.localRelationSizeLimit
 * spark.sql.session.localRelationChunkSizeRows
 * spark.sql.session.localRelationChunkSizeBytes
 * spark.sql.session.localRelationBatchOfChunksSizeBytes



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to