Allison Wang created SPARK-46568: ------------------------------------ Summary: Python data source options should be a case insensitive dictionary Key: SPARK-46568 URL: https://issues.apache.org/jira/browse/SPARK-46568 Project: Spark Issue Type: Sub-task Components: PySpark Affects Versions: 4.0.0 Reporter: Allison Wang
Data source options are stored as a `CaseInsensitiveStringMap` in Scala, however, its behavior is inconsistent in Python: {code:java} class MyDataSource(DataSource): def __init__(self, options): self.api_key = options.get("API_KEY") # <- This is None spark.read.format(..).option("API_KEY", my_key).load(...){code} Currently, options will not have this "API_KEY" as everything is converted to lowercase on the Scala side. This can be confusing to users. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org