cloud-fan commented on pull request #29191: URL: https://github.com/apache/spark/pull/29191#issuecomment-662802631
Let's define the behavior more clear: 1. DFReader/Writer options are case insensitive, so later option should override prior ones if the key equals case-insensitively. This already fixed by your previous PR. We should also clearly document it in DFReader/Writer. 2. For DS V1, the options are passed as `Map[String, String]`, so v1 sources don't really have a way to get the `originalMap`. All keys are lowercased. (unless you cast it to `CaseInsensitiveMap` which is private) 3. For DS v2, the options are passed as `CaseInsensitiveStringMap`, which provides `asCaseSensitiveMap`. So it's case-preserving. This PR is trying to fix 3. There are 2 places we should fix: 1. `val options = sessionOptions ++ extraOptions`. Using `extraOptions` directly as a map will return lower-cased keys. There is another concern that if `sessionOptions` and `extraOptions` have duplicated keys, we should make sure `extraOptions` takes precedence. 2. `AppendData.byName...`. It should also get the case-preserving map, as the `AppendDataExec` will wrap it with `CaseInsensitiveStringMap`. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
