[ https://issues.apache.org/jira/browse/SPARK-34102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noah Kawasaki updated SPARK-34102: ---------------------------------- Labels: escaping filter string-manipulation (was: ) > Spark SQL cannot escape both \ and other special characters > ------------------------------------------------------------ > > Key: SPARK-34102 > URL: https://issues.apache.org/jira/browse/SPARK-34102 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.0.2, 2.1.3, 2.2.2, 2.3.0, 2.4.5, 3.0.1 > Reporter: Noah Kawasaki > Priority: Minor > Labels: escaping, filter, string-manipulation > > Spark literal string parsing does not properly escape backslashes or other > special characters. This is an extension of this issue: > https://issues.apache.org/jira/browse/SPARK-17647# > > The issue is that depending on how spark.sql.parser.escapedStringLiterals is > set, you will either be able to correctly get escaped backslashes in a string > literal, but not escaped other special characters, OR, you can have correctly > escaped other special characters, but not correctly escaped backslashes. > So you have to choose which configuration you care about more. > I have tested Spark versions 2.1, 2.2, 2.3, 2.4, and 3.0 and they all > experience the issue: > {code:java} > # These do not return the expected backslash > SET spark.sql.parser.escapedStringLiterals=false; > SELECT '\\'; > > \ > (should return \\) > SELECT 'hi\hi'; > > hihi > (should return hi\hi) > # These are correctly escaped > SELECT '\"'; > > " > SELECT '\''; > > '{code} > If I switch this: > {code:java} > # These now work > SET spark.sql.parser.escapedStringLiterals=true; > SELECT '\\'; > > \\ > SELECT 'hi\hi'; > > hi\hi > # These are now not correctly escaped > SELECT '\"'; > > \" > (should return ") > SELECT '\''; > > \' > (should return ' ){code} > So basically we have to choose: > SET spark.sql.parser.escapedStringLiterals=false; if we want backslashes > correctly escaped but not other special characters > SET spark.sql.parser.escapedStringLiterals=true; if we want other special > characters correctly escaped but not backslashes -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org