Song Jun created SPARK-19832:
--------------------------------
Summary: DynamicPartitionWriteTask should escape the partition
name
Key: SPARK-19832
URL: https://issues.apache.org/jira/browse/SPARK-19832
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.2.0
Reporter: Song Jun
Currently in DynamicPartitionWriteTask, when we get the paritionPath of a
parition, we just escape the partition value, not escape the partition name.
this will cause some problems for some special partition name situation, for
example :
1) if the partition name contains '%' etc, there will be two partition path
created in the filesytem, one is for escaped path like '/path/a%25b=1', another
is for unescaped path like '/path/a%b=1'.
and the data inserted stored in unescaped path, while the show partitions table
will return 'a%25b=1' which the partition name is escaped. So here it is not
consist. And I think the data should be stored in the escaped path in
filesystem, which Hive2.0.0 also have the same action.
2) if the partition name contains ':', there will throw exception that new
Path("/path","a:b"), this is illegal which has a colon in the relative path.
{code}
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path
in absolute URI: a:b
at org.apache.hadoop.fs.Path.initialize(Path.java:205)
at org.apache.hadoop.fs.Path.<init>(Path.java:171)
at org.apache.hadoop.fs.Path.<init>(Path.java:88)
... 48 elided
Caused by: java.net.URISyntaxException: Relative path in absolute URI: a:b
at java.net.URI.checkPath(URI.java:1823)
at java.net.URI.<init>(URI.java:745)
at org.apache.hadoop.fs.Path.initialize(Path.java:202)
... 50 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
