[
https://issues.apache.org/jira/browse/SPARK-16227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15415991#comment-15415991
]
Davies Liu commented on SPARK-16227:
------------------------------------
[~brkyvz] I can't reproduce this in master (2.1-snapshot),
{code}
jsont/
jsont//._SUCCESS.crc
jsont//_SUCCESS
jsont//a=b:c
jsont//a=b:c/.part-r-00000-67142af0-be62-455f-b3ed-9626ae84ffbd.json.crc
jsont//a=b:c/.part-r-00001-67142af0-be62-455f-b3ed-9626ae84ffbd.json.crc
jsont//a=b:c/.part-r-00002-67142af0-be62-455f-b3ed-9626ae84ffbd.json.crc
jsont//a=b:c/.part-r-00003-67142af0-be62-455f-b3ed-9626ae84ffbd.json.crc
jsont//a=b:c/part-r-00000-67142af0-be62-455f-b3ed-9626ae84ffbd.json
jsont//a=b:c/part-r-00001-67142af0-be62-455f-b3ed-9626ae84ffbd.json
jsont//a=b:c/part-r-00002-67142af0-be62-455f-b3ed-9626ae84ffbd.json
jsont//a=b:c/part-r-00003-67142af0-be62-455f-b3ed-9626ae84ffbd.json
{code}
{code}
>>> spark.read.json("jsont/").show()
+---+---+
| id| a|
+---+---+
| 75|b:c|
| 76|b:c|
| 77|b:c|
| 78|b:c|
| 79|b:c|
| 80|b:c|
| 81|b:c|
| 82|b:c|
| 83|b:c|
| 84|b:c|
| 85|b:c|
| 86|b:c|
| 87|b:c|
| 88|b:c|
| 89|b:c|
| 90|b:c|
| 91|b:c|
| 92|b:c|
| 93|b:c|
| 94|b:c|
+---+---+
only showing top 20 rows
{code}
> Json schema inference fails when `:` exists in file path
> --------------------------------------------------------
>
> Key: SPARK-16227
> URL: https://issues.apache.org/jira/browse/SPARK-16227
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.6.1, 2.0.0
> Reporter: Burak Yavuz
>
> While Hadoop handles `:` in file paths since
> https://issues.apache.org/jira/browse/HDFS-13, the json schema inference
> doesn't behave as nicely.
> If the filename has a timestamp which contains a colon, if this file is in
> parquet, everything works, but when it is in json, it fails. Json should be
> able to also handle these cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
