[
https://issues.apache.org/jira/browse/DRILL-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935943#comment-16935943
]
benj commented on DRILL-6958:
-----------------------------
In the next example, with a table with a column that contain a piece of json
like
{code:sql}
SELECT * FROM ....`example.parquet` LIMIT 2;
+---------+------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| hash | date |
info
|
+---------+------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| B29C56F | 2019-09-23 | {"Number": 322, "scans": {"nameofprocess":
{"detection": false, "version": "1.2"}}, {"othername": {"detection": true,
"version": "0.1"}}} |
| C28956E | 2019-09-22 | {"Number": 312, "scans": {"thirdname": {"detection":
false, "version": "1.0"}}}
|
+---------------+------------------+-------------------------------------------------------------------------------------------------------------------------------+
SELECT typeof(hash) AS hash, typeof(`date`) AS `date`, typeof(info) AS info
FROM ....`example.parquet` LIMIT 1;
+---------+--------+------+
| hash | date | info |
+---------+--------+------+
| VARCHAR | DATE | MAP |
+---------+--------+------+
{code}
It's not possible to push in a right way into a CSV file because of the
presence of separator and quote inside the json.
And there is no possibility to manually avoid this problem with a change of
separator or introduce quote because the type MAP is not convertible in VARCHAR
(DRILL-7375), so it's not possible to manually concatenate data
> CTAS csv with option
> --------------------
>
> Key: DRILL-6958
> URL: https://issues.apache.org/jira/browse/DRILL-6958
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Text & CSV
> Affects Versions: 1.15.0, 1.16.0
> Reporter: benj
> Priority: Major
>
> Currently, it may be difficult to produce well-formed CSV with CTAS (see
> comment below).
> It appears necessary to have some additional/configuratble options to write
> CSV file with CTAS :
> * possibility to change/define the separator,
> * possibility to write or not the header,
> * possibility to force the write of only 1 file instead of lot of parts,
> * possibility to force quoting
> * possibility to use/change escape char
> * ...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
