micomahesh1982 commented on issue #13125:
URL: https://github.com/apache/arrow/issues/13125#issuecomment-1124554021
technically,
csv_file =
StringIO("""int__v|Decimal__v|Float__v|Boolean__v|String__v|Null__v|Date__v|Timestamp__v
1|43.4|11.02|True|'456'|12|2021-03-02|2019-08-07 10:11:12
2|101.4|11.128|False|'456'||2020-09-09|2019-05-04 11:12:13
3|43.4|13.02|True|'456'||2022-03-14|2012-08-07 10:15:12
4|202.4|14.128|False|'456'||2020-03-15|2017-09-04 11:17:13
5|43.4|11.02|True|'456'||2021-03-02|2019-08-07 10:11:12
6|101.4|11.128|False|'456'||2020-09-09|2019-05-04 11:12:13
7|43.4|13.02|True|'456'||2022-03-14|2012-08-07 10:15:12
8|202.4|14.128|False|'456'||2020-03-15|2017-09-04 11:17:13
9|43.4|11.02|True|'456'||2021-03-02|2019-08-07 10:11:12
10|101.4|11.128|False|'456'||2020-09-09|2019-05-04 11:12:13
11|43.4|13.02|True|'456'||2022-03-14|2012-08-07 10:15:12
12|202.4|14.128|False|'456'||2020-03-15|2017-09-04 11:17:13
13|101.4|11.128|False|'456'||2020-09-09|2019-05-04 11:12:13
14|43.4|13.02|True|'456'||2022-03-14|2012-08-07 10:15:12
15|202.4|14.128|False|'456'||2020-03-15|2017-09-04 11:17:13
16|43.4|11.02|True|'456'||2021-03-02|2019-08-07 10:11:12
17|101.4|11.128|False|'456'||2020-09-09|2019-05-04 11:12:13
18|43.4|13.02|True|'456'||2022-03-14|2012-08-07 10:15:12
19|202.4|14.128|False|'456'||2020-03-15|2017-09-04 11:17:13
4|||||||
""")
params = { 'filepath_or_buffer': csv_file, 'chunksize': 10, 'encoding':
'UTF-8', 'sep': '|', 'low_memory': True, 'engine': 'python',
'skip_blank_lines': True }
Next, loop through chunk of data and convert csv files into parquet and
write into s3.
you would be noticing 4th column is 'boolean' datatype and last row has 'NA'
which means null data. That's where getting "parquet conversion failed,Bool
column has NA values in column boolean__v"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
