[
https://issues.apache.org/jira/browse/PHOENIX-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17547357#comment-17547357
]
Rushabh Shah commented on PHOENIX-6426:
---------------------------------------
Reproduced via this example:
{noformat}
Saving all output to "/home/sfdc/us_population.csv". Enter "record" with no
arguments to stop it.
'STATE','CITY','POPULATION'
'NY','New York','8143197'
{noformat}
Schema for US_POPULATION table:
{noformat}
CREATE TABLE IF NOT EXISTS us_population (
state CHAR(2) NOT NULL,
city VARCHAR NOT NULL,
population BIGINT
CONSTRAINT my_pk PRIMARY KEY (state, city));
{noformat}
Ran this command for running MR job
{noformat}
HADOOP_CLASSPATH=$(/usr/hdp/current/hbase-client/bin/hbase
classpath):<phoenix-jar-locatoion>:<hbase-conf> hadoop jar <phoenix-jar>
org.apache.phoenix.mapreduce.CsvBulkLoadTool -libjars commons-csv-1.0.jar
--table US_POPULATION --input /tmp/us_population.csv
{noformat}
I had to pass commons-csv-1.0.jar as libjars. Don't know why.
{noformat}
The above MR job failed with the following exception.
2022-06-03 22:23:51,951 INFO [main] util.UpsertExecutor - Upserting SQL data
with UPSERT INTO "US_POPULATION" ("STATE", "CITY", "0"."POPULATION") VALUES
(?, ?, ?)
2022-06-03 22:23:51,974 ERROR [main] mapreduce.FormatToBytesWritableMapper -
Error on record [Saving all output to "us_population.csv". Enter "record" with
no arguments to stop it.]
java.lang.IllegalArgumentException: CSV record does not have enough values (has
1, but needs 3)
at
org.apache.phoenix.util.csv.CsvUpsertExecutor.execute(CsvUpsertExecutor.java:85)
at
org.apache.phoenix.util.csv.CsvUpsertExecutor.execute(CsvUpsertExecutor.java:55)
at
org.apache.phoenix.util.UpsertExecutor.execute(UpsertExecutor.java:133)
at
org.apache.phoenix.mapreduce.FormatToBytesWritableMapper.map(FormatToBytesWritableMapper.java:175)
at
org.apache.phoenix.mapreduce.FormatToBytesWritableMapper.map(FormatToBytesWritableMapper.java:77)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:794)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1926)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171)
2022-06-03 22:23:51,977 INFO [main]
connqueryservice.ConnectionQueryServicesMetricsManager - connection query
service metrics are disabled for connection query service: null
2022-06-03 22:23:51,978 INFO [main] mapred.MapTask - Starting flush of map
output
2022-06-03 22:23:51,991 INFO [main] compress.CodecPool - Got brand-new
compressor [.snappy]
2022-06-03 22:23:51,995 WARN [main] mapred.YarnChild - Exception running child
: java.lang.RuntimeException: java.lang.IllegalArgumentException: CSV record
does not have enough values (has 1, but needs 3)
at
org.apache.phoenix.mapreduce.FormatToBytesWritableMapper.map(FormatToBytesWritableMapper.java:206)
at
org.apache.phoenix.mapreduce.FormatToBytesWritableMapper.map(FormatToBytesWritableMapper.java:77)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:794)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1926)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171)
Caused by: java.lang.IllegalArgumentException: CSV record does not have enough
values (has 1, but needs 3)
at
org.apache.phoenix.util.csv.CsvUpsertExecutor.execute(CsvUpsertExecutor.java:85)
at
org.apache.phoenix.util.csv.CsvUpsertExecutor.execute(CsvUpsertExecutor.java:55)
at
org.apache.phoenix.util.UpsertExecutor.execute(UpsertExecutor.java:133)
at
org.apache.phoenix.mapreduce.FormatToBytesWritableMapper.map(FormatToBytesWritableMapper.java:175)
... 9 more
{noformat}
Internally this is running some version of 4.16 latest branch.
Cc [~sameerpandit90]
> CsvBulkLoadTool should be able to parse the output of sqlline generated csv
> ---------------------------------------------------------------------------
>
> Key: PHOENIX-6426
> URL: https://issues.apache.org/jira/browse/PHOENIX-6426
> Project: Phoenix
> Issue Type: Improvement
> Reporter: Gokcen Iskender
> Assignee: Sameer Pandit
> Priority: Minor
>
> Sqlline generates the csv file in a format that csvbulkloadtool cannot parse.
> [~rushabh.shah] recently hit this problem
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
