[
https://issues.apache.org/jira/browse/HADOOP-13970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15818275#comment-15818275
]
Steve Loughran commented on HADOOP-13970:
-----------------------------------------
at a guess, there's no untarring taking place: the csv reader is treating the
entire .tar file as a single file, so failing.
Looks like this is the solution: untar before passing in the files:
http://stackoverflow.com/questions/38635905/reading-in-multiple-files-compressed-in-tar-gz-archive-into-spark
> garbage data read from the beginning of a tar file
> --------------------------------------------------
>
> Key: HADOOP-13970
> URL: https://issues.apache.org/jira/browse/HADOOP-13970
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
> Affects Versions: 2.6.0
> Environment: Linux EL6
> Reporter: Steve Yang
> Attachments: taxi_simplified.tar
>
>
> Hadoop from CDH 5.7.1
> on Spark using databricks ('com.databricks:spark-csv_2.10:1.5.0') to read in
> a tar file which consists of 3 .csv files.
> sqlCtx.read().format("com.databricks.spark.csv").option(...)
> .load(objectName);
> The tar file contains 3 files:
> taxi_simplified1.csv
> taxi2.csv
> simplified3.csv
> where the first line (header) is:
> trip_distance,dropoff_datetime,dropoff_geocode,passenger_count,medallion,rate_code,tip_amount,total_amount,store_and_fwd_flag,mta_tax,pickup_geocode,trip_time_in_secs,surcharge,vendor_id,tolls_amount,fare_amount,pickup_datetime,hack_license,payment_type,ordertime
> Note the first column header is "trip_distance". But the read data shows:
> taxi_simplified1.csv^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@0000644^@0010013^@3001121^@00000046004^@13002371150^@013521^@
>
> 0^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ustar
>
> ^@optitest^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@trip_distance
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
