That's funny. The line after is the rest of the whole line that got
split in half. Every following lines after that are fine.
I managed to reproduce without gzip also so maybe it's no gzip's fault
after all..
I'm clueless...
On Tue, Jun 14, 2016 at 12:53 PM, Kristoffer Sjögren <sto...@gmail.com> wrote:
> Seems like it's the gzip. It works if download the file, gunzip and
> put it back to another directory and read it the same way.
>
> Hm.. I wonder what happens with the lines after it..
>
>
>
> On Tue, Jun 14, 2016 at 11:52 AM, Sean Owen <so...@cloudera.com> wrote:
>> What if you read it uncompressed from HDFS?
>> gzip compression is unfriendly to MR in that it can't split the file.
>> It still should just work, certainly if the line is in one file. But,
>> a data point worth having.
>>
>> On Tue, Jun 14, 2016 at 10:49 AM, Kristoffer Sjögren <sto...@gmail.com>
>> wrote:
>>> The line is in one file. I did download the file manually from HDFS,
>>> read and decoded it line-by-line successfully without Spark.
>>>
>>>
>>>
>>> On Tue, Jun 14, 2016 at 11:44 AM, Sean Owen <so...@cloudera.com> wrote:
>>>> The only thing I can think of is that a line is being broken across two
>>>> files?
>>>> Hadoop easily puts things back together in this case, or should. There
>>>> could be some weird factor preventing that. One first place to look:
>>>> are you using a weird line separator? or at least different from the
>>>> host OS?
>>>>
>>>> On Tue, Jun 14, 2016 at 10:41 AM, Kristoffer Sjögren <sto...@gmail.com>
>>>> wrote:
>>>>> I should mention that we're in the end want to store the input from
>>>>> Protobuf binary to Parquet using the following code. But this comes
>>>>> after the lines has been decoded from base64 into binary.
>>>>>
>>>>>
>>>>> public static <T extends Message> void save(JavaRDD<T> rdd, Class<T>
>>>>> clazz, String path) {
>>>>> try {
>>>>> Job job = Job.getInstance();
>>>>> ParquetOutputFormat.setWriteSupportClass(job,
>>>>> ProtoWriteSupport.class);
>>>>> ProtoParquetOutputFormat.setProtobufClass(job, clazz);
>>>>> rdd.mapToPair(order -> new Tuple2<>(null, order))
>>>>> .saveAsNewAPIHadoopFile(path, Void.class, clazz,
>>>>> ParquetOutputFormat.class, job.getConfiguration());
>>>>> } catch (IOException e) {
>>>>> throw new RuntimeException(e);
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> <dependency>
>>>>> <groupId>org.apache.parquet</groupId>
>>>>> <artifactId>parquet-protobuf</artifactId>
>>>>> <version>1.8.1</version>
>>>>> </dependency>
>>>>>
>>>>> On Tue, Jun 14, 2016 at 11:37 AM, Kristoffer Sjögren <sto...@gmail.com>
>>>>> wrote:
>>>>>> I'm trying to figure out exactly what information could be useful but
>>>>>> it's all as straight forward.
>>>>>>
>>>>>> - It's text files
>>>>>> - Lines ends with a new line character.
>>>>>> - Files are gzipped before added to HDFS
>>>>>> - Files are read as gzipped files from HDFS by Spark
>>>>>> - There are some extra configuration
>>>>>>
>>>>>> conf.set("spark.files.overwrite", "true");
>>>>>> conf.set("spark.hadoop.validateOutputSpecs", "false");
>>>>>>
>>>>>> Here's the code using Java 8 Base64 class.
>>>>>>
>>>>>> context.textFile("/log.gz")
>>>>>> .map(line -> line.split("×tamp="))
>>>>>> .map(split -> Base64.getDecoder().decode(split[0]));
>>>>>>
>>>>>>
>>>>>> On Tue, Jun 14, 2016 at 11:26 AM, Sean Owen <so...@cloudera.com> wrote:
>>>>>>> It's really the MR InputSplit code that splits files into records.
>>>>>>> Nothing particularly interesting happens in that process, except for
>>>>>>> breaking on newlines.
>>>>>>>
>>>>>>> Do you have one huge line in the file? are you reading as a text file?
>>>>>>> can you give any more detail about exactly how you parse it? it could
>>>>>>> be something else in your code.
>>>>>>>
>>>>>>> On Tue, Jun 14, 2016 at 10:24 AM, Kristoffer Sjögren <sto...@gmail.com>
>>>>>>> wrote:
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> We have log files that are written in base64 encoded text files
>>>>>>>> (gzipped) where each line is ended with a new line character.
>>>>>>>>
>>>>>>>> For some reason a particular line [1] is split by Spark [2] making it
>>>>>>>> unparsable by the base64 decoder. It does this consequently no matter
>>>>>>>> if I gives it the particular file that contain the line or a bunch of
>>>>>>>> files.
>>>>>>>>
>>>>>>>> I know the line is not corrupt because I can manually download the
>>>>>>>> file from HDFS, gunzip it and read/decode all the lines without
>>>>>>>> problems.
>>>>>>>>
>>>>>>>> Was thinking that maybe there is a limit to number of characters per
>>>>>>>> line but that doesn't sound right? Maybe the combination of characters
>>>>>>>> makes Spark think it's new line?
>>>>>>>>
>>>>>>>> I'm clueless.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> -Kristoffer
>>>>>>>>
>>>>>>>> [1] Original line:
>>>>>>>>
>>>>>>>> CsAJCtwGCghwYWdlVmlldxC4PhjM1v66BSJFaHR0cDovL25hLnNlL3Nwb3J0ZW4vc3BvcnR0dC8xLjM5MTU5MjEtdXBwZ2lmdGVyLXNtZWRlcm5hLW1vdC1rb25rdXJzKjhVcHBnaWZ0ZXI6IFNtZWRlcm5hIG1vdCBrb25rdXJzIC0gU3BvcnQgKFRUKSAtIHd3dy5uYS5zZTJXaHR0cDovL25hLnNlL255aGV0ZXIvb3JlYnJvLzEuMzk2OTU0My1rcnlwaGFsLW9wcG5hci1mb3Itb2JvLWF0dC1iZWhhbGxhLXRqYW5zdGViaWxhcm5hOqECcWdrZWplNGpmLHBkMzBmdDRuNCxwbHB0b3JqNncscGxwczBvamZvLHBkYjVrZGM4eCxwbHBzN293Y3UscGE0czN1bXp5LHBhNHJla25sNyxwYTRyd3dxam4scGE0c21ra2Z4LHBkM2tpa3BmMixwZDNqcjE5dGMscGQ0ZGQ0M2F3LHAwZ3MwbmlqMSxwYTRvZTNrbXoscGE0cWJ3eDZxLHBkM2s2NW00dyxwYTRyazc3Z3IscGQzMHAzdW8wLHBkNGM1ajV5dixwbHB0c211NmcscGM3bXNibmM5LHBhNHFpaTdsZCxwbHB0dnpqdnUscGE0bmlsdmFnLHBhNHB6cjN2cyxwZDNsZDNnYmkscGl1a2M2NmVlLHB5dHoyOThzNErIAgoTNzI0NTY2NzU0MzQxNTUyOTQ4ORAAGAAioQJxZ2tlamU0amYscGQzMGZ0NG40LHBscHRvcmo2dyxwbHBzMG9qZm8scGRiNWtkYzh4LHBscHM3b3djdSxwYTRzM3VtenkscGE0cmVrbmw3LHBhNHJ3d3FqbixwYTRzbWtrZngscGQza2lrcGYyLHBkM2pyMTl0YyxwZDRkZDQzYXcscDBnczBuaWoxLHBhNG9lM2tteixwYTRxYnd4NnEscGQzazY1bTR3LHBhNHJrNzdncixwZDMwcDN1bzAscGQ0YzVqNXl2LHBscHRzbXU2ZyxwYzdtc2JuYzkscGE0cWlpN2xkLHBscHR2emp2dSxwYTRuaWx2YWcscGE0cHpyM3ZzLHBkM2xkM2diaSxwaXVrYzY2ZWUscHl0ejI5OHM0KgkzOTUxLDM5NjAS3gIIxNjxhJTVsJcVEqUBTW96aWxsYS81LjAgKExpbnV4OyBBbmRyb2lkIDUuMS4xOyBTQU1TVU5HIFNNLUczODhGIEJ1aWxkL0xNWTQ4QikgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgU2Ftc3VuZ0Jyb3dzZXIvMy4zIENocm9tZS8zOC4wLjIxMjUuMTAyIE1vYmlsZSBTYWZhcmkvNTM3LjM2IjUKDDYyLjIwLjE5Ni44MBWgd3NBHRgibUIiAlNFKgfDlnJlYnJvMg5UZWxpYVNvbmVyYSBBQigAMdejcD0K1+s/OABCCAiAAhWamRlAQgcIURUAAOBAQggIlAEVzczMP0IHCFQVmpkJQUIICJYBFTMzE0BCBwhYFZqZ+UBCCAj6ARWamdk/QggImwEVzcysQEoHCAYVO6ysPkoHCAQVRYO4PkoHCAEVIg0APw==×tamp=1465887564
>>>>>>>>
>>>>>>>>
>>>>>>>> [2] Line as spark hands it over:
>>>>>>>>
>>>>>>>> CsAJCtwGCghwYWdlVmlldxC4PhjM1v66BSJFaHR0cDovL25hLnNlL3Nwb3J0ZW4vc3BvcnR0dC8xLjM5MTU5MjEtdXBwZ2lmdGVyLXNtZWRlcm5hLW1vdC1rb25rdXJzKjhVcHBnaWZ0ZXI6IFNtZWRlcm5hIG1vdCBrb25rdXJzIC0gU3BvcnQgKFRUKSAtIHd3dy5uYS5zZTJXaHR0cDovL25hLnNlL255aGV0ZXIvb3JlYnJvLzEuMzk2OTU0My1rcnlwaGFsLW9wcG5hci1mb3Itb2JvLWF0dC1iZWhhbGxhLXRqYW5zdGViaWxhcm5hOqECcWdrZWplNGpmLHBkMzBmdDRuNCxwbHB0b3JqNncscGxwczBvamZvLHBkYjVrZGM4eCxwbHBzN293Y3UscGE0czN1bXp5LHBhNHJla25sNyxwYTRyd3dxam4scGE0c21ra2Z4LHBkM2tpa3BmMixwZDNqcjE5dGMscGQ0ZGQ0M2F3LHAwZ3MwbmlqMSxwYTRvZTNrbXoscGE0cWJ3eDZxLHBkM2s2NW00dyxwYTRyazc3Z3IscGQzMHAzdW8wLHBkNGM1ajV5dixwbHB0c211NmcscGM3bXNibmM5LHBhNHFpaTdsZCxwbHB0dnpqdnUscGE0bmlsdmFnLHBhNHB6cjN2cyxwZDNsZDNnYmkscGl1a2M2NmVlLHB5dHoyOThzNErIAgoTNzI0NTY2NzU0MzQxNTUyOTQ4ORAAGAAioQJxZ2tlamU0amYscGQzMGZ0NG40LHBscHRvcmo2dyxwbHBzMG9qZm8scGRiNWtkYzh4LHBscHM3b3djdSxwYTRzM3VtenkscGE0cmVrbmw3LHBhNHJ3d3FqbixwYTRzbWtrZngscGQza2lrcGYyLHBkM2pyMTl0YyxwZDRkZDQzYXcscDBnczBuaWoxLHBhNG9lM2tteixwYTRxYnd4NnEscGQzazY1bTR3LHBhNHJrNzdncixwZDMwcDN1bzAscGQ0YzVqNXl2LHBscHRzbXU2ZyxwYzdtc2JuYzkscGE0cWlpN2xkLHBscHR2emp2dSxwYTRuaWx2YWcscGE0
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>>>>>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>>>>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
