[jira] [Created] (BEAM-8423) Japanese characters encoding issue

[Jyoti Aditya (Jira)]

Jyoti Aditya created BEAM-8423:
----------------------------------

             Summary: Japanese characters encoding issue 
                 Key: BEAM-8423
                 URL: https://issues.apache.org/jira/browse/BEAM-8423
             Project: Beam
          Issue Type: Bug
          Components: beam-model
    Affects Versions: 2.15.0
         Environment: dataflow
            Reporter: Jyoti Aditya


I am running apache beam job to parse japanese html pages. While running the 
job, I see in stackdriver log it is showing japanese character properly. But 
same data written to GCS bucket has encoding issue and it is getting corrupted.

 
{noformat}
//code


Pipeline pipeline = Pipeline.create(options);
                        CoderRegistry cr = pipeline.getCoderRegistry();
                        cr.registerCoderForClass(String.class, 
StringUtf8Coder.of());
                        cr.registerCoderForClass(Integer.class, 
BigEndianIntegerCoder.of());




                        batchTuple = pipeline
                                        .apply("Read from input files",
                                                        
TextIO.read().from(options.getloadingBucketURL()).withCompression(Compression.GZIP)).setCoder(StringUtf8Coder.of())
                                        .apply("Process input files",
                                                        ParDo.of(new 
ExtractDataFromHtmlPage(extractionConfig, beamConfig.getLoadingBucketURL()))
                                                                        
.withOutputTags(successRecord, 
TupleTagList.of(errorRecord).and(deadLetterRecords)));{noformat}
 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)