[
https://issues.apache.org/jira/browse/BEAM-8423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jyoti Aditya updated BEAM-8423:
-------------------------------
Description:
I am running apache beam job to parse japanese html pages. While running the
job, I see in stackdriver log it is showing japanese character properly. But
same data written to GCS bucket has encoding issue and it is getting corrupted.
{noformat}
//code
Pipeline pipeline = Pipeline.create(options);
CoderRegistry cr = pipeline.getCoderRegistry();
cr.registerCoderForClass(String.class,
StringUtf8Coder.of());
cr.registerCoderForClass(Integer.class,
BigEndianIntegerCoder.of());
batchTuple = pipeline
.apply("Read from input files",
TextIO.read().from(options.getloadingBucketURL()).withCompression(Compression.GZIP)).setCoder(StringUtf8Coder.of())
.apply("Process input files",
ParDo.of(new
ExtractDataFromHtmlPage(extractionConfig, beamConfig.getLoadingBucketURL()))
.withOutputTags(successRecord,
TupleTagList.of(errorRecord).and(deadLetterRecords)));{noformat}
was:
I am running apache beam job to parse japanese html pages. While running the
job, I see in stackdriver log it is showing japanese character properly. But
same data written to GCS bucket has encoding issue and it is getting corrupted.
{noformat}
//code
Pipeline pipeline = Pipeline.create(options);
CoderRegistry cr = pipeline.getCoderRegistry();
cr.registerCoderForClass(String.class,
StringUtf8Coder.of());
cr.registerCoderForClass(Integer.class,
BigEndianIntegerCoder.of());
batchTuple = pipeline
.apply("Read from input files",
TextIO.read().from(options.getloadingBucketURL()).withCompression(Compression.GZIP)).setCoder(StringUtf8Coder.of())
.apply("Process input files",
ParDo.of(new
ExtractDataFromHtmlPage(extractionConfig, beamConfig.getLoadingBucketURL()))
.withOutputTags(successRecord,
TupleTagList.of(errorRecord).and(deadLetterRecords)));{noformat}
> Japanese characters encoding issue
> -----------------------------------
>
> Key: BEAM-8423
> URL: https://issues.apache.org/jira/browse/BEAM-8423
> Project: Beam
> Issue Type: Bug
> Components: beam-model
> Affects Versions: 2.15.0
> Environment: dataflow
> Reporter: Jyoti Aditya
> Priority: Major
>
> I am running apache beam job to parse japanese html pages. While running the
> job, I see in stackdriver log it is showing japanese character properly. But
> same data written to GCS bucket has encoding issue and it is getting
> corrupted.
>
> {noformat}
> //code
> Pipeline pipeline = Pipeline.create(options);
> CoderRegistry cr = pipeline.getCoderRegistry();
> cr.registerCoderForClass(String.class,
> StringUtf8Coder.of());
> cr.registerCoderForClass(Integer.class,
> BigEndianIntegerCoder.of());
> batchTuple = pipeline
> .apply("Read from input files",
>
> TextIO.read().from(options.getloadingBucketURL()).withCompression(Compression.GZIP)).setCoder(StringUtf8Coder.of())
> .apply("Process input files",
> ParDo.of(new
> ExtractDataFromHtmlPage(extractionConfig, beamConfig.getLoadingBucketURL()))
>
> .withOutputTags(successRecord,
> TupleTagList.of(errorRecord).and(deadLetterRecords)));{noformat}
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
