partition columns with StreamingFileSink

[Yitzchak Lieberman]

Hi.

I'm using the StreamingFileSink for writing partitioned data to s3.
The code is below:

StreamingFileSink<GenericRecord> sink =
StreamingFileSink.forBulkFormat(new Path("s3a://test-bucket/test"),
            ParquetAvroFactory.getParquetWriter(schema, "GZIP"))
.withBucketAssigner(new PartitionBucketAssigner(partitionColumns))
.build();

How can i remove the partition columns from the data (or not populating
them in the GenericRecord)?
My problem is with AWS Glue crawler which creates duplicate columns in the
table.

Thanks,
Yitzchak.