Hello,
I’m using Spark 3.1.1 and cannot yet update to a newer version. I have got a
data frame with a single column of DataTypes.BinaryType where each row contains
a byte array with generated binary data.
I try now to write this data in a single file with
mydataframe.coalesce( 1 )
.write()
.mode( SaveMode.Overwrite )
.format( "binaryFile" )
.save( „s3a://mybucket/myfile <s3a://mybucket/myfile>.bin“ );
But I get the error message „java.lang.UnsupportedOperationException: Write is
not supported for binary file data source“.
So my question is, how can I add to my current spark version a binary writer? I
have seen this tutorial
https://levelup.gitconnected.com/easy-guide-to-create-a-write-data-source-in-apache-spark-3-f7d1e5a93bdb
<https://levelup.gitconnected.com/easy-guide-to-create-a-write-data-source-in-apache-spark-3-f7d1e5a93bdb>
Is there an existing component which I can user under my version?
Thanks a lot for help
