I am not the producer of data so I can not control encoding. I do receive ByteBuffer and encoding. I can decode data with given encoding and covert to UTF-8 for storing with Parquet. I was thinking to remove that overhead if possible
Regards Manik Singla +91-9996008893 +91-9665639677 "Life doesn't consist in holding good cards but playing those you hold well." On Wed, Feb 6, 2019 at 11:37 AM Uwe L. Korn <[email protected]> wrote: > Hello Manik, > > this is not possible at the moment. As Parquet is a portable on-disk > format, we focus on having a single representation for each data type. Thus > implementing readers/writers is limited to these to make their > implementation simpler. Especially as you are the producer but not the > consumer, even adding a new type would not solve your problem. You really > can only use a new logical type when it has been implemented in all the > readers and your consumers have all updated to these reader versions. > > As Unicode and thus UTF-8 support all characters one can think off, you > should always be able to convert strings to it. Given that Parquet files > encode and compress the data anyway afterwards, the conversion is a bit of > a CPU overhead but should not make a difference in size and form of the > data actually stored in the files. Also I guess that the UTF-16->UTF-8 > conversion costs less CPU that the Parquet compression process. > > Did this help you or is there any reason why you really cannot convert > your data to UTF-8? > > Uwe > > On Wed, Feb 6, 2019, at 6:19 AM, Manik Singla wrote: > > Hi > > > > I am new to Parquet. I am trying to save UTF-16 or some other encoding > than > > UTF-8. > > I am also trying to use encoding hint when saving ByteBuffer. > > > > I don't find way to use any thing other than UTF-8. > > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md > says > > we can extend primitive types to solve cases. > > > > Other thing I want to mention is I am only the producer of parquet file > but > > not consumer. > > > > Could you guide me which examples I can look into or which will be right > way > > > > > > Regards > > Manik Singla > > +91-9996008893 > > +91-9665639677 > > > > "Life doesn't consist in holding good cards but playing those you hold > > well." >
