I am not the producer of data so I can not control encoding. I do receive
ByteBuffer and encoding.
I can decode data with given encoding and covert to UTF-8 for storing with
Parquet.
I was thinking to remove that overhead if possible

Regards
Manik Singla
+91-9996008893
+91-9665639677

"Life doesn't consist in holding good cards but playing those you hold
well."


On Wed, Feb 6, 2019 at 11:37 AM Uwe L. Korn <[email protected]> wrote:

> Hello Manik,
>
> this is not possible at the moment. As Parquet is a portable on-disk
> format, we focus on having a single representation for each data type. Thus
> implementing readers/writers is limited to these to make their
> implementation simpler. Especially as you are the producer but not the
> consumer, even adding a new type would not solve your problem. You really
> can only use a new logical type when it has been implemented in all the
> readers and your consumers have all updated to these reader versions.
>
> As Unicode and thus UTF-8 support all characters one can think off, you
> should always be able to convert strings to it. Given that Parquet files
> encode and compress the data anyway afterwards, the conversion is a bit of
> a CPU overhead but should not make a difference in size and form of the
> data actually stored in the files. Also I guess that the UTF-16->UTF-8
> conversion costs less CPU that the Parquet compression process.
>
> Did this help you or is there any reason why you really cannot convert
> your data to UTF-8?
>
> Uwe
>
> On Wed, Feb 6, 2019, at 6:19 AM, Manik Singla wrote:
> > Hi
> >
> > I am new to Parquet. I am trying to save UTF-16 or some other encoding
> than
> > UTF-8.
> > I am also trying to use encoding hint when saving ByteBuffer.
> >
> > I don't find way to use any thing other than UTF-8.
> > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
> says
> > we can extend primitive types to solve cases.
> >
> > Other thing I want to mention is I am only the producer of parquet file
> but
> > not consumer.
> >
> > Could you guide me which examples I can look into or which will be right
> way
> >
> >
> > Regards
> > Manik Singla
> > +91-9996008893
> > +91-9665639677
> >
> > "Life doesn't consist in holding good cards but playing those you hold
> > well."
>

Reply via email to