Hello Manik,

this is not possible at the moment. As Parquet is a portable on-disk format, we 
focus on having a single representation for each data type. Thus implementing 
readers/writers is limited to these to make their implementation simpler. 
Especially as you are the producer but not the consumer, even adding a new type 
would not solve your problem. You really can only use a new logical type when 
it has been implemented in all the readers and your consumers have all updated 
to these reader versions.

As Unicode and thus UTF-8 support all characters one can think off, you should 
always be able to convert strings to it. Given that Parquet files encode and 
compress the data anyway afterwards, the conversion is a bit of a CPU overhead 
but should not make a difference in size and form of the data actually stored 
in the files. Also I guess that the UTF-16->UTF-8 conversion costs less CPU 
that the Parquet compression process. 

Did this help you or is there any reason why you really cannot convert your 
data to UTF-8?

Uwe

On Wed, Feb 6, 2019, at 6:19 AM, Manik Singla wrote:
> Hi
> 
> I am new to Parquet. I am trying to save UTF-16 or some other encoding than
> UTF-8.
> I am also trying to use encoding hint when saving ByteBuffer.
> 
> I don't find way to use any thing other than UTF-8.
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md says
> we can extend primitive types to solve cases.
> 
> Other thing I want to mention is I am only the producer of parquet file but
> not consumer.
> 
> Could you guide me which examples I can look into or which will be right way
> 
> 
> Regards
> Manik Singla
> +91-9996008893
> +91-9665639677
> 
> "Life doesn't consist in holding good cards but playing those you hold
> well."

Reply via email to