Hi Ying,
I'm not familiar with Presto, so I can't answer this question for sure, but
your assumption sounds logical to me.
Br,
Zoltan
On Mon, May 20, 2019 at 6:44 PM ying wrote:
> Hi Zoltan:
>
> Thanks a lot for the information — wasn’t aware of these Jira efforts
> before.
>
> I assume once it’s in Hive, similar support will propagate to the original
> Presto parquet reader which uses Hive?
>
> Thanks.
>
> -
> Ying
>
> On Fri, May 17, 2019 at 4:00 AM Zoltan Ivanfi
> wrote:
>
> > Hi Ying,
> >
> > Int64 timestamp support is already in the works for Hive, but merging it
> > into the codebase is blocked on the release of parquet-mr 1.11.0 at this
> > moment. Here are the JIRA-s you can track:
> >
> > - HIVE-21215: Read Parquet INT64 timestamp
> > - HIVE-21216: Write Parquet INT64 timestamp
> >
> > There is an ongoing effort for Spark as well:
> >
> > - SPARK-26797: Start using the new logical types API of Parquet 1.11.0
> > instead of the deprecated one
> >
> > Br,
> >
> > Zoltan
> >
> > On Thu, May 16, 2019 at 6:03 PM ying wrote:
> >
> > > Hi Julien:
> > >
> > > Parquet appears to recommend using int64 for representation of
> timestamp
> > > (through the *timestamp-mills *and *timestamp-micros* logical types).
> > >
> > > However, in our use cases we are using Hive/Presto to load Parquet
> files.
> > > And found out that only int96 format is supported to represent
> timestamp
> > > (see below a number of related Hive JIRAs). Specifically, although
> Hive
> > > supports different formats for timestamp, when *loading from Parquet*
> > only
> > > int96 is supported as timestamp.
> > >
> > > https://issues.apache.org/jira/browse/HIVE-15079
> > >
> > > https://issues.apache.org/jira/browse/HIVE-13435
> > >
> > > https://issues.apache.org/jira/browse/HIVE-3844
> > >
> > >
> > > Just to confirm the above are known issues to the Parquet community.
> And
> > > are you aware of past/future efforts to add support for loading Parquet
> > > *int64
> > > as *timestamp in Hive?
> > >
> > >
> > > Thanks.
> > >
> > > -
> > > Ying
> > >
> > > On 2019/05/10 18:03:10, Julien Le Dem wrote:
> > > > Hi Arup,>
> > > > You are correct, you would have to use the lower level APIs or
> > > contribute>
> > > > the int96 support to either protobuf or avro integrations.>
> > > > However we are recommending users to migrate away from the int96 type
> > so
> > > I>
> > > > would not recommend adding that support.>
> > > > https://issues.apache.org/jira/browse/PARQUET-323>
> > > > Maybe check how the tools you use to query that data interpret int96
> > and>
> > > > int64, you might have a better solution moving to the new type and it
> > > being>
> > > > compatible.>
> > > >
> > > > On Fri, May 3, 2019 at 11:34 AM Arup Malakar
> wrote:>
> > > >
> > > > > Following up on the thread, my current understanding is that INT96
> is
> > > not a>
> > > > > native type in either of protobuf/avro, so the corresponding high
> > > level>
> > > > > parquet writers don’t support that. But `INT96` is supported by low
> > > level>
> > > > > parquet writer apis. I was able to generate parquet files with
> INT96
> > > using>
> > > > > examples from:>
> > > > >>
> > > > >
> > >
> > >
> >
> https://stackoverflow.com/questions/54657496/how-to-write-timestamp-logical-type-int96-to-parquet-using-parquetwriter
> > > >
> > >
> > > > >>
> > > > > Arup>
> > > > >>
> > > > > On Wed, May 1, 2019 at 7:32 PM Arup Malakar
> > wrote:>
> > > > >>
> > > > > > Hi parquet-dev,>
> > > > > >>
> > > > > > We have existing parquet files which were generated from json
> using
> > > hive,>
> > > > > > where timestamps live as INT96. We are changing the pipeline
> where
> > we
> > > are>
> > > > > > planning to use flink to generate parquet files from protobuf (or
> > > avro)>
> > > > > > using flink's StreamingFileSink. But from my research I am unable
> > to>
> > > > > find a>
> > > > > > way to write INT96 columns in the parquet either from avro or
> > > protobuf.>
> > > > > We>
> > > > > > would like to keep the same datatype on disk for historical and
> new
> > > data>
> > > > > so>
> > > > > > would like to stick to INT96, any suggestion how to achieve
> that?>
> > > > > >>
> > > > > > -->
> > > > > > Arup Malakar>
> > > > > >>
> > > > >>
> > > > >>
> > > > > -->
> > > > > Arup Malakar>
> > > > >>
> > > >
> > >
> >
>
