I had a look at https://github.com/apache/hive/blob/master/ql/src/java/ org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java#L2031
And read through isStripeSatisfyPredicate and pickStripesInternal and can see how stripe level indices are being used. But I still cannot find where and if File level indices are being used? File level statistics are however read at https://github.com/apache/orc/blob/b10ee7b35386d28b4b6fd2a5e724806d7ceb9db7/java/core/src/java/org/apache/orc/impl/ReaderImpl.java Line: 362,383 private final List<OrcProto.ColumnStatistics> fileStats; If anybody could point me to the right code file or any documentation, that would be great. On Tue, Jan 24, 2017 at 11:50 PM, Owen O'Malley <[email protected]> wrote: > That is my fault. We just haven't ported that part of the functionality > over yet. Hive's OrcInputFormat has a lot of complexity that most users > don't need or want. (It's types such as OrcStruct also don't actually work > as Writables, which causes users outside of Hive problems.) The > orc-mapreduce's types do work as Writables and thus work better outside of > Hive. That said, no one has ported the split elimination yet. > > .. Owen > > On Tue, Jan 24, 2017 at 2:45 PM, Lawan Subba < > [email protected]> wrote: > > > Hi Gopal, > > > > Thank you for the quick reply. > > > > I am new to open source projects, can you also tell me why this > > functionality is missing from the github repository for Apache ORC. > > > > Regards, > > Lawan Subba > > > > On Tue, Jan 24, 2017 at 8:26 PM, Gopal Vijayaraghavan <[email protected] > > > > wrote: > > > > > > > > > I can see that row indices are being used to select only rowgroups > > > that > > > > satisfy a search predicate in > > > … > > > > But, I cannot find where and if the stripe level indices are being > > > used? > > > > > > https://github.com/apache/hive/blob/master/ql/src/java/ > > > org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java#L2031 > > > > > > Read through isStripeSatisfyPredicate and pickStripesInternal. > > > > > > Cheers, > > > Gopal > > > > > > > > > > > > > > >
