Thank you for the response. Can I please know the reason why bit map indexes are not appropriate for big data. Rather than using the traditional bitmap indexing techniques we are planning to implement a combination of novel bitmap indexing techniques like bit sliced indexes and projection indexes. Furthermore, can I please know whether bloom filters have already been implemented in Spark.
Thank you On Thu, Jun 30, 2016 at 12:51 AM, Jörn Franke <jornfra...@gmail.com> wrote: > > Is it the traditional bitmap indexing? I would not recommend it for big > data. You could use bloom filters and min/max indexes in-memory which look > to be more appropriate. However, if you want to use bitmap indexes then you > would have to do it as you say. However, bitmap indexes may consume a lot > of memory, so I am not sure that simply caching them in-memory is desired. > > > On 29 Jun 2016, at 19:49, Nishadi Kirielle <ndime...@gmail.com> wrote: > > > > Hi All, > > > > I am a CSE undergraduate and as for our final year project, we are > expecting to construct a cluster based, bit-oriented analytic platform > (storage engine) to provide fast query performance when used for OLAP with > the use of novel bitmap indexing techniques when and where appropriate. > > > > For that we are expecting to use Spark SQL. We will need to implement a > way to cache the bit map indexes and in-cooperate the use of bitmap > indexing at the catalyst optimizer level when it is possible. > > > > I would highly appreciate your feedback regarding the proposed approach. > > > > Thank you & Regards > > > > Nishadi Kirielle > > Department of Computer Science and Engineering > > University of Moratuwa > > Sri Lanka >
