Hi All, Currently I am using ElasticSearch for a logging system. My first solution is that every log will put on ES and index will rolling by date. To do real time stats, I will use Aggregation. To do statistic I will use Spark (or Hive, Shark whatever) on ES data (thanks to ElasticSearch-Hadoop plugin All is fine, but when my data grows (currently 17M record/index/date), Spark (Hive also) becomes very slow.
I did benmark with the same data on ES and Hadoop, and I saw that Spark (Hive) run on Hadoop is much faster. Is there something I have missed ? Currently I maintain two storage, one is ES for realtime stats, another is Hadoop for other statistics. Is it fine ? I really appreciate any idea or solution., Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/103fb68e-65e8-4b1c-9e75-b34d393b7210%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
