-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Rahul,
On 4/23/20 21:49, dhurandar S wrote: > Thank you for your reply. The reason we are looking for S3 is since > the volume is close to 10 Petabytes. We are okay to have higher > latency of say twice or thrice that of placing data on the local > disk. But we have a requirement to have long-range data and > providing Seach capability on that. Every other storage apart from > S3 turned out to be very expensive at that scale. > > Basically I want to replace > > -Dsolr.directoryFactory=HdfsDirectoryFactory \ > > with S3 based implementation. Can you clarify whether you have 10 PiB of /source data/ or 10 PiB of /index data/? You can theoretically store your source data anywhere, of course. 10 PiB sounds like a truly enormous index. - -chris > On Thu, Apr 23, 2020 at 3:12 AM Jan Høydahl <jan....@cominvent.com> > wrote: > >> Hi, >> >> Is your data so partitioned that it makes sense to consider >> splitting up in multiple collections and make some arrangement >> that will keep only a few collections live at a time, loading >> index files from S3 on demand? >> >> I cannot see how an S3 directory would be able to effectively >> cache files in S3 and what units the index files would be stored >> as? >> >> Have you investigated EFS as an alternative? That would look like >> a normal filesystem to Solr but might be cheaper storage wise, >> but much slower. >> >> Jan >> >>> 23. apr. 2020 kl. 06:57 skrev dhurandar S >>> <dhurandarg...@gmail.com>: >>> >>> Hi, >>> >>> I am looking to use S3 as the place to store indexes. Just how >>> Solr uses HdfsDirectory to store the index and all the other >>> documents. >>> >>> We want to provide a search capability that is okay to be a >>> little slow >> but >>> cheaper in terms of the cost. We have close to 2 petabytes of >>> data on >> which >>> we want to provide the Search using Solr. >>> >>> Are there any open-source implementations around using S3 as >>> the >> Directory >>> for Solr ?? >>> >>> Any recommendations on this approach? >>> >>> regards, Rahul >> >> > -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIyBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl6iTwUACgkQHPApP6U8 pFjRaw/4sGbH286gZJe+wfKsLc4JPvyJZjjwVDCdpiR2SHt50IA23wYSK97R6xRj dbWWReA7C3JNWp6x21i8Bb6sIeLDnotbc7IOSmOMuNep1BtVaYBMJ8wyW6uUtXf6 hQbY0Ew93ZhDlS9CWMJqbQtWfrQEqH51Xbz+4uqqvJU8Bq9o9Vv0rnuVp/5f73lV ihek0sbA73oGle0gC5NFmrKItnn+14X8vIxUC8JRZlY4rDSiOdOcIil3DExxOQNQ UodIvwKKhzALFY77PeGSSjKiy0X3JJ1rKzLeIBrW0JCNMprYLzL2CQjZ5F09MraZ WxXdA64lEg2diEwHywNrsaaygbEZYTWd8gaeGA7kzCk78Y2KuhWuEQej6KmE3Iq2 AW+K7JgFakUpzB5oorCtKNLQOqFHX85ne57gCYKr42S3Htfxmf98pBdudQy4RvuT +tJvGYx8NLqgeOoZN4u+G/8WunlzUC+u2vUxVcIoK3Ozz0usMioFDqn69vmOxxoH cN2Y4T1ZZZGtndiAGZww1JXKAbVN0U41isXg2F8tHQV9dxaeoYDQ/xYbAoWEhhlM SVtEdr76eMJ08T6h5711gtrhSK+RQFPD2Jbr8B/Xl063xPfN2TpqmcJCKXkucvpc CEDLFqeKX6qIRZDgMf8EICmbFl6aF5knbDP0MkyYk4urB+uFaw== =Y/6Y -----END PGP SIGNATURE-----
