Re: Filesystems supported by Solr
On 3/2/2018 2:30 PM, Ritesh Chaman wrote: > I am trying to deploy solr on my ADLS subscription. can you tell me if that > is tested and is compatible. Walter says that this is storage related to Azure. https://azure.microsoft.com/en-us/services/data-lake-store/ If this is what you are talking about, that page says they use HDFS APIs. Which means that you MIGHT be able to use it with the HDFS support built into Solr. You'd have to ask Microsoft what they can support. But I can almost guarantee that unless you have plenty of memory in the system for HDFS to cache that data, that it's going to be VERY slow, just due to the amount of latency involved in accessing data over the Internet. Accessing the data after a reboot or a restart of the particular service that caches the data is probably going to also be extremely slow, even if you DO have plenty of memory for caching. You'll see far better performance if you just install enough disk space in your Solr servers to hold your index data, and install enough memory that the OS can effectively cache that index data. Thanks, Shawn
Re: Filesystems supported by Solr
From a quick google search, ADLS seems like the Azure version of S3. Putting Solr indexes on S3 would be unbelievably slow, if it worked at all. Years ago, I accidentally put indexes on NFS and it was 100X slower. Tell us more about what you are trying to do. It is unusual to put Solr indexes on anything but a local filesystem. HDFS is the only exception I can think of. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Mar 2, 2018, at 1:30 PM, Ritesh Chamanwrote: > > Hi team > > I am trying to deploy solr on my ADLS subscription. can you tell me if that > is tested and is compatible. > > Regards > > On Tue, Feb 20, 2018 at 2:22 PM, Ritesh Chaman > wrote: > >> Hi team >> >> May I know what all filesystems are supported by Solr. For eg ADLS,WASB, >> S3 etc. Thanks. >> >> Ritesh >>
Re: Filesystems supported by Solr
Hi team I am trying to deploy solr on my ADLS subscription. can you tell me if that is tested and is compatible. Regards On Tue, Feb 20, 2018 at 2:22 PM, Ritesh Chamanwrote: > Hi team > > May I know what all filesystems are supported by Solr. For eg ADLS,WASB, > S3 etc. Thanks. > > Ritesh >
Re: Filesystems supported by Solr
On 2/20/2018 3:22 PM, Ritesh Chaman wrote: > May I know what all filesystems are supported by Solr. For eg ADLS,WASB, S3 > etc. Thanks. Solr supports whatever your operating system supports. It will expect file locking to work be fully functional, so things like NFS don't always work. Local filesystems are very much preferred, and will generally have the best performance. As far as I am aware, the only filesystem that Solr has explicit support for (outside of what the OS itself provides) is HDFS. https://lucene.apache.org/solr/guide/7_2/running-solr-on-hdfs.html There may be plugins available to store indexes in other stores like S3, but if those exist, I am not immediately aware of them. They would be third-party plugins, not supported by the Solr project. Thanks, Shawn
Re: Filesystems supported by Solr
Ritesh The filesystems you mention are used by Spark so it can stream huge quantities of data (corrections please). By comparison, Solr uses a more 'reasonable' sized filesystem, but needs enough memory that all the index data can be resident. The regular Linux ext3 or ext4 is fine. If you are integrating Solr with Spark, then the filesystems you mention would be for Spark not Solr. Cheers -- Rick On February 20, 2018 5:22:33 PM EST, Ritesh Chamanwrote: >Hi team > >May I know what all filesystems are supported by Solr. For eg >ADLS,WASB, S3 >etc. Thanks. > >Ritesh -- Sorry for being brief. Alternate email is rickleir at yahoo dot com