> > Most generally, final users of X feature that a vendor sells supports > will not read the user and dev list because they buy that support. They > will say "do the needful" to that vendor. > Even if there are many of those final users, this is valid not only for > Solr but also for other Apache and non-Apache projects. >
As a project maintainer, I will also say "do the needful" to that vendor. Opensource is great, but it's not a free way to create custom features for your clients. If you are selling support for an opensource project, then you must support the features you use, especially when no one else is using them. It's not anyone else's responsibility to do that for you. Opensource communities are a two-way street. In this particular case, this is a very very niche part of the project that adds quite a lot of headache around dependencies and tests. If someone is selling a product with Solr that relies on these modules, then they need to maintain them. Currently it's falling to many of us, who do not use it, and would love to focus on making other (more widely used) parts of the project better. I think this is particularly true of the HDFS module, but it is also very true of the WIndows support (that's less taking our time, and rather just doing poorly). - Houston On Fri, Jan 17, 2025 at 9:39 PM David Smiley <dsmi...@apache.org> wrote: > This test failure report: > http://fucit.org/solr-jenkins-reports/failure-report.html > -- shows that HDFS tests have been failing for a while and are not > receiving any love. > > The best outcome for Solr's HDFS module is that someone / business that > uses it helps out to ensure it's maintained. We should probably remove the > module until/unless that happens, if it ever does. If someone's looking > into these failures to help out, please make your voice heard! > > Meanwhile, the removal appears underway: > https://issues.apache.org/jira/browse/SOLR-17609 so there is some urgency > of action. > > On Sat, Dec 21, 2024 at 4:56 PM Arrieta, Alejandro < > aarri...@perrinsoftware.com> wrote: > > > Hi, > > > > I read both dev and user lists every day in lurker mode, and yes, I read > > the Hadoop auth mail threads. > > But I am too low on the food chain pyramid, like bottom level, lol :-) > > I mentioned this mail list thread in the internal appropriate chat room. > > > > 2 comments: > > 1) Most generally, final users of X feature that a vendor sells supports > > will not read the user and dev list because they buy that support. They > > will say "do the needful" to that vendor. > > Even if there are many of those final users, this is valid not only for > > Solr but also for other Apache and non-Apache projects. > > 2) Indexing to a local file system (spinning rust and even faster on SSD) > > is faster than indexing to a distributed file system, which is generally > > correct. That does not mean a distributed file system is no longer used > or > > has advantages in specific scenarios, like indexers, mentioned later in > the > > documentation. > > > > Now, back below my rock. > > Happy holidays to all. > > > > Alejandro Arrieta > > > > On Sat, Dec 21, 2024 at 5:46 PM David Smiley <dsmi...@apache.org> wrote: > > > > > +1 to LinkedIn. As I said, let's see if they even notice without you > > > finding a POC to tell them. At least for a month, if you don't mind > :-) > > > I'm really curious if we hear from them. > > > > > > On Sat, Dec 21, 2024 at 11:13 AM David Eric Pugh > <de...@yahoo.com.invalid > > > > > > wrote: > > > > > > > > > > > I will drop an email to the user list in January when folks are back > to > > > > get some feedback. I will check my LinkedIn and see who I know that > > is > > > > involved in the Hadoop project and the companies that supported it > and > > > see > > > > if that shakes some feedback loose. > > > > > > > > In a perfect world, if we had someone excited about HDFS, they would > > > > either step up to become an involved committer in this project, OR > take > > > it > > > > over and move the code to their own independent repo. > > > > I did a bit more poking on Cloudera's site and they did an update to > > Solr > > > > 8.11. > > > > > > > > > > https://docs.cloudera.com/runtime/7.2.18/release-notes/topics/rt-pubc-whats-new-solr.html > > > . > > > > Also, I am not even sure that they use the HDFS setup!?? At least, > the > > > way > > > > I read > > > > > > > > > > https://community.cloudera.com/t5/Community-Articles/Understanding-Solr-Architecture-and-Best-practices/ta-p/248788 > > > > is that you want to use "Local FS" for best performance. So it may > be > > > that > > > > they can continue to use SolrCloud without HDFS. > > > > > > > > > > > > > > > > > > > > On Saturday, December 21, 2024 at 10:03:49 AM EST, David Smiley < > > > > dsmi...@apache.org> wrote: > > > > > > > > Any such proposal should start with an attempt to solicit user > input. > > > > > > > > I do wonder if we are doing enough to communicate important decisions > > > like > > > > this to our users (to solicit this feedback). I worded that poorly > > > maybe; > > > > I don't mean to suggest inadequacy on us necessarily as I have a > > greater > > > > concern on Solr users not adequately paying attention to the > > > news/direction > > > > of the Solr project. Ah; I'm reminded sadly of a failed attempt to > > have > > > a > > > > newsletter -- perhaps the perfect solution to this and > > > > other project engagement. > > > > I suggest we not reach out to specific users/organizations this time > > and > > > we > > > > see who responds. We know of a certain organization who contributed > > > these > > > > modules in the first place who almost certainly still use it. Let's > > see > > > if > > > > our outreach efforts catch their notice (and they respond) or not. > > > > > > > > Obviously we should get rid of it if nobody will maintain it. My > hope > > is > > > > that users/organizations step up and give the module whatever love it > > > > needs. Note that the HDFS module is Solr's *only* solution to > > something > > > > vaguely "cloud-native", at least a separation of storage from > compute. > > > It > > > > was only advertised for being about HDFS (the backend storage > solution) > > > and > > > > not advertised for its broader ability to use more modern choices > like > > > S3. > > > > I'm sure potential users made a hard pass on this because, of course, > > > they > > > > don't have and don't want to run HDFS. > > > > > > > > On Sat, Dec 21, 2024 at 8:36 AM David Eric Pugh > > <de...@yahoo.com.invalid > > > > > > > > wrote: > > > > > > > > > Should we remove the hdfs module from Solr 10? Inspired by some of > > the > > > > > discussions in > > > > > https://lists.apache.org/thread/lltc0wjdghq18tt37zlrsd8ty35qsytl > > > around > > > > > removing the hadoop-auth module, I think that this is a real > > > possiblity. > > > > > I found some earlier work in > > > > > https://issues.apache.org/jira/browse/SOLR-14660 and > > > > > https://issues.apache.org/jira/browse/SOLR-14021 that put us on > the > > > path > > > > > for removing hdfs from Solr. > > > > > > > > > > If folks wanted to make it a third party package or modules for > Solr, > > > the > > > > > code has been separated, so that should be much more feasible. > > > > > I know a lot of folks are on holiday, so I won't make any commits > > till > > > > > January when folks are back in order to let folks voice their > > opinons, > > > > > however I may explore putting together a PR to remove it to see > what > > > that > > > > > looks like... > > > > > Also, this is an interesting read: > > > > > https://cwiki.apache.org/confluence/display/SOLR/Deprecations > > > > > Eric > > > > > > > > > > > > > > > > > > > > > > > > >
