> What is the way forward? My proposal would be to proceed with SIP-24 as an > ADDED layer of defense ASAP. If it after a year turns out it is hard to maintain or otherwise not worth it, just drop it.
I don't expect we'll have much feedback-wise in a year, given how long it takes many folks to upgrade. But otherwise this SGTM. On the network side in particular I have qualms about taking ownership of this stuff when there's great third-party firewalls out there. But lacking broad community agreement on that point I don't want to let "perfect be the enemy of good" here. SIP-24 is undoubtedly an improvement, so let's move that direction! On Fri, Jun 19, 2026 at 5:10 AM Jan Høydahl <[email protected]> wrote: > > Hi, > > Thanks for the feedbacks and reviews David, Eric, Gus and Jason. Trying to > sum up: > > Core principle is we'd like to push as much of network protection, i.e. who > can talk to who, > downstream to user's network environment, firewalls etc. > > Encourage users to utilize OS primitives to sandbox the Solr process, whether > that be > linux seccomp, CAP_* controls and filesystem isolation or similar on other > OS'es. > > It was discussed whether we can frame our new Threat Model in a way that > redefines some > attack vectors to be the responsibility of the USER. But we cannot push > everything on them. > > > Even if we encourage the above, and it will limit the blast radius of a > successfuly exploited > code vulnerability in Solr, I still sense an agreement that we do want to > avoid app-level > code in Solr that allows unintended filesystem path traversal, file read > outside known > solr areas, file write outside index or backup folders etc. And also that our > current allowPaths > strategy is spotty and that the agent approach centralises and improves upon > this. > > The utility for intercepting outbound network traffic in the agent is a > balancing act between > convenience and security, and it overlaps partly with external firewall > configs. However, it is > not uncommon to deploy Solr on the same network segment as databases and > other services. > The agent gives us and users a tool to contain a possible SSRF resulting from > a Solr bug. > > > So given the above, if we want to give SIP-24 and an agent a try, the > concerns flagges are > twofold - more code to maintain and more code that could break. > > Wrt "more code to maintain" an option would be to gather core agent code into > a separate > project, perhaps shared with Opensearch and others with similar needs. > > Wrt "more code that could break", yes there could be bugs, but I'd argue that > on JDK25 without > JSM, the new agent would not increase attack surface, but there could be bugs > making it > fail to protect certain cases we thought were protected. > > > What is the way forward? My proposal would be to proceed with SIP-24 as an > ADDED layer > of defense ASAP. If it after a year turns out it is hard to maintain or > otherwise not worth it, > just drop it. Once released, investigate possibility of lifting the core > agent out into a new > project (apache-commons?) and refactor if successful in a later release. Then > we get the > benefit of added protection immediately and get a delayed benefit of lower > maintenance. > > Jan > > > > > 10. juni 2026 kl. 17:01 skrev David Smiley <[email protected]>: > > > > Presumably the OpenSearch project would also appreciate reduced maintenance > > if this aspect of their project were extracted to an independent project. > > It's worth chatting with them about this, yeah? It's also maybe exciting > > for those involved in that mechanism to get more exposure -- more projects > > using their cool stuff. > > > > On Wed, Jun 10, 2026 at 4:01 AM Jason Gerlowski <[email protected]> > > wrote: > > > >> To clarify - I'm not assuming any particular answer to that question. > >> If folks think the module won't be high-maintenance, then +1. I just > >> want to make sure the question gets asked is all. > >> > >> (And if we're not sure about the maintenance implications, IMO it'd be > >> a great option to move forward with SIP-24 and aim to re-evaluate in a > >> year or two when we do have a better sense. I see SIP-24 as a clear > >> improvement, so we shouldn't let "perfect be the enemy of good" here.) > >> > >> On Wed, Jun 10, 2026 at 9:54 AM Jason Gerlowski <[email protected]> > >> wrote: > >>> > >>>> Or are you hinting that we'll define Solr's security model in such a > >> way that > >>>> we'll no longer treat arbitrary filesystem read/write or arbitrary > >> network access > >>>> as a vulnerability? > >>> > >>> Kindof, yeah, I think we should at least consider it. (Obviously > >>> communication-wise I wouldn't phrase it in terms of what reports we'd > >>> reject going forward. I'd frame it in terms of shifting where the > >>> protection is coming from. e.g. "Rather than relying on our security > >>> code, rely instead on proven security tooling like: <list of > >>> options>") > >>> > >>> I love everything about the ByteBuddy / security-agent approach, in a > >>> technical sense. It's a huge improvement over our current setup. But > >>> it does come with a maintenance cost and it's worth weighing that for > >>> ourselves. OpenSearch decided the tradeoff was worth it. But they > >>> have more corporate backing and likely more community bandwidth than > >>> we do, so we might weigh the same concerns and reach a different > >>> conclusion. > >>> > >>> If we're fully confident in the protections offered by security-agent, > >>> and we're OK spending at least some bandwidth on responding to future > >>> security-agent bugs, vuln reports, code maintenance, etc....great! If > >>> not, then maybe a more off-the-shelf approach to securing this stuff > >>> is a better fit for us. > >>> > >>> Best, > >>> > >>> Jason > >>> > >>> On Tue, Jun 9, 2026 at 7:51 AM Jan Høydahl <[email protected]> > >> wrote: > >>>> > >>>> Could have tried to use stock OpenSearch agent, but they do things a > >> bit differently in the bootstrap and we may end up asking them to do > >> changes to be more generic and accommodate us. > >>>> > >>>> Another option is to start a new Apache commons sub project, > >> commons-sec-agent pull that as a dependency, and promote it for others to > >> use and help maintain. > >>>> > >>>> Jan Høydahl > >>>> > >>>>> 7. juni 2026 kl. 18:15 skrev David Smiley <[email protected]>: > >>>>> > >>>>> Well written Jason! I agree 100% and I didn't communicate this > >> well enough > >>>>> previously when I used fewer words to essentially communicate the > >> same. > >>>>> > >>>>> I see the agent coming in SIP-24 is a forked copy, and thus we must > >>>>> maintain it. That's unfortunate. I hoped we could instead depend on > >>>>> another project regularly maintaining and producing artifacts for > >> 3rd party > >>>>> (us) consumption. > >>>>> > >>>>> Perhaps the agent will allow us to remove from Solr the > >> AllowUrlChecker > >>>>> (whatever it's called) and similar for file paths. I hope we get > >> some > >>>>> maintenance "relief". > >>>>> > >>>>>> On Fri, Jun 5, 2026 at 2:48 PM Jason Gerlowski < > >> [email protected]> > >>>>>> wrote: > >>>>>> > >>>>>> Apologies for chiming in so late on this. > >>>>>> > >>>>>> The Java agent route seems like a huge improvement over our existing > >>>>>> approach. I love that it's a global protection - it doesn't rely on > >>>>>> devs remembering to call the required "check the path" method in > >> every > >>>>>> single place we might read a file in our code. > >>>>>> > >>>>>> But ultimately I still think systemd/seccomp is a much more > >>>>>> sustainable route for us than the Java-agent approach > >>>>>> > >>>>>> We're a Search community that's here to solve people's Search > >>>>>> problems. I won't minimize for a second all the effort we've put > >> into > >>>>>> security over the years. But there are entire projects that exist to > >>>>>> solve these sorts of problems. IMO steering users towards something > >>>>>> like systemd that has a whole community of security folks behind is > >>>>>> going to leave them more secure, and be more sustainable for us. > >>>>>> > >>>>>> (To be clear I'm not advocating for systemd specifically, but for > >> the > >>>>>> general idea of relying on well-known, OS-level protections that > >> exist > >>>>>> outside of the JVM and Solr.) > >>>>>> > >>>>>> I've heard two objections to the "systemd" approach so far: that > >>>>>> systemd isn't cross-platform, and that some folks won't bother to > >>>>>> enable it. > >>>>>> > >>>>>> To the first objection, I'd say that our responsibility has never > >> been > >>>>>> to find one security solution that works for everyone. We never > >>>>>> mandated that folks use a particular firewall: we let users choose > >> how > >>>>>> they wanted to provide that network isolation. IMO this could be > >>>>>> handled the same way: point folks to a few different tools, > >> optionally > >>>>>> provide a few example configs, and let them pick based on their > >>>>>> OS/platform. > >>>>>> > >>>>>> To the second objection: this is the whole point of having a > >> security > >>>>>> model. It's expected and normal for projects to describe how to > >> make > >>>>>> a deployment of their software "secure". It's neither fair nor > >>>>>> possible to secure folks who won't read our "Going to Production" > >> docs > >>>>>> before deploying. > >>>>>> > >>>>>>> On Thu, May 28, 2026 at 8:06 AM Jan Høydahl <[email protected]> > >> wrote: > >>>>>>> > >>>>>>> So the implementation of SIP-24 is now ready for review at > >>>>>>> > >>>>>>> https://github.com/apache/solr/pull/4471 > >>>>>>> > >>>>>>> The PR description gives through description of the change and > >> what to > >>>>>> look for. > >>>>>>> I have run several rounds of Copilot reviews and Claude reviews. > >>>>>>> We have unit tests and BATS tests, so coverage should be fairly > >> good. > >>>>>>> > >>>>>>> The PR is fairly large +5454 lines across 62 files and 54 commits, > >> but > >>>>>> mostly > >>>>>>> contained in the new gradle module, plus one hook in CoreContainer. > >>>>>>> > >>>>>>> Take it for a spin and let me know how it works. > >>>>>>> > >>>>>>> Jan > >>>>>>> > >>>>>>>> 11. mai 2026 kl. 00:04 skrev Jan Høydahl <[email protected]>: > >>>>>>>> > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>> Welcoming more feedback on this approach. > >>>>>>>> > >>>>>>>> I'm planning to move to implementation phase on Wednesday, to get > >> a > >>>>>> first > >>>>>>>> version of the agent ready for testing. But it would be helpful > >> with > >>>>>> as much > >>>>>>>> feedback on the SIP high level design before I start implementing. > >>>>>> Thanks David > >>>>>>>> for your intial feedback. > >>>>>>>> > >>>>>>>> Jan > >>>>>>>> > >>>>>>>>> 1. mai 2026 kl. 14:30 skrev Gus Heck <[email protected]>: > >>>>>>>>> > >>>>>>>>> I know it's probably unrealistic because corporate environments > >> into > >>>>>> which > >>>>>>>>> solr is deployed likely would have difficulty with it, but there > >> is a > >>>>>> JDK > >>>>>>>>> fork that keeps and improves the Security Manager... It's > >> supported > >>>>>> by the > >>>>>>>>> descendants of the Apache River project. > >>>>>>>>> https://github.com/pfirmstone/DirtyChai > >>>>>>>>> > >>>>>>>>> Probably not useful, but perhaps interesting. Certainly we are > >> not > >>>>>> the only > >>>>>>>>> ones irritated by the loss of the security manager. > >>>>>>>>> > >>>>>>>>> On Thu, Apr 30, 2026 at 4:08 AM Jan Høydahl < > >> [email protected]> > >>>>>> wrote: > >>>>>>>>> > >>>>>>>>>>> I should clarify something. My objections mostly relate to the > >>>>>> insistent > >>>>>>>>>>> language in the SIP requiring Solr to have a substitute for the > >>>>>> JSM. I'm > >>>>>>>>>>> not quite against doing something but I might vote -0 on > >> something > >>>>>> that > >>>>>>>>>>> seems to have a poor security payoff relative to the > >> maintenance > >>>>>> burden. > >>>>>>>>>>> The more off-the-shelf, the better IMO. > >>>>>>>>>>> Thank you for investing your time researching some options. > >>>>>>>>>> > >>>>>>>>>> The security landscape has dramatically changed during the years > >>>>>> that we > >>>>>>>>>> have enjoyed JSM protection for Solr. It's a crazy world out > >> there > >>>>>> and > >>>>>>>>>> every > >>>>>>>>>> single code flaw, existing and future, will be found and > >> exploited or > >>>>>>>>>> published > >>>>>>>>>> by so called security researchers. That's why I believe we need > >> a > >>>>>>>>>> centralized > >>>>>>>>>> solution. > >>>>>>>>>> > >>>>>>>>>>>> Rejected Alternatives > >>>>>>>>>>>> > >>>>>>>>>>>> - *Staying on Java < 24* — not viable long-term; Solr must > >> support > >>>>>>>>>>>> current Java LTS releases. > >>>>>>>>>>>> - *Removing JSM protections without any replacement* — > >> unacceptable > >>>>>>>>>>>> security regression. > >>>>>>>>>>>> > >>>>>>>>>>> Both Eric Pugh and I have challenged this. > >>>>>>>>>>>> > >>>>>>>>>>>> - *OS-level hardening only (systemd, seccomp)* — not > >>>>>> cross-platform; > >>>>>>>>>>>> does not cover Windows or macOS. > >>>>>>>>>>>> > >>>>>>>>>>> I challenge this. Why should the Solr project burden itself > >> with > >>>>>> building > >>>>>>>>>>> & maintaining security mechanisms already provided by > >> off-the-shelf > >>>>>>>>>> tools? > >>>>>>>>>>> If a user/operator wishes to run on Windows/macOS that may not > >> have > >>>>>> this > >>>>>>>>>>> protection mechanism, it is a risk consideration for that user > >> to > >>>>>>>>>> consider, > >>>>>>>>>>> but isn't a deal-breaker. The JSM wasn't/isn't a front-line > >>>>>> defense; > >>>>>>>>>> it's > >>>>>>>>>>> a defense-in-depth strategy. Put differently, the protections > >> here > >>>>>> are > >>>>>>>>>>> "best effort" but not worthy of a CVE if they were to falter. > >> I > >>>>>> want to > >>>>>>>>>>> get Arnout's opinion on this supposition. > >>>>>>>>>> > >>>>>>>>>> Java Security Manager is a beast, but its file system read/write > >>>>>> controls > >>>>>>>>>> have > >>>>>>>>>> saved us many a CVE in 9.x when JSM have been on by default. > >>>>>>>>>> The SIP primarily focuses on sandboxing Solr for wrt file and > >> network > >>>>>>>>>> access. > >>>>>>>>>> The security landscape has changed dramatically last few years, > >> and > >>>>>> solving > >>>>>>>>>> file- and network restrictions in a central place through > >>>>>> interception > >>>>>>>>>> rathen than > >>>>>>>>>> at each of the 100+ call sites is a more sustainable way. > >>>>>>>>>> > >>>>>>>>>> Users are free to add layers outside the JVM as well, such as > >>>>>> systemd, > >>>>>>>>>> container, > >>>>>>>>>> SELinux etc. But be honest - how many small/medium organizations > >>>>>> really do > >>>>>>>>>> this? > >>>>>>>>>> > >>>>>>>>>>>> - *Dynamic ZK-watcher-based network policy* — correct but > >>>>>>>>>>>> significantly more complex; adds ZK client dependency to the > >> agent > >>>>>>>>>> JAR. > >>>>>>>>>>>> Superseded by port-based wildcards for intra-cluster traffic. > >>>>>>>>>>>> - *Building a Java agent from scratch* — higher effort with no > >>>>>>>>>>>> functional advantage over adapting the Apache 2.0-licensed > >>>>>> OpenSearch > >>>>>>>>>>>> implementation. > >>>>>>>>>>>> > >>>>>>>>>>>> I agree with not burdening this project with building & > >>>>>> maintainining > >>>>>>>>>> such > >>>>>>>>>>> a mechanism. > >>>>>>>>>>> > >>>>>>>>>>> I'm not sure, to what degree, we can leverage that existing > >> agent > >>>>>> you > >>>>>>>>>> speak > >>>>>>>>>>> of without further burdening us. It's a burden/reward > >> trade-off. > >>>>>>>>>> > >>>>>>>>>> If the agent becomes a true maintenance burden (i.e. a larger > >> burden > >>>>>> than > >>>>>>>>>> handling the CVEs it would prevent, then a valid action is to > >> remove > >>>>>> the > >>>>>>>>>> agent again. > >>>>>>>>>> Like any other feature we develop and maintain. Good this is > >> that > >>>>>> this is > >>>>>>>>>> pure Java, > >>>>>>>>>> and production-ready stable code. > >>>>>>>>>> > >>>>>>>>>>> In your updated proposal, you point > >>>>>>>>>>> out org.apache.solr.core.CoreContainer#assertPathAllowed That > >> is > >>>>>> called > >>>>>>>>>> by > >>>>>>>>>>> a number of places... although I *think* the original > >> intention was > >>>>>> only > >>>>>>>>>> to > >>>>>>>>>>> limit where cores are created? Can you elaborate on what the > >> role > >>>>>> of > >>>>>>>>>> that > >>>>>>>>>>> method *should* be and how the JSM might also or work? > >>>>>>>>>> > >>>>>>>>>> The pathAllowed checks (also the ones pre-dating the > >> centralization > >>>>>> in > >>>>>>>>>> CoreContainer, > >>>>>>>>>> were introduced in 8.x, before JSM was introduced or enabled by > >>>>>> default. > >>>>>>>>>> Initially > >>>>>>>>>> assertPathAllowed would validate end-user API input to avoid > >> cores > >>>>>> being > >>>>>>>>>> created > >>>>>>>>>> or loaded outside blessed folders. It has then been re-used for > >>>>>> other code > >>>>>>>>>> locations > >>>>>>>>>> that may do file access based on user-input in API or config. > >> It as > >>>>>> also > >>>>>>>>>> extended to > >>>>>>>>>> block UNC paths after many attacks with this as a vector. > >>>>>>>>>> > >>>>>>>>>> I believe perhaps pathAllowed would not have been written had > >> JSM > >>>>>> already > >>>>>>>>>> been > >>>>>>>>>> enforced. Although I don't know if you can block UNC with JSM? > >>>>>>>>>> The SIP does not propose to remove pathAllowed now. One benefit > >> of > >>>>>> early > >>>>>>>>>> detection > >>>>>>>>>> is that we can give better context-sensitive error- and log > >> messages > >>>>>>>>>> rather than throwing > >>>>>>>>>> an exception. > >>>>>>>>>> > >>>>>>>>>> The agent approach laid out sandboxes four attack vectors > >>>>>>>>>> * File access outside a limited set of folders and full block of > >>>>>> Windows > >>>>>>>>>> UNC > >>>>>>>>>> * Network access other than peer solr nodes and zk nodes > >>>>>>>>>> * Disabllow calling System.exit() (mainly useful for 3rd party > >>>>>> plugins?) > >>>>>>>>>> * Disallow spawning processes > >>>>>>>>>> > >>>>>>>>>> A nice thing with the agent approach is that it is unintrusive > >> and > >>>>>> easy to > >>>>>>>>>> disable, > >>>>>>>>>> so users who want to take care of all this on OS level may > >> disable > >>>>>> it, and > >>>>>>>>>> if we or > >>>>>>>>>> users don't find value in it down the road they can disable it > >> or we > >>>>>> can > >>>>>>>>>> remove it. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Jan > >>>>>>>>>> > >> --------------------------------------------------------------------- > >>>>>>>>>> To unsubscribe, e-mail: [email protected] > >>>>>>>>>> For additional commands, e-mail: [email protected] > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> http://www.needhamsoftware.com (work) > >>>>>>>>> https://a.co/d/b2sZLD9 (my fantasy fiction book) > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >> --------------------------------------------------------------------- > >>>>>> To unsubscribe, e-mail: [email protected] > >>>>>> For additional commands, e-mail: [email protected] > >>>>>> > >>>>>> > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: [email protected] > >>>> For additional commands, e-mail: [email protected] > >>>> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
