Any further thoughts on this? I'd like to proceed with the PR for SOLR-16962 [1], which I've updated with a fix to address now-once-again-relevant SOLR-6537 [2] (at whatever point SOLR-16962 made it impossible to configure an external tlog, SOLR-6537 (cleanup/deletion of external tlog) became practically not relevant).
[1] https://issues.apache.org/jira/browse/SOLR-16962 [2] https://issues.apache.org/jira/browse/SOLR-6537 On Mon, Sep 11, 2023 at 12:36 PM Michael Gibney <mich...@michaelgibney.net> wrote: > > > useful to separate the location to use some fast local SSDs vs slower but > > larger storage for the main index > > Exactly. > > And yes, the duplicity of nomenclature is a large part of what I'm > suggesting to deal with here ... it's very confusing. But having > recently been pretty deep in this part of the current code, it looks > very much like "ulogDir" refers to the _parent_ directory of the tlog > dir -- i.e., `[instanceDir]/data/`, which is just weird IMO ... so I > guess I was also thinking of consistency, but hoping to break more in > the direction of using "tlogDir", since "ulogDir" currently > (confusingly) seems to mean `[instanceDir]/data/`. > > My current understanding of the situation: as far as the filesystem is > concerned, updateLog/ulog/solr.ulog.dir/etc. has no significance other > than as the parent of the tlogDir -- and as the parent, it also > happens to be the same dir as the default dataDir -- i.e., the parent > of the `index` dir, etc... so in fact there's nothing at all uniquely > "ulog" about the "ulogDir". > > Michael > > On Mon, Sep 11, 2023 at 12:24 PM David Smiley <david.w.smi...@gmail.com> > wrote: > > > > FWIW I've never heard of anyone customizing that. I suppose it might be > > useful to separate the location to use some fast local SSDs vs slower but > > larger storage for the main index? > > > > I've noticed an annoying duplicity in nomenclature here -- is it the Update > > Log or the Transaction Log? And furthermore, we have a TLOG replica type. > > In our configuration, the nomenclature is "update log" or "ulog". If we > > add a new setting, let's at least be consistent and call the new proposed > > dir "ulogDir". > > > > ~ David > > > > > > On Mon, Sep 11, 2023 at 9:45 AM Michael Gibney <mich...@michaelgibney.net> > > wrote: > > > > > Thanks Ilan, this seems good. In addition to addressing practical > > > concerns, this could also be an opportunity to clarify the semantics > > > around updateLog/tlog configuration: > > > > > > In looking closely at this code, my understanding is that the > > > `<updateLog><str name="dir">${solr.ulog.dir:}</str></updateLog>` > > > setting in solrconfig.xml currently defines the _parent_ directory > > > that contains the tlog dir. Except that afaict, the tlog dir is the > > > _only_ path that's actually relevant. In fact, the portion of the > > > refguide that mentions this is called "Transaction Log Configuration" > > > [1]. > > > > > > This muddles the intent of this setting; it would be much clearer if, > > > rather than a "dir" property (defaulting to `[instanceDir]/data/`) > > > defining the _parent_ directory of the tlog dir, we simply had a > > > "tlogDir" property (defaulting to `[instanceDir]/data/tlog/`) that > > > directly defines the tlog dir. The semantics of this could then be > > > clarified to indicate that: > > > 1. if tlogDir is relative, it is resolved relative to core > > > instanceDir, and must resolve to a descendent dir of instanceDir > > > (i.e., cannot "escape" via ../../../). > > > 2. if tlogDir (whether absolute or relative) resolves to a descendant > > > of instanceDir, it is already implicitly namespaced by core, and the > > > property defines the tlog dir directly. > > > 3. if tlogDir is absolute and resolves to a directory _outside_ of the > > > core instanceDir, it is _not_ implicitly namespaced by core, and the > > > property therefore defines a parent directory, within which a > > > core-namespaced tlog dir will be created (for a final tlog file path > > > of, e.g., `[tlogDir]/[core_name]/tlog.0000000000000001`) > > > > > > As it stands currently, the above three cases are already ambiguous; > > > the proposed bugfix PR [2] necessarily resolves this ambiguity, but > > > even so, the semantics of the "dir" property map poorly onto what > > > actually needs to be configured. > > > > > > Does this make sense, and/or am I missing something? > > > > > > Michael > > > > > > [1] > > > https://solr.apache.org/guide/solr/latest/configuration-guide/commits-transaction-logs.html#transaction-log-configuration > > > [2] https://github.com/apache/solr/pull/1895 > > > > > > On Fri, Sep 8, 2023 at 4:59 PM Ilan Ginzburg <ilans...@gmail.com> wrote: > > > > > > > > Re backcompat issue: might be an option to fix that config but use a > > > > different name for it then, so that any currently existing (and ignored) > > > > config continues to be ignored. > > > > > > > > Ilan > > > > > > > > On Thu, Sep 7, 2023, 6:14 PM Michael Gibney <mich...@michaelgibney.net> > > > > wrote: > > > > > > > > > I'd like to call some extra attention to SOLR-16962 [1]. I'm fairly > > > > > certain that it's been years since it has actually worked to specify > > > > > tlog dir in solrconfig.xml `<updateLog/>` element. I've opened a PR > > > > > that fixes this, but it involves making some decisions about edge-case > > > > > handling, for which I'd appreciate feedback. > > > > > > > > > > Of course, if anyone _has_ successfully configured a custom tlog dir > > > > > location with a recent solr version, I'd be curious to hear about that > > > > > too! > > > > > > > > > > Also worth mentioning: if anyone currently has this latently > > > > > configured in solrconfig.xml, I wonder if it's possible that fixing > > > > > this bug could cause a change in behavior that might cause a > > > > > backcompat issue. > > > > > > > > > > [1] https://issues.apache.org/jira/browse/SOLR-16962 > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > > > > > For additional commands, e-mail: dev-h...@solr.apache.org > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > > > For additional commands, e-mail: dev-h...@solr.apache.org > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org For additional commands, e-mail: dev-h...@solr.apache.org
