There are two issues here: the only supported DV format today is all-memory, with no (supported) fallback. This we can resolve by making a compromise and changing the default to be disk or semi-disk format.
The second issue is what do we do with the other formats. Do they remain experimental forever? Or over time, we will migrate the good and stable ones, those that eg improve things but require resources so they cannot make it to default, out of the experimental state? I hope that we will have more non experimental Codecs in the future. Well documented and well tested (to me that should also apply to experimental ones!), so that users can choose between them based on their needs. Much like we have highlighters, spell checkers etc. (I'm aware that these are also experimental, but I don't think we intend to keep them like that forever). Maybe we don't need to decide anything now but act on a per case basis. IMO where we can, we should strive to support back-compat, in the form of copy-on-write. And do that for the "serious" Codecs only. Shai On Jul 19, 2013 7:23 PM, "Robert Muir" <[email protected]> wrote: > This causes a lot of copy-on-write/baggage for things in heavy flux. > > I don't understand why we need to go overboard when a user can migrate > their index to the official format with addIndexes... > > if someone wants to make a command-line tool around addIndexes(Reader) to > assist in that, I'm all for that. > > On Fri, Jul 19, 2013 at 12:19 PM, Michael McCandless < > [email protected]> wrote: > >> Maybe, for the experimental formats with no back compat, we could at >> least rename them when there is a breaking change? >> >> This way, assuming there were no changes to the codec APIs, an app >> could upgrade to Lucene 4.4 while continuing to use e.g. >> Disk42DocValuesFormat, without having CLASSPATH horrors. >> >> Ie they could decouple the decision of upgrading the index from >> upgrading the JARs. >> >> When they are ready to upgrade, they could then switch to >> Disk44DocValuesFormat. >> >> Mike McCandless >> >> http://blog.mikemccandless.com >> >> >> On Fri, Jul 19, 2013 at 11:27 AM, Robert Muir <[email protected]> wrote: >> > >> > >> > On Fri, Jul 19, 2013 at 11:18 AM, Shai Erera <[email protected]> wrote: >> > >> >> I feel that we should offer both... somehow. If we make the default >> Codec >> >> conservative, and all the "good stuff" experimental, people might >> prefer to >> >> use a conservative Lucene (just because the other path means being >> >> adventurous or very expert), and perhaps then we'll get conservative >> results >> >> compared to others... >> >> >> >> We should at least try not to break back compat when it's not >> necessary. >> >> Eg if the format changes such that we can still keep the old format >> for old >> >> indexes (read-only) with a different name, then we should consider it. >> I >> >> know it means we'll need to maintain two Codecs wrt API changes, but I >> hope >> >> that's not common. >> >> >> >> >> > >> > I don't agree with this, sorry. a lot of that experimental stuff is >> > experimental because its still in flux. >> > >> > if we are going to provide file back compat for something, this means: >> > * thoroughly documenting the file formats involved >> > * adding and maintaining the appropriate indexes to >> > TestBackwardsCompatibility >> > * doing copy-on-write whenever we want to make a change to the codec and >> > dragging lots of baggage through 2 major release cycles. >> > * makes it harder to do major refactoring or fixing (take a look at >> some of >> > the special cases and backwards stuff to make Lucene3xCodec work!!!!) >> > >> > Its nice to have alternative codecs for people to play around with in >> the >> > codecs/ package, but that doesnt need to equate to additional back >> compat >> > hell. Its a serious committment and I think we should just support our >> > official file format this way. >> > >> > But this is a totally separate discussion from what our defaults should >> be. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> >
