I believe so. The encoding of a string in Flatbuffers is [byte] with a null terminator not included in the length, so old files should still be readable (they would simply not see the terminator anymore). And conversely, continuing to write the null terminator means new files should still be readable by existing implementations, so long as they don't have binary metadata (but we could increment the metadata version before allowing that). There still remains the issue of non-UTF8 data inside the "string" value; on that point, this is a non-answer (or I guess, it canonicalizes the current unintentional behavior).
It would complicate the Java API as that currently works with Strings, but that will be an issue for any attempt to add binary metadata. -David On Mon, Aug 23, 2021, at 13:40, Antoine Pitrou wrote: > > Le 23/08/2021 à 17:52, David Li a écrit : > > Another way forward might be to relax the value type to [byte], but also > > require implementations to null-terminate binary values regardless. The C++ > > Flatbuffers implementation does this already [1] (though not the Java one > > [2]). Old implementations validating UTF8-ness would still be unable to > > read anything with binary metadata (which is the case they're already in), > > but implementations just validating for null-termination would continue to > > work. And again, I believe this is in-spec for Flatbuffers since string > > lengths don't include the null terminator. > > > > [1]: > > https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/include/flatbuffers/flatbuffers.h#L1612-L1670 > > [2]: > > https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/java/com/google/flatbuffers/FlatBufferBuilder.java#L594-L637 > > Though it seems we could emulate C++ here by calling addByte(0) ourselves > > before createByteVector. > > Would that allow for both forwards and backwards compatibility when the > metadata is valid UTF8? > > Regards > > Antoine. > >
