It still sounds like adding a new type might be the safest approach (and marking the old type as discouraged).
On Mon, Aug 23, 2021 at 11:18 AM David Li <lidav...@apache.org> wrote: > I believe so. > > The encoding of a string in Flatbuffers is [byte] with a null terminator > not included in the length, so old files should still be readable (they > would simply not see the terminator anymore). And conversely, continuing to > write the null terminator means new files should still be readable by > existing implementations, so long as they don't have binary metadata (but > we could increment the metadata version before allowing that). There still > remains the issue of non-UTF8 data inside the "string" value; on that > point, this is a non-answer (or I guess, it canonicalizes the current > unintentional behavior). > > It would complicate the Java API as that currently works with Strings, but > that will be an issue for any attempt to add binary metadata. > > -David > > On Mon, Aug 23, 2021, at 13:40, Antoine Pitrou wrote: > > > > Le 23/08/2021 à 17:52, David Li a écrit : > > > Another way forward might be to relax the value type to [byte], but > also require implementations to null-terminate binary values regardless. > The C++ Flatbuffers implementation does this already [1] (though not the > Java one [2]). Old implementations validating UTF8-ness would still be > unable to read anything with binary metadata (which is the case they're > already in), but implementations just validating for null-termination would > continue to work. And again, I believe this is in-spec for Flatbuffers > since string lengths don't include the null terminator. > > > > > > [1]: > https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/include/flatbuffers/flatbuffers.h#L1612-L1670 > > > [2]: > https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/java/com/google/flatbuffers/FlatBufferBuilder.java#L594-L637 > > > Though it seems we could emulate C++ here by calling addByte(0) > ourselves before createByteVector. > > > > Would that allow for both forwards and backwards compatibility when the > > metadata is valid UTF8? > > > > Regards > > > > Antoine. > > > > >
