I would reiterate that in my view sketches should not care about validation. If the user desires validation, he can instantiate, say, frequent_items_sketch<utf8_string> instead of frequent_items_sketch<std::string>. utf8_string should perform validation.
On Fri, Mar 6, 2026 at 10:17 PM Hyeonho Kim <[email protected]> wrote: > Hi all, > > I realized there is one more design point that may need discussion. > > For sketches that validate UTF-8 at update() time by default, with an > explicit opt-out, that setting affects the behavior of future update() > calls even after deserialization. > > So there seems to be a broader design choice here for string-specific > sketches / update APIs: > > 1. > > Treat the UTF-8 validation setting as part of the serialized sketch > state, so it is preserved across serialization/deserialization. > 2. > > Treat it as a runtime policy only, in which case it would need to be > specified again after deserialization (or when constructing a new sketch). > > The first option would preserve behavioral consistency, so it seems like > the more semantically consistent choice. However, it also seems like a much > bigger decision in practice, since it would require a serialization format > change / versioning. > > The second option avoids changing the serialized format, but a > deserialized sketch may not behave exactly the same for future update() > calls unless the caller explicitly restores the same policy. > > What do others think? > > On Wed, Mar 4, 2026 at 5:30 AM Lee Rhodes <[email protected]> wrote: > >> I agree. Here is a proposed wording that is a sort of a "policy" way to >> think about this: >> >> For "container" type sketches that can potentially retain Strings: >> >> - If a sketch has the word "string" as part of its name, then UTF-8 >> validation at update() should be the default with an explicit >> opt-out. Example: ArrayOfStringsTupleSketch. >> - If an update method to a sketch has an explicit "string" parameter, >> then UTF-8 validation should be the default with an explicit opt-out. >> Example FdtSketch::update(String[]). >> - Otherwise, if a sketch or update method accepts just a generic type >> T, then we will provide a UTF-8 validating "SerDe" object that can be >> optionally used for type T. >> >> >> >> On Tue, Mar 3, 2026 at 7:32 AM Hyeonho Kim <[email protected]> wrote: >> >>> Hi all! >>> >>> Unless there are objections, I propose the following: >>> >>> 1. >>> >>> Introduce an opt-in UTF-8 validating SerDe for std::string >>> (validation OFF by default). >>> 2. >>> >>> For AoS string items, enable UTF-8 validation at update() by >>> default, with an explicit opt-out. >>> >>> If this direction looks reasonable, I will proceed accordingly in the >>> AoS PR and follow up with a separate PR for the SerDe option. >>> >>> >>> Thanks, >>> >>> Hyeonho >>> >>> On Fri, Feb 20, 2026 at 11:59 PM Hyeonho Kim <[email protected]> wrote: >>> >>>> Thanks all for the feedback. >>>> >>>> >>>> We can preserve backward compatibility for existing C++ users while >>>> also providing a clear path for cross-language portability. >>>> >>>> How do you think about the following approach? >>>> >>>> - SerDe with string: Add an option to validate whether the string >>>> contains valid UTF-8 sequences. The default would be validation OFF to >>>> preserve existing compatibility. >>>> >>>> - AoS tuple sketch: Validate UTF-8 at the update method (fail-fast). >>>> Enabling validation by default, with an explicit opt-out for users who >>>> want. >>>> >>>> >>>> For DS-Go, we can follow the same policy as C++. >>>> >>>> >>>> Feedback is welcome. >>>> >>>> On Wed, Feb 18, 2026 at 3:24 AM Jon Malkin <[email protected]> >>>> wrote: >>>> >>>>> Gonna agree with Alexander here. I think we should provide a serde >>>>> option for c++, but that we should not reject non-UTF-8 strings. >>>>> >>>>> That wouldn’t just be an API-breaking change. It would break >>>>> compatibility of c++ with itself for anyone who doesn’t need language >>>>> portability. >>>>> >>>>> A separate utf8_serde option gets my vote. >>>>> >>>>> jon >>>>> >>>>> On Tue, Feb 17, 2026 at 10:12 AM Alexander Saydakov via dev < >>>>> [email protected]> wrote: >>>>> >>>>>> Regarding C++, I would think that the easiest approach is to instruct >>>>>> the user to use a UTF8-validating string substitute instead of >>>>>> std::string. >>>>>> I am not sure whether we should provide such a thing or let the user >>>>>> to come up with their own implementation. >>>>>> Consider having a uft8_string that would validate the input in the >>>>>> constrtuctor but otherwise identical to std::string >>>>>> So the user can instantiate, for example, >>>>>> frequent_items_sketch<utf8_string> instead of >>>>>> frequent_items_sketch<std::string> if validation is necessary. >>>>>> >>>>>> >>>>>> On Sun, Feb 15, 2026 at 8:38 PM Hyeonho Kim <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Thanks for the feedback. I agree that for container sketches that >>>>>>> retain and serialize strings, we should validate that string payloads >>>>>>> are >>>>>>> valid UTF-8 sequences to preserve cross-language portability. >>>>>>> >>>>>>> On *where* to validate in DS-CPP: validating at update() (ingest >>>>>>> time) is attractive because it is fail-fast, but it also adds additional >>>>>>> cost on the hot path. If the community is comfortable with that overhead >>>>>>> for string-based container sketches, I’m happy to pursue the >>>>>>> update()-time validation approach. >>>>>>> >>>>>>> If performance sensitivity is a concern, an alternative would be to >>>>>>> always validate at (de)serialization boundaries (to guarantee artifact >>>>>>> correctness), and optionally provide a “fail-fast” mode that enables >>>>>>> validation at update() as well. >>>>>>> >>>>>>> For DS-Go, we can follow the same policy. Go’s situation is a bit >>>>>>> simpler in implementation because it provides UTF-8 validation in the >>>>>>> standard library (unicode/utf8), so we wouldn’t need an external >>>>>>> dependency for the validator. >>>>>>> >>>>>>> On Mon, Feb 16, 2026 at 6:29 AM Lee Rhodes <[email protected]> wrote: >>>>>>> >>>>>>>> This issue, raised by Hyeonho Kim, relates to sketches that allow a >>>>>>>> user to update the sketch with a string and the sketch also retains >>>>>>>> within >>>>>>>> the sketch a sample of the input strings seen. When serialized, there >>>>>>>> is an >>>>>>>> implicit assumption that another user, possibly in a different >>>>>>>> language, >>>>>>>> can successfully deserialize those sketch images. These sketches >>>>>>>> include KLL, >>>>>>>> REQ, Classic Quantiles, Sampling, FrequentItems, and Tuple. We >>>>>>>> informally call these "container" sketches, because they contain actual >>>>>>>> samples from the input stream. HLL, Theta, CPC, BloomFilter, etc., >>>>>>>> are not >>>>>>>> container sketches. >>>>>>>> >>>>>>>> In the DS-Java library, all container sketches that allow strings >>>>>>>> always use UTF_8. So the sketch images produced will contain proper >>>>>>>> UTF_8 >>>>>>>> sequences. >>>>>>>> >>>>>>>> In the DS-CPP library, all the various data types are abstracted >>>>>>>> via templates. The serialization operation is declared similar to >>>>>>>> >>>>>>>> >>>>>>>> *sketch<T>::serialize(std::ostream& os, const SerDe& sd)where T *is >>>>>>>> the item type*, os is the output stream and sd* *is the SerDe that >>>>>>>> performs the conversion to bytes. * >>>>>>>> >>>>>>>> >>>>>>>> If the user wants to use an item of type string, *T* would >>>>>>>> typically be of type *std::string*, which is just a blob of bytes >>>>>>>> and no requirement that it is UTF_8. >>>>>>>> >>>>>>>> >>>>>>>> So far, we have trusted users of the library to know that if they >>>>>>>> update one of these container classes with a type *T,* that the >>>>>>>> downstream user can successfully decode it. But this could be >>>>>>>> catastrophic: A downstream user of a sketch image could be separated >>>>>>>> from >>>>>>>> the creation of the sketch image by years and be using a different >>>>>>>> language. >>>>>>>> >>>>>>>> One of the big advantages of our DataSketches project is that our >>>>>>>> serialization images should be language and platform independent, >>>>>>>> allowing >>>>>>>> cross-language and cross platform interchange of sketches. >>>>>>>> >>>>>>>> Hyeonho Kim's recommendation makes sense: For serialized sketch >>>>>>>> images that contain strings, those strings must be UTF_8. >>>>>>>> >>>>>>>> So how do we implement that? My thoughts are as follows: >>>>>>>> >>>>>>>> 1. We should document now in the website and in appropriate >>>>>>>> places in the library the potential danger of not using UTF_8 >>>>>>>> strings. (At >>>>>>>> least until we have a more robust solution) >>>>>>>> 2. I think implementing validation checks on UTF_8 strings at >>>>>>>> the SerDe boundaries may be too late. A user could have processed >>>>>>>> a large >>>>>>>> stream of data only to discover a failure at serialization time, >>>>>>>> which >>>>>>>> could be much later in time. The other possibility would be to >>>>>>>> validate >>>>>>>> the strings at the input into the sketch, typically in the *update() >>>>>>>> *method. >>>>>>>> 3. For C++, there are 3rd party libraries that specialize in >>>>>>>> UTF_8 validation, including ICU >>>>>>>> >>>>>>>> <https://urldefense.com/v3/__https://github.com/unicode-org/icu__;!!Op6eflyXZCqGR5I!Hr1GVWHWpCX58DUhmQXYJ9srUYP2YzNW09vCpXOXZ8v4t3inaSAg9EewqhWEuJKCGoolYxZAnpPC5K7q2A$> >>>>>>>> , UTF8-CPP >>>>>>>> >>>>>>>> <https://urldefense.com/v3/__https://github.com/nemtrif/utfcpp__;!!Op6eflyXZCqGR5I!Hr1GVWHWpCX58DUhmQXYJ9srUYP2YzNW09vCpXOXZ8v4t3inaSAg9EewqhWEuJKCGoolYxZAnpNk0hS7xg$> >>>>>>>> and simjson >>>>>>>> >>>>>>>> <https://urldefense.com/v3/__https://lemire.me/blog/2020/10/20/ridiculously-fast-unicode-utf-8-validation/__;!!Op6eflyXZCqGR5I!Hr1GVWHWpCX58DUhmQXYJ9srUYP2YzNW09vCpXOXZ8v4t3inaSAg9EewqhWEuJKCGoolYxZAnpMVUko1NQ$>. >>>>>>>> (These have standard licensing). From what I've read, UTF-8 >>>>>>>> validation, if >>>>>>>> done correctly, can be done very fast, with only a small section of >>>>>>>> code. >>>>>>>> 4. I am not sure what the solutions are for Rust or Go. >>>>>>>> >>>>>>>> I welcome your feedback. >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Feb 14, 2026 at 1:47 AM tison <[email protected]> wrote: >>>>>>>> >>>>>>>>> This PR [1] of datasketches-rust demonstrates how the Rust impl >>>>>>>>> deserializes String values. >>>>>>>>> >>>>>>>>> [1] https://github.com/apache/datasketches-rust/pull/82 >>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/datasketches-rust/pull/82__;!!Op6eflyXZCqGR5I!Hr1GVWHWpCX58DUhmQXYJ9srUYP2YzNW09vCpXOXZ8v4t3inaSAg9EewqhWEuJKCGoolYxZAnpN3yo3d3w$> >>>>>>>>> >>>>>>>>> If it's std::string::String, then it must be of UTF-8 encoding. >>>>>>>>> And we check the encoding on deserialization. >>>>>>>>> >>>>>>>>> However, the Rust ecosystem also supports "strings" that do not >>>>>>>>> use UTF-8, such as BStr. >>>>>>>>> >>>>>>>>> So, my opinions are: >>>>>>>>> >>>>>>>>> 1. It's good to assume serialized string data to be valid UTF-8. >>>>>>>>> 2. Even if it isn't, for datasketches-rust, users should be able >>>>>>>>> to choose a proper type to deserialize the bytes into a type that >>>>>>>>> doesn't >>>>>>>>> require UTF-8 encoding. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> tison. >>>>>>>>> >>>>>>>>> >>>>>>>>> Hyeonho Kim <[email protected]> 于2026年2月14日周六 17:24写道： >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> While working on UTF-8 validation for the AoS tuple sketch in C++ >>>>>>>>>> (ref: https://github.com/apache/datasketches-cpp/pull/476 >>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/datasketches-cpp/pull/476__;!!Op6eflyXZCqGR5I!Hr1GVWHWpCX58DUhmQXYJ9srUYP2YzNW09vCpXOXZ8v4t3inaSAg9EewqhWEuJKCGoolYxZAnpPslrtDnQ$>), >>>>>>>>>> a broader design question came up that may affect multiple sketches. >>>>>>>>>> >>>>>>>>>> Based on my current understanding: >>>>>>>>>> >>>>>>>>>> - In datasketches-java, string serialization already produces >>>>>>>>>> valid UTF-8 bytes via getBytes(StandardCharsets.UTF_8). So >>>>>>>>>> Java-generated >>>>>>>>>> artifacts already assume valid UTF-8 string encoding. >>>>>>>>>> - Rust and Python string types represent Unicode text and can be >>>>>>>>>> encoded to UTF-8. Please correct me if I am mistaken. (I don't know >>>>>>>>>> Rust >>>>>>>>>> and Python well) >>>>>>>>>> - In Go, string is a byte sequence and may contain invalid UTF-8 >>>>>>>>>> unless explicitly validated. So during serialization, it may produce >>>>>>>>>> invalid UTF-8 sequences. >>>>>>>>>> - In C++, std::string is also a byte container and does not >>>>>>>>>> enforce UTF-8 validity. So during serialization, it may produce >>>>>>>>>> invalid >>>>>>>>>> UTF-8 sequences. >>>>>>>>>> >>>>>>>>>> If I am mistaken on any of these points, I would appreciate >>>>>>>>>> corrections. >>>>>>>>>> >>>>>>>>>> If we want to maintain cross-language portability for serialized >>>>>>>>>> artifacts, one possible approach would be to ensure that any >>>>>>>>>> serialized >>>>>>>>>> string data is valid UTF-8. This could potentially apply to any >>>>>>>>>> sketches >>>>>>>>>> that serialize or deserialize string data. >>>>>>>>>> >>>>>>>>>> There seem to be several possible approaches: >>>>>>>>>> - Validate UTF-8 at serialization boundaries >>>>>>>>>> - Document that input strings must be valid UTF-8 and rely on >>>>>>>>>> caller discipline >>>>>>>>>> >>>>>>>>>> At this point I am not proposing a specific solution. I would >>>>>>>>>> like to hear opinions from the community on: We want to require >>>>>>>>>> serialized >>>>>>>>>> string data to be valid UTF-8 for cross-language portability >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Hyeonho >>>>>>>>>> >>>>>>>>>
