Hi all, As discussed at the last Vector sync, I wrote some code and ran benchmarks to measure how well Parquet handles embeddings stored as FLOAT and as FLOAT16.
*ALP*: Parquet Java doesn't support ALP yet, so I prototyped it. It doesn't compress these FLOAT16 embeddings well: - Plain ALP needs 36.5 bits/value even with zstd, which is more than raw float32. - ALP_rd + zstd needs 17.9 bits/value, 18% more than dictionary encoding (15.1). - Dictionary encoding needs 15.1 bits/value even without zstd: the embeddings have only ~24k distinct values, so every value becomes a 15-bit id. *I have ruled out using ALP.* *Read/write/heap*: Java has no native float16 type yet ( https://openjdk.org/jeps/508), so I expect Java users to store float16 values in shorts for now. The benchmark therefore uses float[] and short[] as the in-memory embeddings. Optimizations I made for both FLOAT and FLOAT16: - FLOAT16 values are decoded straight from the page, without allocating a Binary per value - Bulk reads of float/float16 lists, with faster PLAIN, BYTE_STREAM_SPLIT and dictionary decoding - Fixed-size lists: repetition/definition levels are derived instead of decoded - About 20% faster zstd page decompression - Column/size statistics turned off for columns nested in lists/maps, which Iceberg doesn't use anyway Results on my MacBook Pro (M4 Pro, JDK 21), with 100,000 embeddings of 2,560 dimensions in local files: - *Smallest*: FLOAT16 + BSS + zstd is 10% smaller than FLOAT + DICT + zstd (415 vs 459 MiB), and also reads 22% faster (284 vs 362 ms). - *Fastest reads*: FLOAT16 + PLAIN + uncompressed reads 26% faster than FLOAT + PLAIN + uncompressed (134 vs 180 ms), at half the file size. On blob storage the extra transfer would increase the gap even more. - *Overall*: the best FLOAT setup is DICT + uncompressed, and the best FLOAT16 setup is PLAIN + uncompressed. They have similar sizes (459 vs 489 MiB), but FLOAT16 reads 57% faster (134 vs 312 ms), writes 55% faster, and uses half the heap. *I clearly see a lot to gain from Parquet's FLOAT16 type.* *ALP numbers*: storage encoding bits/value bits/value + zstd details *FLOAT (32-bit)* *DICTIONARY* *15.16* *15.13* *23,908 distinct values, 15-bit ids, 93 KiB dictionary* FLOAT (32-bit) ALP 39.77 36.47 e=10, f=0 in every vector, avg 31.0-bit integers, 18.06% exceptions FLOAT (32-bit) ALP_rd 27.52 17.89 8 left + 24 right bits, 2.09% exceptions FLOAT16 (16-bit) DICTIONARY 15.09 15.09 23,908 distinct values, 15-bit ids, 47 KiB dictionary FLOAT16 (16-bit) ALP_rd (not part of ALP) 14.46 14.46 5 left + 11 right bits, 2.09% exceptions *Embedding benchmark numbers*: source / storage encoding codec file MiB write ms read ms heap MiB float[] / FLOAT PLAIN uncompressed 977.1 1,707 ±48 180 ±5 979 *float[] / FLOAT* *DICT* *uncompressed* *459.2* *4,119 ±12* *312 ±2* *980* float[] / FLOAT PLAIN zstd 579.2 5,730 ±30 950 ±3 980 float[] / FLOAT DICT zstd 459.1 4,304 ±16 362 ±2 979 *short[] / FLOAT16* *PLAIN* *uncompressed* *488.8* *1,871 ±8* *134 ±27* *491* short[] / FLOAT16 BSS uncompressed 488.8 2,813 ±17 156 ±16 491 short[] / FLOAT16 PLAIN zstd 449.3 2,214 ±8 444 ±15 490 short[] / FLOAT16 BSS zstd 415.0 3,003 ±33 284 ±17 491 Code: https://github.com/pvary/iceberg/commits/parquet_float16_benchmark/ To run the benchmarks: *./gradlew :iceberg-data:jmh * *-PjmhIncludeRegex=ParquetFloat16EmbeddingBenchmark* *./gradlew :iceberg-data:jmh -PjmhIncludeRegex=AlpPrototype* The first command runs all 24 combinations, which takes about 30 minutes. The ALP prototype takes under a minute. Thanks, Peter Russell Spitzer <[email protected]> ezt írta (időpont: 2026. okt. 21., Sze, 17:38): > I think all these expansions of types understood by Iceberg belong in a > separate discussion. I think vector basically can exist > completely independently of introducing more subtypes for Vectors. I think > there is probably a lot more discussion needed regarding which specific > variants for sub-width types we want to support and which engines are > actually capable of supporting them. I would get Vector in first, then > expand the spec later with additional types. > > My ideal path forward would be > > 1. Vote in current vector type (regardless of what is happening in > Parquet since I don't think that really changes what we want to allow here, > just the complexity of the implementation) > 2. Start a new discussion about other vector subtypes. > > On Wed, Oct 21, 2026 at 6:43 AM Péter Váry <[email protected]> > wrote: > >> Hi Kurtis, >> Are the bf16/bf8 widely used? >> I know that int8, and float16 are used by our users, and we already have >> Parquet logical, and physical types defined for them. >> >> There is none for bf16/bf8, so we should first make them available there >> if we want to use them. >> >> Thanks, >> Peter >> >> <[email protected]> ezt írta (időpont: 2026. okt. 21., Sze, 1:03): >> >>> If we are discussing int8 and float16 for AI support shouldn’t we also >>> consider brain float (bf16/bf8) as it’s often used in AI (training and >>> inference) as well. >>> >>> On Oct 20, 2026, at 04:25, Péter Váry <[email protected]> >>> wrote: >>> >>> >>> int8 and float16 are needed for AI Vectors. There were suggestions to >>> support them only in Vector types, but I think it would be a bad idea. >>> Let's discuss it in the upcoming Vector type sync, and we could start >>> its own DISCUSS thread if/once we agree on how to proceed. >>> >>> Daniel Weeks <[email protected]> ezt írta (időpont: 2026. okt. 19., H, >>> 17:27): >>> >>>> Peter, >>>> >>>> I think we need to have a broader discussion around adding narrower >>>> types. This probably deserves its own DISCUSS thread (not buried in a sync >>>> follow up). >>>> >>>> -Dan >>>> >>>> On Thu, Oct 15, 2026 at 2:01 PM Péter Váry <[email protected]> >>>> wrote: >>>> >>>>> Hi Philipp, >>>>> >>>>> We’re working on vector indexes on top of the upcoming index >>>>> framework, primarily for AI workloads. Supporting int8 and float16 vectors >>>>> is a must for these use cases. >>>>> >>>>> The main motivation is vectors, but I’m not convinced these should be >>>>> vector-specific types. Since int8 and float16 are well-defined scalar >>>>> types >>>>> that Iceberg would need to support as vector elements anyway, I think it’s >>>>> cleaner to make them regular primitive types rather than special-case them >>>>> as only valid inside vectors. >>>>> >>>>> This is also the feedback I heard during the community sync and my >>>>> discussions at ApacheCon. >>>>> >>>>> Thanks, >>>>> Peter >>>>> >>>>> >>>>> Philipp Fischbeck via dev <[email protected]> ezt írta (időpont: >>>>> 2026. okt. 15., Cs, 17:54): >>>>> >>>>>> Hi Peter, >>>>>> >>>>>> I see that in the current draft, you suggest adding int8 and float16 >>>>>> as primitive Iceberg types overall, not just vector element types. I >>>>>> don't >>>>>> think this is the right approach because I don't see how engines would >>>>>> use >>>>>> these types outside of vectors. Do you have use cases in mind here? >>>>>> >>>>>> One alternative (as discussed in the design doc and during calls) >>>>>> would be to allow such types only as vector element types. We would still >>>>>> have to define the Parquet/Avro type mapping, but the type definition >>>>>> would >>>>>> be limited to the encoding itself, not "spilling" to other parts like >>>>>> bounds or schema evolution. >>>>>> >>>>>> Best, >>>>>> Philipp >>>>>> >>>>>> On Wed, Oct 14, 2026 at 11:37 PM Péter Váry < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Talked with a bunch of people on ApacheCon, and I’d still very much >>>>>>> like to see int8 and float16 vector support in Iceberg. >>>>>>> >>>>>>> I’ve created a draft PR to add these types: >>>>>>> https://github.com/apache/iceberg/pull/18485. >>>>>>> Parquet already provides a logical type for int8 and a physical type >>>>>>> for float16, so adding support in Iceberg seems like a reasonable >>>>>>> effort. >>>>>>> >>>>>>> I’d be happy to spearhead this work so we can get these types into >>>>>>> V4, if the community agrees. >>>>>>> >>>>>>> Thanks, >>>>>>> Peter >>>>>>> >>>>>>> >>>>>>> Yan Yan <[email protected]> ezt írta (időpont: 2026. okt. 9., P, >>>>>>> 2:33): >>>>>>> >>>>>>>> Thank you everyone for the discussion today! The meeting notes can >>>>>>>> be found here: >>>>>>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.v399sh5bz8he >>>>>>>> and below is a TLDR version: >>>>>>>> - Focus on a 1D numeric vector type with fixed length and non-null, >>>>>>>> finite coordinates; +0 and -0 remain valid. >>>>>>>> - Keep the initial spec simple and build on Parquet’s emerging >>>>>>>> vector representation. >>>>>>>> - Initially support int, long, float, and double; defer >>>>>>>> narrower/special types such as int8 as additional Iceberg primitive >>>>>>>> types >>>>>>>> (while allowing such parquet representation for vector). >>>>>>>> - Keep higher level properties such as distance functions out of >>>>>>>> scope, and revisit them with the index design. Sparse vectors are also >>>>>>>> out >>>>>>>> of scope for now: alternative options exist and waiting for concrete >>>>>>>> use >>>>>>>> cases. >>>>>>>> - For formats without native vector support, use a list >>>>>>>> representation with writer enforcement. Will explore an Avro vector >>>>>>>> logical >>>>>>>> type with Avro community; ORC support remains open. >>>>>>>> >>>>>>>> I'll update the spec PR in the next few days (also appreciate the >>>>>>>> quick feedback from everyone!) >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Yan >>>>>>>> >>>>>>>> On Wed, Oct 7, 2026 at 5:31 PM Yan Yan <[email protected]> wrote: >>>>>>>> >>>>>>>>> Thank you Steven for helping secure the time slot for the vector >>>>>>>>> type sync! And thanks everyone for your feedback on the design doc. >>>>>>>>> >>>>>>>>> In preparation for tomorrow's discussion, I've created a draft PR >>>>>>>>> https://github.com/apache/iceberg/pull/18416 for the spec changes >>>>>>>>> based on the progress in the parquet community on the Vector type >>>>>>>>> proposal [ >>>>>>>>> 1 <https://github.com/apache/parquet-format/pull/624>] [2 >>>>>>>>> <https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0>] >>>>>>>>> , and will continue to iterate on both the spec PR and the design doc >>>>>>>>> based >>>>>>>>> on the discussion feedback. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Yan >>>>>>>>> >>>>>>>>> On Tue, Sep 29, 2026 at 10:10 AM Steven Wu <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> As some of you are aware, there is an ongoing discussion about a >>>>>>>>>> new *Vector* type in both the Parquet and Iceberg communities. >>>>>>>>>> >>>>>>>>>> Iceberg Vector type design doc (from Yan Yan): >>>>>>>>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.0 >>>>>>>>>> >>>>>>>>>> To accelerate the Vector type discussion, we are >>>>>>>>>> reusing/extending Thursday's biweekly series to cover both File and >>>>>>>>>> Vector >>>>>>>>>> types (*starting on Oct 8*). Since the file type discussion is >>>>>>>>>> reaching consensus, more time will likely be spent on the Vector >>>>>>>>>> type discussion. The corresponding dev calendar event title has been >>>>>>>>>> updated. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Steven >>>>>>>>>> >>>>>>>>>
