int8 and float16 are needed for AI Vectors. There were suggestions to support them only in Vector types, but I think it would be a bad idea. Let's discuss it in the upcoming Vector type sync, and we could start its own DISCUSS thread if/once we agree on how to proceed.
Daniel Weeks <[email protected]> ezt írta (időpont: 2026. okt. 19., H, 17:27): > Peter, > > I think we need to have a broader discussion around adding narrower > types. This probably deserves its own DISCUSS thread (not buried in a sync > follow up). > > -Dan > > On Thu, Oct 15, 2026 at 2:01 PM Péter Váry <[email protected]> > wrote: > >> Hi Philipp, >> >> We’re working on vector indexes on top of the upcoming index framework, >> primarily for AI workloads. Supporting int8 and float16 vectors is a must >> for these use cases. >> >> The main motivation is vectors, but I’m not convinced these should be >> vector-specific types. Since int8 and float16 are well-defined scalar types >> that Iceberg would need to support as vector elements anyway, I think it’s >> cleaner to make them regular primitive types rather than special-case them >> as only valid inside vectors. >> >> This is also the feedback I heard during the community sync and my >> discussions at ApacheCon. >> >> Thanks, >> Peter >> >> >> Philipp Fischbeck via dev <[email protected]> ezt írta (időpont: >> 2026. okt. 15., Cs, 17:54): >> >>> Hi Peter, >>> >>> I see that in the current draft, you suggest adding int8 and float16 as >>> primitive Iceberg types overall, not just vector element types. I don't >>> think this is the right approach because I don't see how engines would use >>> these types outside of vectors. Do you have use cases in mind here? >>> >>> One alternative (as discussed in the design doc and during calls) would >>> be to allow such types only as vector element types. We would still have to >>> define the Parquet/Avro type mapping, but the type definition would be >>> limited to the encoding itself, not "spilling" to other parts like bounds >>> or schema evolution. >>> >>> Best, >>> Philipp >>> >>> On Wed, Oct 14, 2026 at 11:37 PM Péter Váry <[email protected]> >>> wrote: >>> >>>> Talked with a bunch of people on ApacheCon, and I’d still very much >>>> like to see int8 and float16 vector support in Iceberg. >>>> >>>> I’ve created a draft PR to add these types: >>>> https://github.com/apache/iceberg/pull/18485. >>>> Parquet already provides a logical type for int8 and a physical type >>>> for float16, so adding support in Iceberg seems like a reasonable effort. >>>> >>>> I’d be happy to spearhead this work so we can get these types into V4, >>>> if the community agrees. >>>> >>>> Thanks, >>>> Peter >>>> >>>> >>>> Yan Yan <[email protected]> ezt írta (időpont: 2026. okt. 9., P, >>>> 2:33): >>>> >>>>> Thank you everyone for the discussion today! The meeting notes can be >>>>> found here: >>>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.v399sh5bz8he >>>>> and below is a TLDR version: >>>>> - Focus on a 1D numeric vector type with fixed length and non-null, >>>>> finite coordinates; +0 and -0 remain valid. >>>>> - Keep the initial spec simple and build on Parquet’s emerging vector >>>>> representation. >>>>> - Initially support int, long, float, and double; defer >>>>> narrower/special types such as int8 as additional Iceberg primitive types >>>>> (while allowing such parquet representation for vector). >>>>> - Keep higher level properties such as distance functions out of >>>>> scope, and revisit them with the index design. Sparse vectors are also out >>>>> of scope for now: alternative options exist and waiting for concrete use >>>>> cases. >>>>> - For formats without native vector support, use a list representation >>>>> with writer enforcement. Will explore an Avro vector logical type with >>>>> Avro >>>>> community; ORC support remains open. >>>>> >>>>> I'll update the spec PR in the next few days (also appreciate the >>>>> quick feedback from everyone!) >>>>> >>>>> Thanks, >>>>> Yan >>>>> >>>>> On Wed, Oct 7, 2026 at 5:31 PM Yan Yan <[email protected]> wrote: >>>>> >>>>>> Thank you Steven for helping secure the time slot for the vector type >>>>>> sync! And thanks everyone for your feedback on the design doc. >>>>>> >>>>>> In preparation for tomorrow's discussion, I've created a draft PR >>>>>> https://github.com/apache/iceberg/pull/18416 for the spec changes >>>>>> based on the progress in the parquet community on the Vector type >>>>>> proposal [ >>>>>> 1 <https://github.com/apache/parquet-format/pull/624>] [2 >>>>>> <https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0>] >>>>>> , and will continue to iterate on both the spec PR and the design doc >>>>>> based >>>>>> on the discussion feedback. >>>>>> >>>>>> Thanks, >>>>>> Yan >>>>>> >>>>>> On Tue, Sep 29, 2026 at 10:10 AM Steven Wu <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> As some of you are aware, there is an ongoing discussion about a new >>>>>>> *Vector* type in both the Parquet and Iceberg communities. >>>>>>> >>>>>>> Iceberg Vector type design doc (from Yan Yan): >>>>>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.0 >>>>>>> >>>>>>> To accelerate the Vector type discussion, we are reusing/extending >>>>>>> Thursday's biweekly series to cover both File and Vector types >>>>>>> (*starting >>>>>>> on Oct 8*). Since the file type discussion is reaching consensus, more >>>>>>> time will likely be spent on the Vector type discussion. The >>>>>>> corresponding >>>>>>> dev calendar event title has been updated. >>>>>>> >>>>>>> Thanks, >>>>>>> Steven >>>>>>> >>>>>>
