Hi Philipp, We’re working on vector indexes on top of the upcoming index framework, primarily for AI workloads. Supporting int8 and float16 vectors is a must for these use cases.
The main motivation is vectors, but I’m not convinced these should be vector-specific types. Since int8 and float16 are well-defined scalar types that Iceberg would need to support as vector elements anyway, I think it’s cleaner to make them regular primitive types rather than special-case them as only valid inside vectors. This is also the feedback I heard during the community sync and my discussions at ApacheCon. Thanks, Peter Philipp Fischbeck via dev <[email protected]> ezt írta (időpont: 2026. okt. 15., Cs, 17:54): > Hi Peter, > > I see that in the current draft, you suggest adding int8 and float16 as > primitive Iceberg types overall, not just vector element types. I don't > think this is the right approach because I don't see how engines would use > these types outside of vectors. Do you have use cases in mind here? > > One alternative (as discussed in the design doc and during calls) would be > to allow such types only as vector element types. We would still have to > define the Parquet/Avro type mapping, but the type definition would be > limited to the encoding itself, not "spilling" to other parts like bounds > or schema evolution. > > Best, > Philipp > > On Wed, Oct 14, 2026 at 11:37 PM Péter Váry <[email protected]> > wrote: > >> Talked with a bunch of people on ApacheCon, and I’d still very much like >> to see int8 and float16 vector support in Iceberg. >> >> I’ve created a draft PR to add these types: >> https://github.com/apache/iceberg/pull/18485. >> Parquet already provides a logical type for int8 and a physical type for >> float16, so adding support in Iceberg seems like a reasonable effort. >> >> I’d be happy to spearhead this work so we can get these types into V4, if >> the community agrees. >> >> Thanks, >> Peter >> >> >> Yan Yan <[email protected]> ezt írta (időpont: 2026. okt. 9., P, 2:33): >> >>> Thank you everyone for the discussion today! The meeting notes can be >>> found here: >>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.v399sh5bz8he >>> and below is a TLDR version: >>> - Focus on a 1D numeric vector type with fixed length and non-null, >>> finite coordinates; +0 and -0 remain valid. >>> - Keep the initial spec simple and build on Parquet’s emerging vector >>> representation. >>> - Initially support int, long, float, and double; defer narrower/special >>> types such as int8 as additional Iceberg primitive types (while allowing >>> such parquet representation for vector). >>> - Keep higher level properties such as distance functions out of scope, >>> and revisit them with the index design. Sparse vectors are also out of >>> scope for now: alternative options exist and waiting for concrete use cases. >>> - For formats without native vector support, use a list representation >>> with writer enforcement. Will explore an Avro vector logical type with Avro >>> community; ORC support remains open. >>> >>> I'll update the spec PR in the next few days (also appreciate the quick >>> feedback from everyone!) >>> >>> Thanks, >>> Yan >>> >>> On Wed, Oct 7, 2026 at 5:31 PM Yan Yan <[email protected]> wrote: >>> >>>> Thank you Steven for helping secure the time slot for the vector type >>>> sync! And thanks everyone for your feedback on the design doc. >>>> >>>> In preparation for tomorrow's discussion, I've created a draft PR >>>> https://github.com/apache/iceberg/pull/18416 for the spec changes >>>> based on the progress in the parquet community on the Vector type proposal >>>> [ >>>> 1 <https://github.com/apache/parquet-format/pull/624>] [2 >>>> <https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0>] >>>> , and will continue to iterate on both the spec PR and the design doc based >>>> on the discussion feedback. >>>> >>>> Thanks, >>>> Yan >>>> >>>> On Tue, Sep 29, 2026 at 10:10 AM Steven Wu <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> As some of you are aware, there is an ongoing discussion about a new >>>>> *Vector* type in both the Parquet and Iceberg communities. >>>>> >>>>> Iceberg Vector type design doc (from Yan Yan): >>>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.0 >>>>> >>>>> To accelerate the Vector type discussion, we are reusing/extending >>>>> Thursday's biweekly series to cover both File and Vector types (*starting >>>>> on Oct 8*). Since the file type discussion is reaching consensus, more >>>>> time will likely be spent on the Vector type discussion. The corresponding >>>>> dev calendar event title has been updated. >>>>> >>>>> Thanks, >>>>> Steven >>>>> >>>>
