Peter, I think we need to have a broader discussion around adding narrower types. This probably deserves its own DISCUSS thread (not buried in a sync follow up).
-Dan On Thu, Oct 15, 2026 at 2:01 PM Péter Váry <[email protected]> wrote: > Hi Philipp, > > We’re working on vector indexes on top of the upcoming index framework, > primarily for AI workloads. Supporting int8 and float16 vectors is a must > for these use cases. > > The main motivation is vectors, but I’m not convinced these should be > vector-specific types. Since int8 and float16 are well-defined scalar types > that Iceberg would need to support as vector elements anyway, I think it’s > cleaner to make them regular primitive types rather than special-case them > as only valid inside vectors. > > This is also the feedback I heard during the community sync and my > discussions at ApacheCon. > > Thanks, > Peter > > > Philipp Fischbeck via dev <[email protected]> ezt írta (időpont: > 2026. okt. 15., Cs, 17:54): > >> Hi Peter, >> >> I see that in the current draft, you suggest adding int8 and float16 as >> primitive Iceberg types overall, not just vector element types. I don't >> think this is the right approach because I don't see how engines would use >> these types outside of vectors. Do you have use cases in mind here? >> >> One alternative (as discussed in the design doc and during calls) would >> be to allow such types only as vector element types. We would still have to >> define the Parquet/Avro type mapping, but the type definition would be >> limited to the encoding itself, not "spilling" to other parts like bounds >> or schema evolution. >> >> Best, >> Philipp >> >> On Wed, Oct 14, 2026 at 11:37 PM Péter Váry <[email protected]> >> wrote: >> >>> Talked with a bunch of people on ApacheCon, and I’d still very much like >>> to see int8 and float16 vector support in Iceberg. >>> >>> I’ve created a draft PR to add these types: >>> https://github.com/apache/iceberg/pull/18485. >>> Parquet already provides a logical type for int8 and a physical type for >>> float16, so adding support in Iceberg seems like a reasonable effort. >>> >>> I’d be happy to spearhead this work so we can get these types into V4, >>> if the community agrees. >>> >>> Thanks, >>> Peter >>> >>> >>> Yan Yan <[email protected]> ezt írta (időpont: 2026. okt. 9., P, 2:33): >>> >>>> Thank you everyone for the discussion today! The meeting notes can be >>>> found here: >>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.v399sh5bz8he >>>> and below is a TLDR version: >>>> - Focus on a 1D numeric vector type with fixed length and non-null, >>>> finite coordinates; +0 and -0 remain valid. >>>> - Keep the initial spec simple and build on Parquet’s emerging vector >>>> representation. >>>> - Initially support int, long, float, and double; defer >>>> narrower/special types such as int8 as additional Iceberg primitive types >>>> (while allowing such parquet representation for vector). >>>> - Keep higher level properties such as distance functions out of scope, >>>> and revisit them with the index design. Sparse vectors are also out of >>>> scope for now: alternative options exist and waiting for concrete use >>>> cases. >>>> - For formats without native vector support, use a list representation >>>> with writer enforcement. Will explore an Avro vector logical type with Avro >>>> community; ORC support remains open. >>>> >>>> I'll update the spec PR in the next few days (also appreciate the quick >>>> feedback from everyone!) >>>> >>>> Thanks, >>>> Yan >>>> >>>> On Wed, Oct 7, 2026 at 5:31 PM Yan Yan <[email protected]> wrote: >>>> >>>>> Thank you Steven for helping secure the time slot for the vector type >>>>> sync! And thanks everyone for your feedback on the design doc. >>>>> >>>>> In preparation for tomorrow's discussion, I've created a draft PR >>>>> https://github.com/apache/iceberg/pull/18416 for the spec changes >>>>> based on the progress in the parquet community on the Vector type >>>>> proposal [ >>>>> 1 <https://github.com/apache/parquet-format/pull/624>] [2 >>>>> <https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0>] >>>>> , and will continue to iterate on both the spec PR and the design doc >>>>> based >>>>> on the discussion feedback. >>>>> >>>>> Thanks, >>>>> Yan >>>>> >>>>> On Tue, Sep 29, 2026 at 10:10 AM Steven Wu <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> As some of you are aware, there is an ongoing discussion about a new >>>>>> *Vector* type in both the Parquet and Iceberg communities. >>>>>> >>>>>> Iceberg Vector type design doc (from Yan Yan): >>>>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.0 >>>>>> >>>>>> To accelerate the Vector type discussion, we are reusing/extending >>>>>> Thursday's biweekly series to cover both File and Vector types (*starting >>>>>> on Oct 8*). Since the file type discussion is reaching consensus, more >>>>>> time will likely be spent on the Vector type discussion. The >>>>>> corresponding >>>>>> dev calendar event title has been updated. >>>>>> >>>>>> Thanks, >>>>>> Steven >>>>>> >>>>>
