Hi Peter, I see that in the current draft, you suggest adding int8 and float16 as primitive Iceberg types overall, not just vector element types. I don't think this is the right approach because I don't see how engines would use these types outside of vectors. Do you have use cases in mind here?
One alternative (as discussed in the design doc and during calls) would be to allow such types only as vector element types. We would still have to define the Parquet/Avro type mapping, but the type definition would be limited to the encoding itself, not "spilling" to other parts like bounds or schema evolution. Best, Philipp On Wed, Oct 14, 2026 at 11:37 PM Péter Váry <[email protected]> wrote: > Talked with a bunch of people on ApacheCon, and I’d still very much like > to see int8 and float16 vector support in Iceberg. > > I’ve created a draft PR to add these types: > https://github.com/apache/iceberg/pull/18485. > Parquet already provides a logical type for int8 and a physical type for > float16, so adding support in Iceberg seems like a reasonable effort. > > I’d be happy to spearhead this work so we can get these types into V4, if > the community agrees. > > Thanks, > Peter > > > Yan Yan <[email protected]> ezt írta (időpont: 2026. okt. 9., P, 2:33): > >> Thank you everyone for the discussion today! The meeting notes can be >> found here: >> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.v399sh5bz8he >> and below is a TLDR version: >> - Focus on a 1D numeric vector type with fixed length and non-null, >> finite coordinates; +0 and -0 remain valid. >> - Keep the initial spec simple and build on Parquet’s emerging vector >> representation. >> - Initially support int, long, float, and double; defer narrower/special >> types such as int8 as additional Iceberg primitive types (while allowing >> such parquet representation for vector). >> - Keep higher level properties such as distance functions out of scope, >> and revisit them with the index design. Sparse vectors are also out of >> scope for now: alternative options exist and waiting for concrete use cases. >> - For formats without native vector support, use a list representation >> with writer enforcement. Will explore an Avro vector logical type with Avro >> community; ORC support remains open. >> >> I'll update the spec PR in the next few days (also appreciate the quick >> feedback from everyone!) >> >> Thanks, >> Yan >> >> On Wed, Oct 7, 2026 at 5:31 PM Yan Yan <[email protected]> wrote: >> >>> Thank you Steven for helping secure the time slot for the vector type >>> sync! And thanks everyone for your feedback on the design doc. >>> >>> In preparation for tomorrow's discussion, I've created a draft PR >>> https://github.com/apache/iceberg/pull/18416 for the spec changes based >>> on the progress in the parquet community on the Vector type proposal [1 >>> <https://github.com/apache/parquet-format/pull/624>] [2 >>> <https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0>] >>> , and will continue to iterate on both the spec PR and the design doc based >>> on the discussion feedback. >>> >>> Thanks, >>> Yan >>> >>> On Tue, Sep 29, 2026 at 10:10 AM Steven Wu <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> As some of you are aware, there is an ongoing discussion about a new >>>> *Vector* type in both the Parquet and Iceberg communities. >>>> >>>> Iceberg Vector type design doc (from Yan Yan): >>>> https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?tab=t.0 >>>> >>>> To accelerate the Vector type discussion, we are reusing/extending >>>> Thursday's biweekly series to cover both File and Vector types (*starting >>>> on Oct 8*). Since the file type discussion is reaching consensus, more >>>> time will likely be spent on the Vector type discussion. The corresponding >>>> dev calendar event title has been updated. >>>> >>>> Thanks, >>>> Steven >>>> >>>
