Hi all, I don't have a strong preference for any option over the others, but want to respond to a few of Philipp's points.
> On the Parquet > level, it might be okay to mix both fixed-size list and vector use > cases, but on an Iceberg level, we would prefer to separate the two > cases, based on current discussions. Curious to hear what others > think. I agree that having dedicated numeric vector types at the engine/application level makes sense, but don't see a good reason to only support numeric vectors in Parquet if we can find a solution that also supports fixed-size lists of other non-numeric types without being detrimental to the numeric case. > Out of the presented Parquet options, option A is closest to such a > dedicated numeric vector type in its layout/composability I don't follow this argument. All three options end up with the same layout for the values on disk when using a plain encoding, so aren't they equivalent from this point of view? SIMD operations will work just as effectively with all options. > For option B, such types would have to be added as full Parquet > primitive types, even though they would likely be only used in a > vector context. They don't need to be added as physical types, but could be new logical types. Float16 was added recently as a logical type annotating a 2-byte FLBA for example. I think if Parquet were to support vectors of bfloat16 or fp8, it would make sense to also introduce logical types for the scalar values at the same time. We probably want to ensure vectors of opaque FLBA types are supported too though. > My intuition is > that for option B/C, many writers would only utilize such fixed-size > lists for primitive types, so I’m wondering how big the composability > gap really is. I agree that we should prioritize vectors of primitive types. Fixed size lists of more complex types can always fall back to the existing list logical type if necessary, and the performance impact is probably not such a concern for these use cases. Thanks, Adam On Fri, 3 Jul 2026 at 02:48, Philipp Fischbeck via dev < [email protected]> wrote: > Hi all, > > I wanted to reiterate and expand the points I raised in the Parquet > community sync yesterday for option A. > > Several engines, including Snowflake, Oracle, Microsoft SQL Server, > and Spark have a dedicated numeric vector type [1, 2, 3, 4] for vector > use cases, instead of handling both use cases with one generic > fixed-size list type. They treat numeric vectors as a dedicated > semantic type with shared operations like distance computations. These > engines would likely map fixed-size lists not to one type, but either > their dedicated numeric vector type (when applicable), or a > variable-length list with additional length verification. Based on > what current engines do, we think it could be good for Iceberg to also > clearly separate these two types and their semantics. On the Parquet > level, it might be okay to mix both fixed-size list and vector use > cases, but on an Iceberg level, we would prefer to separate the two > cases, based on current discussions. Curious to hear what others > think. > > Out of the presented Parquet options, option A is closest to such a > dedicated numeric vector type in its layout/composability, and > somewhat easy to implement for readers and writers. For many vector > engines, FLBA representation like in option A is what they would use > during computation anyways for effective SIMD computations and > transfer. Additionally, the use case for numeric vectors as embeddings > (e.g., for similarity computations) does not require element > nullability. > > If encodings are indeed a big concern, it would also be an option to > introduce such encodings based on the logical type, as discussed in > Rok’s doc [5]. Furthermore, vectors might also require additional > vector-specific encodings in the future that are separate from the > current Parquet encodings. > > Some data types used in vector embeddings, like fp8 and bfloat16 [6], > are currently not native Parquet types. Adjusting option A to its enum > version, i.e., specifying a list of allowed vector element types (and > their semantics/byte encoding) that is independent of Parquet native > types, allows for easy support for various specialized element types. > For option B, such types would have to be added as full Parquet > primitive types, even though they would likely be only used in a > vector context. > > Regarding composability: The current spec draft [7] mentions that > vector element types cannot contain repeated fields. My intuition is > that for option B/C, many writers would only utilize such fixed-size > lists for primitive types, so I’m wondering how big the composability > gap really is. > > [1] https://docs.snowflake.com/en/sql-reference/data-types-vector > [2] > https://docs.oracle.com/en/database/oracle/oracle-database/26/vecse/create-tables-using-vector-data-type.html > [3] > https://learn.microsoft.com/en-us/sql/t-sql/data-types/vector-data-type?view=sql-server-ver17&tabs=csharp > [4] > https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.linalg.Vectors.html > [5] > https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?disco=AAAB9j9KyAI > [6] https://en.wikipedia.org/wiki/Bfloat16_floating-point_format > [7] https://github.com/apache/parquet-format/pull/592 > > Philipp > > > On Wed, Jul 1, 2026 at 8:35 PM Andrew McCormick via dev > <[email protected]> wrote: > > > > Hi Rok, > > > > Just posted a comment to the doc but wanted to add it here, plus add a > > little bit of extra info. > > > > Option C looks like by far the strongest option to me. Here's my take: > > > > Option A: not backwards compatible, poor encodings -- little upside. > > Option B: clearly the path we'd take if we were designing a new format, > but > > for parquet as-is it would require a tremendous amount of work, and that > > alone makes it untenable unless the gain it gives is way better than > other > > options. > > Option C: Fully backwards compatible, keeps full encodings, small amount > of > > work to implement on readers and writers. The only downside is that we > > still have to store the rep levels on disk (and load them), but due to > the > > fixed length arrays they compress to almost nothing under the RLE > > encodings, so the cost is tiny. > > > > The extra info I wanted to add is that even for files without anything > > added to them, you can in fact cheaply detect whether the array in > question > > is of a fixed size by doing some logic on the compressed rle data. On an > > example dataset I benchmarked (100k rows of 4k-float array features) I > > measured the decompression time to be 2.3x faster than our baseline > reading > > path. That verification does of course cost some time, and if we had the > > hint from C in the data we could skip it, giving us another 1.5x. > > > > On Wed, Jul 1, 2026 at 11:24 AM Jiayi Wang <[email protected]> wrote: > > > > > > > > > > > > ---------- Forwarded message --------- > > > 发件人： Rok Mihevc <[email protected]> > > > Date: 2026年6月29日周一 20:13 > > > Subject: Re: [DISCUSSION] Introduce FIXED_SIZE_LIST logical type > > > To: <[email protected]> > > > Cc: <[email protected]> > > > > > > > > > Hi all, > > > > > > Based on benchmarks [1] comparing implementation variants, feedback > > > received via > > > the design doc [2] and in-person feedback I would like to move forward > with > > > the approach that introduces a new repetition type (Option B in the > design > > > doc and benchmarks) unless there are objections. Either here or on this > > > week's community call would be a good venue to raise early objections > > > or provide feedback on how to proceed. > > > > > > To show what a new repetition type would look like I've opened a > > > parquet-format PR [3] and a draft Go implementation [4]. > > > > > > [1] https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef > > > [2] > > > > > > > https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0 > > > [3] https://github.com/apache/parquet-format/pull/592 > > > [4] https://github.com/apache/arrow-go/pull/854 > > > > > > Rok > > > > > > On Sun, Jun 14, 2026 at 10:10 PM Rok Mihevc <[email protected]> > wrote: > > > > > > > Hi all, > > > > > > > > A short update on the progress of this work. State of discussion can > be > > > > seen here [1]. > > > > I've created a set of naive C++ implementations of the discussed > designs; > > > > see here: > https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef > > > > Results should be taken with a grain of salt and more of a > directional > > > > rather than quantitative information. > > > > > > > > Personally I'm leaning towards option B because it is quite > expressive > > > > while still providing significant storage and writing performance > > > > improvement. > > > > > > > > [1] > > > > > > > > https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?usp=sharing > > > > [2] https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef - > > > > benchmarks > > > > [3] https://github.com/rok/arrow/pull/53 - option A > > > > [4] https://github.com/rok/arrow/pull/51 - option B > > > > [5] https://github.com/rok/arrow/pull/52 - option C > > > > > > > > Rok > > > > > > > > On Tue, May 5, 2026 at 3:21 PM Rok Mihevc <[email protected]> > wrote: > > > > > > > >> Hi all, > > > >> > > > >> Picking this thread back up. I've put together a design doc > outlining > > > >> three options we've discussed: > > > >> > > > >> > > > > https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?usp=sharing > > > >> > > > >> * Option A: logical type annotating FIXED_LEN_BYTE_ARRAY. > > > >> * Option B: new VECTOR repetition type. > > > >> * Option C: logical type annotating a normal LIST, where a > recognizing > > > >> reader skips rep-level decode and an unknown reader still sees a > working > > > >> LIST. A future revision would let writers omit rep-levels entirely. > > > >> > > > >> The document evaluates these against the same requirements and > compares > > > >> them along six axes (backwards compatibility, composability, > encoding > > > >> flexibility, implementation complexity, on-disk overhead, read > > > >> performance). The doc aims to centralize the discussion and help us > > > pick a > > > >> direction. > > > >> Comments are open. Most useful pushback would be on the requirements > > > >> (especially the "no-fallback breaks adoption" one). > > > >> > > > >> Best, > > > >> Rok > > > >> > > > >> On Tue, Mar 3, 2026 at 8:58 PM Antoine Pitrou <[email protected]> > > > wrote: > > > >> > > > >>> > > > >>> Hello, > > > >>> > > > >>> The downside with this approach is that the top-level "unit" type > is > > > not > > > >>> the element type. > > > >>> > > > >>> For example, if you have a FIXED_SIZE_LIST(FLOAT32, 3), then the > > > >>> top-level unit type is FIXED_LEN_BYTE_ARRAY(12). This means that > > > >>> specialized encodings such as BYTE_STREAM_SPLIT, > DELTA_BINARY_PACKED or > > > >>> ALP may either be less efficient (for BYTE_STREAM_SPLIT) or not be > > > >>> applicable at all (for the latter two). > > > >>> > > > >>> I wonder if we can find an approach that doesn't emit repetition > levels > > > >>> but still allows using efficient encodings for the element type. > > > >>> > > > >>> Regards > > > >>> > > > >>> Antoine. > > > >>> > > > >>> > > > >>> Le 03/03/2026 à 01:13, Rok Mihevc a écrit : > > > >>> > Hi all, > > > >>> > > > > >>> > I'd like to resurrect this thread in light of recent vectors in > > > Parquet > > > >>> > discussion [1]. > > > >>> > There is a (now updated) proposal PR from when the thread was > started > > > >>> that > > > >>> > has a nice discussion [2]. > > > >>> > > > > >>> > TLDR of the current proposal: > > > >>> > - FIXED_SIZE_LIST annotates a FIXED_LEN_BYTE_ARRAY primitive leaf > > > with > > > >>> > FixedSizeListType { type, num_values }. > > > >>> > - type must be fixed-width and non-array (INT32, INT64, FLOAT, > > > DOUBLE, > > > >>> > FIXED_LEN_BYTE_ARRAY); num_values > 0. > > > >>> > - type_length must match num_values encoded with PLAIN > representation > > > >>> of > > > >>> > type. > > > >>> > - If the field is optional, the whole list value may be null; > > > elements > > > >>> are > > > >>> > always non-null. > > > >>> > - Intentionally not a `LIST` encoding (no def/rep levels). > > > >>> > - Outer page/column encoding behavior is unchanged (any encoding > > > valid > > > >>> for > > > >>> > `FIXED_LEN_BYTE_ARRAY` remains valid). > > > >>> > > > > >>> > I also added explicit validity requirements: writers must not > emit > > > >>> > violating metadata, and readers must treat violating metadata as > > > >>> invalid. > > > >>> > > > > >>> > > > > >>> > Rok > > > >>> > > > > >>> > [1] > https://lists.apache.org/thread/nmq7odlbg1p6yx0hg00clzjbc3tb1tc3 > > > >>> > [2] https://github.com/apache/parquet-format/pull/241 > > > >>> > > > > >>> > On Thu, May 16, 2024 at 4:34 AM Jan Finis <[email protected]> > wrote: > > > >>> > > > > >>> >> I would love to see this! > > > >>> >> > > > >>> >> It is an important optimization for vectors, which become more > and > > > >>> more > > > >>> >> important and ubiquitous for grounding of LLMs. > > > >>> >> > > > >>> >> Note however that the logical type route has one drawback: A > logical > > > >>> type > > > >>> >> may not change the physical representation of values! Thus, if > we > > > make > > > >>> >> FIXED_SIZE_LIST just a logical type, we would still need to > write > > > >>> R-Levels, > > > >>> >> as even clients not knowing this logical type need to be able to > > > >>> decode the > > > >>> >> column. We could avoid reading the R-Levels and just assume that > > > each > > > >>> list > > > >>> >> has the fixed size, so the read path would be optimized but the > > > write > > > >>> path > > > >>> >> wouldn't. > > > >>> >> > > > >>> >> If we want to avoid writing R-Levels altogether, a logical type > > > >>> doesn't cut > > > >>> >> it. It needs to be something different. E.g., in the schema, we > > > could > > > >>> store > > > >>> >> an optional `count` for repeated fields. Whenever this count is > > > >>> present, we > > > >>> >> would not write R-Levels for this field (or more precisely, this > > > field > > > >>> >> would not take part in the R-Level computation, as if it wasn't > a > > > >>> repeated > > > >>> >> field). This of course is a more intrusive change, as legacy > clients > > > >>> >> couldn't read such columns anymore. > > > >>> >> > > > >>> >> I don't know which of the two alternatives is better. I agree > with > > > >>> Gang > > > >>> >> that we should probably discuss this in a PR. > > > >>> >> > > > >>> >> Cheers, > > > >>> >> Jan > > > >>> >> > > > >>> >> > > > >>> >> Am Mi., 15. Mai 2024 um 14:03 Uhr schrieb Gang Wu < > [email protected] > > > >: > > > >>> >> > > > >>> >>> Hi Rok, > > > >>> >>> > > > >>> >>> Happy to see you here :) > > > >>> >>> > > > >>> >>> According to my past experience, it would be more helpful to > open > > > >>> >>> a PR against the parquet-format repository and post it here. > > > >>> >>> > > > >>> >>> Best, > > > >>> >>> Gang > > > >>> >>> > > > >>> >>> On Wed, May 15, 2024 at 7:25 PM Rok Mihevc < > [email protected]> > > > >>> wrote: > > > >>> >>> > > > >>> >>>> Hi all, > > > >>> >>>> > > > >>> >>>> Arrow recently introduced FixedShapeTensor and > VariableShapeTensor > > > >>> >>>> canonical extension types [1] that use FixedSizeList and > > > >>> >>> StructArray(List, > > > >>> >>>> FixedSizeList) as storage respectfully. These are targeted at > > > >>> machine > > > >>> >>>> learning and scientific applications that deal with large > datasets > > > >>> and > > > >>> >>>> would benefit from using Parquet as on disk storage. > > > >>> >>>> > > > >>> >>>> However currently FixedSizeList is stored as List in Parquet > which > > > >>> adds > > > >>> >>>> significant conversion overhead when reading and writing [2]. > It > > > >>> would > > > >>> >>>> therefore be beneficial to introduce a FIXED_SIZE_LIST logical > > > type. > > > >>> >>>> > > > >>> >>>> I would like to open a discussion on potentially adding > > > >>> FIXED_SIZE_LIST > > > >>> >>>> type and prepare a proposal if discussion supports it. > > > >>> >>>> > > > >>> >>>> > > > >>> >>>> Best, > > > >>> >>>> Rok > > > >>> >>>> > > > >>> >>>> [1] > > > >>> >>>> > > > >>> >>> > > > >>> >> > > > >>> > > > > https://arrow.apache.org/docs/format/CanonicalExtensions.html#official-list > > > >>> >>>> [2] https://github.com/apache/arrow/issues/34510 > > > >>> >>>> > > > >>> >>> > > > >>> >> > > > >>> > > > > >>> > > > >>> > > > >>> > > > >
