This is a guestimate based on current parsing speeds I am measuring for 3k columns. Statistics are not optimized yet (which is the bulk of the metadata) and I am at 800us.
On Thu, Jun 6, 2024 at 5:48 PM Antoine Pitrou <anto...@python.org> wrote: > On Wed, 5 Jun 2024 21:41:39 +0200 > Alkis Evlogimenos > <alkis.evlogime...@databricks.com.INVALID> > wrote: > > (2) would take unduly long - if the metadata decoder is not performant > > enough. The speed of the decoder strongly depends on the encoding of > > choice. If we choose flatbuffers, 100'000 columns would parse in a few ms > > (with verification) or some much less significant time without. > > That's few 10s of ns for each column, which sounds small even for > flatbuffers. Did you actually measure this, or is it a guesstimate? > > Regards > > Antoine. > > >
