I'm ok with the change and 0.15 release to better manage it.
> I've always understood the metadata to be a few dozen/hundred KB, a > small percentage of the total message size. I could be underestimating > the ratios though -- is it common to have tables w/ 1000+ columns? I've > seen a few reports like that in cuDF, but I'm curious to hear > Jacques'/Dremio's experience too. > Metadata size has been an issue at different points for us. We do definitely see datasets with 1000+ columns. It is also compounded by the fact that as we add more columns, we typically decrease row count so that the individual batches are still easily pipelined--which further increases the relative ratio between data and metadata.
