Hi all, I have a question on the complex Parquet reader.
I note in code that if anything in the Parquet file in non-primitive, it falls back to the other Parquet reader. I've also noted, when this happens, no matter how many drill bits I have, it doesn't seem to parallelize the processing - or at least, there is no improvement in performance. Does this still segment and process with all drillbits? Finally, would it be better in the short term to work out what columns are projected and check if those are primitives, then use the normal reader, as opposed to using the fallback reader any time a file contains anything complex? Let me know your thoughts.
