tustvold commented on issue #4764: URL: https://github.com/apache/arrow-rs/issues/4764#issuecomment-1704391735
> feature gating the IPC readers The IPC readers are necessary to read the embedded arrow schema data, I'm also surprised if they are a meaningful bottleneck in compile times. > and take kernels out The parquet crate doesn't make use of the take kernel, it does make use of the cast kernels though which indirectly use the cast kernel. This would be non-trivial to remove, especially for decimals > This should not hurt your test times as I will ensure the default features remain the same Yes until someone uses disabled default features :sweat_smile: I dunno, I have found features to not really justify their downsides. > But I don't think it's needed here as I don't plan to mess with the defaults. We still need to ensure compilation with non-default sets of feature flags? We currently already have a fairly long list of combinations in CI... > The compile times in polars are already way longer than we want. That's why I'd like to cherry pick on functionality Perhaps we might do this empirically? If there are kernels that represent an outsize impact on compile times we then work out mitigation strategies for them? I strongly suspect that there are couple of particularly problematic sub-kernels (likely dictionaries) that have an outsize impact, and which it might be able to address this without having to resort to feature flags? FWIW I have some ideas on how to make the take kernel less expensive. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
