Hey Pritam, Thanks, and glad you liked the launch!
It's a great question; I'm not aware of any ongoing discussions around this, and I wouldn't want to speak to whether it makes sense on Spark's side. If anyone's interested in exploring this, perhaps starting from a lightweight angle of a Hadoop-free metadata/IO/filtering layer underneath an engine's own decoders, I'd be genuinely happy to dig in. Any feedback on how Hardwood could best facilitate this would be very welcome. Best, --Gunnar On Tue, 30 Jun 2026 at 22:55, Pritam Pan <[email protected]> wrote: > > Hi Gunner, congrats on the recent launch! Are there currently any > discussions or roadmaps regarding the integration of Hardwood into Apache > Spark? I was also wondering if you anticipate any related work starting in > the near future. > Thanks > > On Sat, Jun 27, 2026 at 11:47 AM Steve Loughran <[email protected]> wrote: > > > On Fri, 26 Jun 2026 at 21:22, Gunnar Morling < > > [email protected]> > > wrote: > > > > > Hey Steve, > > > > > > Thanks a lot, much appreciated! > > > > > > As for your pending parquet-testing PR, that's great! I ran all the > > > fixtures through Hardwood: everything is rejected except the depth > > > case (no guard yet, we'll add one). But several only reject > > > incidentally rather than for the right reason; a few expose missing > > > validation, and one even surfaced a latent metadata-parsing bug none > > > of our existing tests caught. > > > > > > > > That's what tests are for. Happy. > > > > > > > > > Everything is captured in [1], we'll get these fixed for Hardwood > > > 1.0.1. Once your PR has landed, we'll assert these new fixtures as > > > part of our existing sweep against parquet-testing. > > > > > > Thanks again for putting these together and for the pointer, that's > > > super-useful! > > > > > > > > > > >
