kartik00052 commented on issue #27905: URL: https://github.com/apache/arrow/issues/27905#issuecomment-5301494929
Thanks! Yeah, makes sense — #24418 has no champion and isn't happening anytime soon, so I don't want to bite that off. Since this is tagged as good-first-issue + docs now, I was thinking of just documenting the actual behavior instead of trying to fix it: Add a note in the schema param docs for pyarrow.dataset.dataset() — basically that if you don't pass schema explicitly, it's inferred from a single fragment (not unified across the whole dataset), so sparse/mismatched files can quietly turn into null columns or blow up later with ArrowTypeError. Mention passing schema= explicitly as the fix. Add a small example to the Dataset guide (docs/source/python/dataset.rst) showing this exact gotcha, using something like the NA-first-chunk repro from this thread. Seems like the realistic scope here. Should I also throw in the exclude_invalid_files default-value mismatch from #46181, or keep that separate? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
