On Tue, 10 May 2022 14:21:23 -0700 William Butler <w...@google.com.INVALID> wrote: > We recently upgraded our version of Arrow and came across an interesting > issue. The version that we had been using was pre-logical types and the new > version is post-logical types. It turns out some customers have files with > invalid logical types. The old version, being oblivious to logical types, > reads them fine whereas the new version, which can see logical types, > rejects them. Parquet-mr seems a lot more forgiving of these situations. It > prints a warning and uses the converted type. What is the correct behavior > here? Should we make Parquet C++ more like the Java version?
Can you give an example of what an invalid logical type look like? Generally, I don't think errors should pass silently, especially when several implementations claim to support the same standard. The days of lax HTML in the late 90s/early 2000s, and the resulting compatibility hell, come to mind. > Similarly, while I haven't tried this, it looks like Parquet C++ will not > accept unrecognized logical types from Thrift. Can you point to the relevant piece of code? Regards Antoine.
