> the core problem we need to solve here is how Parquet writers communicate to Parquet readers which features must be supported in order to guarantee they can completely read a given file.
+1 - Preview features are a good idea, but thinking through how they work is secondary to fixing metadata to enable versions. I have specific replies to Micah below, but after having written responses, I think his 4 points are mostly about preview features (except 3, but that's phrased as a "SHOULD"). Unless I'm mistaken, these aren't really blockers and I think we should try to move forward with the magic and version field discussion as Andrew suggests. > I do not think 1 and 2 are contentious; they follow from the vote. I agree with Russell here. These points are good limits on preview features. We can never produce incorrect results, so unsupported preview features must cause reasonable failures (2). And we shouldn't release features as a preview if they are unfinished and could change in a way that causes incorrect results or failures in readers with full support (1). > A reader SHOULD be able to give a detailed error message about why it couldn't read the file, regardless of if it can parse the thrift metadata structure. The most important clarification to this is that not supporting the format version is the correct error message when you can't decode the footer. The second part, "regardless of if it can parse the thrift metadata" is very confusing to me. When a reader encounters a newer format version, it can't just assume that the metadata is thrift and attempt to read it. > Old readers should be able to read partial data from new files even if they don't support a particular encoding or version the encoding was added in. I _think_ this is referring to preview features, because readers cannot read newer format versions and forward compatibility is already required within a format version: new features within a format version cannot cause older readers to fail or read incorrect data. However, we want more flexibility with preview features and in that context, this is a good idea. We want to minimize the disruption caused by preview features. I'm not sure we want to say more than that or how we might phrase this to be narrow and clear. The encoding example is good, but just an example. This phrasing is okay, but is still just a recommendation since it uses "should be able to". I'm not sure it's worth making this guideline "official" because it is going to come down to judgement.
