etseidl commented on code in PR #258: URL: https://github.com/apache/parquet-format/pull/258#discussion_r1628790163
########## CONTRIBUTING.md: ########## @@ -29,3 +29,92 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +The general steps for adding features to the format are as follows: + +1. Discuss changes on on the developer mailing list ([email protected]). Often times it is helpful to link to a draft pull request to make the discussion concrete. This step is complete when there lazy consensus. + +2. One a change has lazy consensus two implementations of the feature +demonstrating interopability must also be provided. One implementation MUST be [parquet-java](http://github.com/apache/parquet-java). It is preferred that the second implementation be [parquet-cpp](https://github.com/apache/arrow) or [parquet-rs](https://github.com/apache/arrow-rs), however at the discretion of the PMC any +open source Parquet implementation may be acceptable. Implementations +whose contributors actively +participate in the community (e.g. keep their feature matrix +up-to-date on parquet-site) are more likely to be considered. + +Unless otherwise discussed, it is expected the implementations will +develop from the main branch (i.e. packporting is not expected). Review Comment: ```suggestion develop from the main branch (i.e. backporting is not expected). ``` ########## CONTRIBUTING.md: ########## @@ -29,3 +29,92 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +The general steps for adding features to the format are as follows: + +1. Discuss changes on on the developer mailing list ([email protected]). Often times it is helpful to link to a draft pull request to make the discussion concrete. This step is complete when there lazy consensus. + +2. One a change has lazy consensus two implementations of the feature Review Comment: ```suggestion 2. Once a change has lazy consensus two implementations of the feature ``` ########## CONTRIBUTING.md: ########## @@ -29,3 +29,92 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +The general steps for adding features to the format are as follows: + +1. Discuss changes on on the developer mailing list ([email protected]). Often times it is helpful to link to a draft pull request to make the discussion concrete. This step is complete when there lazy consensus. + +2. One a change has lazy consensus two implementations of the feature +demonstrating interopability must also be provided. One implementation MUST be [parquet-java](http://github.com/apache/parquet-java). It is preferred that the second implementation be [parquet-cpp](https://github.com/apache/arrow) or [parquet-rs](https://github.com/apache/arrow-rs), however at the discretion of the PMC any +open source Parquet implementation may be acceptable. Implementations +whose contributors actively +participate in the community (e.g. keep their feature matrix +up-to-date on parquet-site) are more likely to be considered. + +Unless otherwise discussed, it is expected the implementations will +develop from the main branch (i.e. packporting is not expected). + +In some cases in addition to library level implementations it is +expected the changes to be justified with integration into a +processing engine to show there viability. + +3. After the first two steps are complete a formal vote is held on the Parquet mailing list to officially +ratify the feature. After the vote passes the format change is merged into the parquet-format repository +and it is expected the change in step 2 will also be merged soon after. Before merging into Parquet-java a parquet-format release Review Comment: ```suggestion and it is expected the changes from step 2 will also be merged soon after. Before merging into Parquet-java a parquet-format release ``` ########## CONTRIBUTING.md: ########## @@ -29,3 +29,92 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +The general steps for adding features to the format are as follows: + +1. Discuss changes on on the developer mailing list ([email protected]). Often times it is helpful to link to a draft pull request to make the discussion concrete. This step is complete when there lazy consensus. + +2. One a change has lazy consensus two implementations of the feature +demonstrating interopability must also be provided. One implementation MUST be [parquet-java](http://github.com/apache/parquet-java). It is preferred that the second implementation be [parquet-cpp](https://github.com/apache/arrow) or [parquet-rs](https://github.com/apache/arrow-rs), however at the discretion of the PMC any +open source Parquet implementation may be acceptable. Implementations +whose contributors actively +participate in the community (e.g. keep their feature matrix +up-to-date on parquet-site) are more likely to be considered. + +Unless otherwise discussed, it is expected the implementations will +develop from the main branch (i.e. packporting is not expected). + +In some cases in addition to library level implementations it is +expected the changes to be justified with integration into a Review Comment: ```suggestion expected the changes will be justified via integration into a ``` ########## CONTRIBUTING.md: ########## @@ -29,3 +29,92 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +The general steps for adding features to the format are as follows: + +1. Discuss changes on on the developer mailing list ([email protected]). Often times it is helpful to link to a draft pull request to make the discussion concrete. This step is complete when there lazy consensus. + +2. One a change has lazy consensus two implementations of the feature +demonstrating interopability must also be provided. One implementation MUST be [parquet-java](http://github.com/apache/parquet-java). It is preferred that the second implementation be [parquet-cpp](https://github.com/apache/arrow) or [parquet-rs](https://github.com/apache/arrow-rs), however at the discretion of the PMC any +open source Parquet implementation may be acceptable. Implementations +whose contributors actively +participate in the community (e.g. keep their feature matrix +up-to-date on parquet-site) are more likely to be considered. + +Unless otherwise discussed, it is expected the implementations will +develop from the main branch (i.e. packporting is not expected). + +In some cases in addition to library level implementations it is +expected the changes to be justified with integration into a +processing engine to show there viability. + +3. After the first two steps are complete a formal vote is held on the Parquet mailing list to officially +ratify the feature. After the vote passes the format change is merged into the parquet-format repository +and it is expected the change in step 2 will also be merged soon after. Before merging into Parquet-java a parquet-format release +must be performed. + +#### General guidelines/preferences on additions. + +1. To the greatest extent possible changes should have an option for forwards compatibility (old readers can still read files). +2. New encodings should be fully specified in this repository and ideally not rely on an external dependencies for implementation (i.e. Parquet is the source of truth for the encoding). +3. New compression mechanisms must have a pure Java implementation that can be used as dependency in parquet-java. + +### Releases + +The Parquet community aims to do releases of the format package only as needed when new features are introduced. +If multiple new features are being proposed simultaneously some features might be consolidated into the same release. Guidance is provided below on when implementations should enable features added to the specification. +Due to confusion in the past over parquet versioning it is not expected that there will be a 3.0 release of the specification in the foreseeable future. + +### Compatibility and Feature Enablement + +For the purposes of this discussion we classify features into the following buckets: + +1. Backwards compatible. A file written by an older version of a library can be read by a newer version of the +library. + +2. Forwards compatible. A file written by a new version of the library can be read by an older version +of the library. + +3. Forward compatible with suboptimal performance. A file written by a new version of the library can +be read an older version of the library but performance might be suboptimal (e.g. statistics might be missing +from the older reader's perspective). + +4. Forward incompatible. A file written with a new version of the library cannot be read by an older version +of the library. + +The Parquet community hopes that new features are widely beneficial +to users of Parquet, and therefore third-party implementations will +adopt them quickly after they are introduced. It is assumed that most new features will be implemented behind a feature flag that defaults to "off".To avoid, compatibility issues across the ecosystem some amount of lead time is desirable to ensure a critical mass of Parquet implementations support a feature. Therefore, the Parquet PMC gives the following guidance for changing a feature to be "on" by default: + +1. Backwards compatibility is the concern of implementations but given the ubiquity of Parquet and the length +of time it has been used, libraries SHOULD support reading older +file variants. + +2. Forward compatible changes MAY be used by default in implementations once the parquet-format containing +those changes has been formally released. These features SHOULD be turned on 1 year after the parquet-java +implementation containing feature is released (e.g. it is expected +the Java implementation itself will turn them on for the first +release 1 year after a features initial introduction). + +3. Forward compatible with suboptimal performance features MAY be made default after +the parquet-java implementation containing the feature is released. Features in this category +SHOULD be turned on 1 year after the parquet-java +implementation containing the feature is released. Implementations MAY choose +to do a major version bump when turning on a feature in this category. + +4. Forwards incompatible changes MAY be made default 2 years after the parquet-java +implementation containing the feature is released. Features in this category SHOULD be turned on by +default 3 years after the parquet-java implementation containing feature is released. Implementations MUST do +a major version bump when enabling a forward incompatible feature by default. Review Comment: Do we want to mandate versioning here? Or should this just apply to parquet-java? ########## CONTRIBUTING.md: ########## @@ -29,3 +29,92 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +The general steps for adding features to the format are as follows: Review Comment: Is it worth leading this off with something to the effect that minor changes to the spec (fixing typos, providing clarification) can be made at the discretion of the committers without the need for a more formal process? Or is that just obvious? ########## CONTRIBUTING.md: ########## @@ -29,3 +29,92 @@ Recommendations and requirements for how to best contribute to Parquet. We striv ### License By contributing your code, you agree to license your contribution under the terms of the APLv2: https://github.com/apache/parquet-format/blob/master/LICENSE + +### Additions/Changes to the Format + +The general steps for adding features to the format are as follows: + +1. Discuss changes on on the developer mailing list ([email protected]). Often times it is helpful to link to a draft pull request to make the discussion concrete. This step is complete when there lazy consensus. + +2. One a change has lazy consensus two implementations of the feature +demonstrating interopability must also be provided. One implementation MUST be [parquet-java](http://github.com/apache/parquet-java). It is preferred that the second implementation be [parquet-cpp](https://github.com/apache/arrow) or [parquet-rs](https://github.com/apache/arrow-rs), however at the discretion of the PMC any +open source Parquet implementation may be acceptable. Implementations +whose contributors actively +participate in the community (e.g. keep their feature matrix +up-to-date on parquet-site) are more likely to be considered. + +Unless otherwise discussed, it is expected the implementations will +develop from the main branch (i.e. packporting is not expected). + +In some cases in addition to library level implementations it is +expected the changes to be justified with integration into a +processing engine to show there viability. Review Comment: ```suggestion processing engine to show their viability. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
