Well if I am correct, Parquet version 2 support was introduced in Spark version 2.4.0. Therefore, any version of Spark starting from 2.4.0 supports Parquet version 2. Assuming that you are using Spark version 2.4.0 or later, you should be able to take advantage of Parquet version 2 features.
HTH Mich Talebzadeh, Technologist | Solutions Architect | Data Engineer | Generative AI London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* The information provided is correct to the best of my knowledge but of course cannot be guaranteed . It is essential to note that, as with any advice, quote "one test result is worth one-thousand expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". On Mon, 15 Apr 2024 at 20:53, Prem Sahoo <prem.re...@gmail.com> wrote: > Thank you for the information! > I can use any version of parquet-mr to produce parquet file. > > regarding 2nd question . > Which version of spark is supporting parquet version 2? > May I get the release notes where parquet versions are mentioned ? > > > On Mon, Apr 15, 2024 at 2:34 PM Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > >> Parquet-mr is a Java library that provides functionality for working >> with Parquet files with hadoop. It is therefore more geared towards >> working with Parquet files within the Hadoop ecosystem, particularly using >> MapReduce jobs. There is no definitive way to check exact compatible >> versions within the library itself. However, you can have a look at this >> >> https://github.com/apache/parquet-mr/blob/master/CHANGES.md >> >> HTH >> >> Mich Talebzadeh, >> Technologist | Solutions Architect | Data Engineer | Generative AI >> London >> United Kingdom >> >> >> view my Linkedin profile >> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >> >> >> https://en.everybodywiki.com/Mich_Talebzadeh >> >> >> >> *Disclaimer:* The information provided is correct to the best of my >> knowledge but of course cannot be guaranteed . It is essential to note >> that, as with any advice, quote "one test result is worth one-thousand >> expert opinions (Werner >> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >> >> >> On Mon, 15 Apr 2024 at 18:59, Prem Sahoo <prem.re...@gmail.com> wrote: >> >>> Hello Team, >>> May I know how to check which version of parquet is supported by >>> parquet-mr 1.2.1 ? >>> >>> Which version of parquet-mr is supporting parquet version 2 (V2) ? >>> >>> Which version of spark is supporting parquet version 2? >>> May I get the release notes where parquet versions are mentioned ? >>> >>
