Thanks for replying and reviewing the PR, everyone! I'm going to merge this so we can unblock other PRs.
On Thu, Sep 3, 2026 at 7:55 AM Russell Spitzer <[email protected]> wrote: > 1) Feels like the best option for right now. It's imho, not worth debating > the mechanics of a better solution till we have the simplest one in place > first. > > On Thu, Sep 3, 2026 at 9:44 AM Divjot Arora via dev < > [email protected]> > wrote: > > > Thanks for all of the discussion here folks. The PR has two approvals, so > > I'd like to merge it soon. > > For posterity, the following options were considered: > > > > 1. Inline parquet.thrift (chosen option): There is precedent for this in > > projects like apache/arrow. > > The final PR [1] only contains a script to update the file based on a > > parquet-format ref. We removed previous > > iterations of scripts that validated that the file matches a released > > parquet-format version during the > > parquet-java release process as there is no precedent for such checks in > > other projects. > > > > 2. Use parquet-format snapshot releases. This solves local development > > issues, but CI checks would still > > fail for reference implementations of in-progress Parquet spec changes. > > > > 3. Use git submodules. As mentioned above, this would likely solve the > > development issues, but adds > > friction for users/maintainers. > > > > Overall, I feel (1) is the simplest and most flexible option. It comes > with > > the minor risk that we accidentally > > release parquet-java pointing to arbitrary parquet.thrift changes that > > don't match parquet-format, but the > > release process is quite sturdy with many people double-checking things, > so > > I think the risk is low. > > It is easy to add additional scripts for verifying the parquet.thrift > file > > in the future if necessary. > > > > Best, > > Div > > > > [1] https://github.com/apache/parquet-java/pull/3709 > > > > On Thu, Sep 3, 2026 at 7:31 AM Fokko Driesprong <[email protected]> > wrote: > > > > > Thanks Gang, I had the same thought, and I think it can work. However, > in > > > practice, I find submodules to be clunky and user-unfriendly. For > > example, > > > you need to run additional git commands to fetch the submodules. > > Sometimes > > > it is stuck on an older version, and I always find it awkward to work > > with. > > > > > > Another possibility I suggested in the PR itself is to use nightly > > > snapshots. What I like is that we don't duplicate the thrift > definitions > > in > > > the Java repo, and we can work on something that's unreleased but on > the > > > parquet-format main branch. I noticed that we stopped producing nightly > > > snapshots a while ago. I created a PR to reinstate that [1]. On the PR, > > we > > > ended up on the fact that working from snapshots isn't as flexible as > > Div's > > > setup, where you can make the changes in the same PR. > > > > > > At all costs, I think we want to avoid forcing folks to write Perl > > scripts > > > to amend the definitions [2] 🤣. > > > > > > I believe inlining is the most flexible approach, since it makes the > > > changes transparent. When releasing, we should double-check that the > > > current version of the definitions matches the latest released version > of > > > parquet-format. > > > > > > Kind regards, > > > Fokko > > > > > > [1] https://github.com/apache/parquet-format/pull/614 > > > [2] > > > > > > https://github.com/apache/parquet-java/pull/3397/changes#diff-07fef3853dd3df8f18fa5b6192e36d1f60a081591a529cf0c3558ab4c093322fR69-R97 > > > > > > On 2026/09/03 02:16:42 Gang Wu wrote: > > > > Ah sorry, my reply above is **NOT** blocking > > > > > > > > On Thu, Sep 3, 2026 at 10:16 AM Gang Wu <[email protected]> wrote: > > > > > > > > > I'm in favor of inlining parquet.thrift overall so my reply below > is > > > > > blocking. > > > > > > > > > > I'm not sure if we have explored making parquet-format as a git > > > submodule > > > > > in the parquet-java. Then we can point to a commit hash without any > > > script > > > > > to > > > > > validate the thrift file. It would also be easy to validate if that > > > commit > > > > > belongs to a release tag when releasing parquet-java. > > > > > > > > > > On Thu, Sep 3, 2026 at 6:29 AM Ryan Blue <[email protected]> wrote: > > > > > > > > > >> I just approved the PR that Div opened (linked in his email). > > > > >> > > > > >> Right now, the PR updates the build to use a local copy of the > > thrift > > > file > > > > >> and adds a script to download new copies from parquet-format. New > > > copies > > > > >> are specified by commit hash or by a ref, and the resolved version > > is > > > > >> tracked in a parquet-format.version file. > > > > >> > > > > >> The script makes it easy to update to the latest copy, but rather > > than > > > > >> having scripts to check and diff the local copy I suggested that > we > > > use > > > > >> `git` instead. Since the file is version controlled, we have very > > good > > > > >> tools to view changes and history already. > > > > >> > > > > >> I'd like to merge this soon, unless there are objections. Please > > take > > > a > > > > >> look at the PR if you're interested. > > > > >> > > > > >> Ryan > > > > >> > > > > >> On Wed, Aug 5, 2026 at 2:02 PM Divjot Arora via dev < > > > > >> [email protected]> > > > > >> wrote: > > > > >> > > > > >> > I've put up a PR [1] to inline parquet.thrift into parquet-java > > and > > > > >> > remove the upstream > > > > >> > dependency on parquet-format. There are some scripts to help > > manage > > > > >> > upgrading the inlined > > > > >> > copy as well as to check whether it references a released > > > parquet-format > > > > >> > version. Please take > > > > >> > a look if interested, I look forward to any feedback. > > > > >> > > > > > >> > Best, > > > > >> > Div > > > > >> > > > > > >> > [1] https://github.com/apache/parquet-java/pull/3709 > > > > >> > > > > > >> > On Mon, Jul 27, 2026 at 12:25 PM Divjot Arora < > > > [email protected] > > > > >> > > > > > >> > wrote: > > > > >> > > > > > >> > > Thanks for the input folks. I agree we want to keep this > > > discussion > > > > >> > > focused on > > > > >> > > parquet-java's use of parquet.thrift rather than the > versioning > > > > >> cadence > > > > >> > of > > > > >> > > parquet-format. > > > > >> > > I'll start a POC so we can discuss specifics. My thinking is > > that > > > > >> there > > > > >> > > should be some > > > > >> > > script to update the inlined version to match a commit hash in > > > > >> > > parquet-format. The main > > > > >> > > branch can track parquet-format commits without any issue > (this > > > will > > > > >> > allow > > > > >> > > reference > > > > >> > > implementations for new features to actually get merged). I > can > > > look > > > > >> into > > > > >> > > a release-time > > > > >> > > check to ensure that released versions of parquet-java > strictly > > > track > > > > >> > > released versions of > > > > >> > > parquet.format. > > > > >> > > > > > > >> > > -- Div > > > > >> > > > > > > >> > > On Fri, Jul 24, 2026 at 7:47 PM Julien Le Dem < > > [email protected]> > > > > >> wrote: > > > > >> > > > > > > >> > >> On Fri, Jul 24, 2026 at 10:30 AM Ryan Blue <[email protected]> > > > wrote: > > > > >> > >> > > > > >> > >> > > I think we should also make more frequent releases of > > > > >> > parquet-format. > > > > >> > >> > > > > > >> > >> > I'd like to keep this a separate topic from how we work > with > > > the > > > > >> > thrift > > > > >> > >> > file in parquet-java so that we can move forward > > > independently. We > > > > >> > don't > > > > >> > >> > need to know how parquet-format is released to agree that > we > > > want > > > > >> to > > > > >> > be > > > > >> > >> > able to build against newer structures without a formal > > > release, > > > > >> just > > > > >> > >> like > > > > >> > >> > the other implementations do. > > > > >> > >> > > > > > >> > >> > > > > >> > >> Fair enough. > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > > >> > >> > > What is the mechanism to track what version of > > > parquet-format, > > > > >> > >> > parquet-java > > > > >> > >> > depends on? > > > > >> > >> > > > > > >> > >> > Like the other projects, I don't think we track a specific > > > version > > > > >> of > > > > >> > >> the > > > > >> > >> > one from parquet-format, since we may pick changes that > have > > > not > > > > >> been > > > > >> > >> > released (as that's the pain point we're trying to solve). > > > > >> > >> > > > > > >> > >> > That said, I suggest that we sync the format into > > parquet-java > > > when > > > > >> > the > > > > >> > >> > upstream format is released, if not sooner. We should > always > > > use a > > > > >> > >> version > > > > >> > >> > that corresponds to a parquet-format commit to avoid drift. > > > > >> > >> > > > > > >> > >> > > > > > >> > >> Either in the main branch of parquet-java or at a minimum in > > > > >> releases of > > > > >> > >> parquet-java we should have an automated check that this file > > is > > > in > > > > >> sync > > > > >> > >> with an official release of parquet-format. > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > >> > >> > Ryan > > > > >> > >> > > > > > >> > >> > On Fri, Jul 24, 2026 at 10:21 AM Julien Le Dem < > > > [email protected]> > > > > >> > >> wrote: > > > > >> > >> > > > > > >> > >> > > Hello, > > > > >> > >> > > I'm strongly in favor of enabling a passing build on PRs > > like > > > > >> #3610 > > > > >> > >> > > What is the mechanism to track what version of > > > parquet-format, > > > > >> > >> > > parquet-java depends on? Is there a check to track > > > discrepancies > > > > >> or > > > > >> > >> > drift? > > > > >> > >> > > A > > > > >> > >> > > POC would help clarify this. > > > > >> > >> > > > > > > >> > >> > > Separately, I think we should also make more frequent > > > releases of > > > > >> > >> > > parquet-format. > > > > >> > >> > > That could be formalized in our new versioning process. > > > > >> > >> > > In particular every time we merge a new change to the > > thrift > > > > >> file, > > > > >> > we > > > > >> > >> > could > > > > >> > >> > > systematically create a new minor-patch release of > Parquet > > > format > > > > >> > with > > > > >> > >> > the > > > > >> > >> > > addition in preview or experimental depending where we're > > at. > > > > >> > >> > > The documentation would clarify whether a new optional > > field > > > is > > > > >> part > > > > >> > >> of > > > > >> > >> > the > > > > >> > >> > > official version yet or not. > > > > >> > >> > > That would decouple publishing versioned artifacts and > > > declaring > > > > >> > >> > something > > > > >> > >> > > part of the next official version. > > > > >> > >> > > (and independently of that we can either depend on a > tagged > > > > >> version > > > > >> > of > > > > >> > >> > the > > > > >> > >> > > file rather than the maven artifact) > > > > >> > >> > > > > > > >> > >> > > On Fri, Jul 24, 2026 at 7:42 AM Russell Spitzer < > > > > >> > >> > [email protected] > > > > >> > >> > > > > > > > >> > >> > > wrote: > > > > >> > >> > > > > > > >> > >> > > > +1 for all the reasons stated above. > > > > >> > >> > > > > > > > >> > >> > > > On Thu, Jul 23, 2026 at 10:21 PM Gang Wu < > > [email protected] > > > > > > > > >> > wrote: > > > > >> > >> > > > > > > > >> > >> > > > > I'm in favor of this proposal. > > > > >> > >> > > > > > > > > >> > >> > > > > PoCing a Java implementation in the current setup is > > > > >> painful. CI > > > > >> > >> > builds > > > > >> > >> > > > > will > > > > >> > >> > > > > always fail, and PRs cannot merge before releasing a > > new > > > > >> > >> > parquet-format > > > > >> > >> > > > > jar. > > > > >> > >> > > > > > > > > >> > >> > > > > Best, > > > > >> > >> > > > > Gang > > > > >> > >> > > > > > > > > >> > >> > > > > On Tue, Jul 21, 2026 at 8:46 PM Antoine Pitrou < > > > > >> > >> [email protected]> > > > > >> > >> > > > wrote: > > > > >> > >> > > > > > > > > >> > >> > > > > > Le 21/07/2026 à 14:05, Andrew Lamb a écrit : > > > > >> > >> > > > > > > As a maintainer of arrow-rs 's Parquet > > > implementation, my > > > > >> > >> > > experience > > > > >> > >> > > > > has > > > > >> > >> > > > > > > been that our copy of parquet.thrift (rather > than a > > > > >> > >> dependency) > > > > >> > >> > has > > > > >> > >> > > > > > worked > > > > >> > >> > > > > > > great, and we have never had an issue with this > > > scheme > > > > >> that > > > > >> > I > > > > >> > >> can > > > > >> > >> > > > > > remember > > > > >> > >> > > > > > > > > > >> > >> > > > > > We do the same in Arrow C++ and never had any > problem > > > with > > > > >> it > > > > >> > >> > either. > > > > >> > >> > > > > > > > > > >> > >> > > > > > Regards > > > > >> > >> > > > > > > > > > >> > >> > > > > > Antoine. > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > > > > > > Andrew > > > > >> > >> > > > > > > > > > > >> > >> > > > > > > On Tue, Jul 21, 2026 at 7:19 AM Divjot Arora via > > dev > > > < > > > > >> > >> > > > > > [email protected]> > > > > >> > >> > > > > > > wrote: > > > > >> > >> > > > > > > > > > > >> > >> > > > > > >> Hi folks, > > > > >> > >> > > > > > >> > > > > >> > >> > > > > > >> The parquet-java project currently declares a > > > dependency > > > > >> > on a > > > > >> > >> > > pinned > > > > >> > >> > > > > > >> version of parquet-format and dynamically pulls > in > > > the > > > > >> > >> > > > parquet.thrift > > > > >> > >> > > > > > file > > > > >> > >> > > > > > >> for the pinned version. > > > > >> > >> > > > > > >> This makes it impossible to validate and merge > PRs > > > to > > > > >> > support > > > > >> > >> > new > > > > >> > >> > > > > > features > > > > >> > >> > > > > > >> that have been merged into parquet-format but > are > > > not > > > > >> yet > > > > >> > >> part > > > > >> > >> > of > > > > >> > >> > > an > > > > >> > >> > > > > > >> official parquet-format > > > > >> > >> > > > > > >> release ( > > > > >> https://github.com/apache/parquet-java/pull/3610 > > > > >> > >> is an > > > > >> > >> > > > > > example). > > > > >> > >> > > > > > >> In contrast, the arrow-rs and arrow-cpp projects > > > have a > > > > >> > >> vendored > > > > >> > >> > > > copy > > > > >> > >> > > > > of > > > > >> > >> > > > > > >> parquet.thrift in their > > > > >> > >> > > > > > >> repos, which allows them to bypass this problem. > > > What do > > > > >> > >> folks > > > > >> > >> > > think > > > > >> > >> > > > > > about > > > > >> > >> > > > > > >> adopting this vendored approach in parquet-java? > > > > >> > >> > > > > > >> > > > > >> > >> > > > > > >> -- Div Arora > > > > >> > >> > > > > > >> > > > > >> > >> > > > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >
