Re: [DISCUSS] Moving the format directory to arrow-format repository

2021-05-02 Thread Neville Dipale
Hi, Thanks for the feedback, in light of what's been said, I'm also now fine with leaving the format as is. Changes to the format are visible enough that we shouldn't miss them, as there's normally be a discussion in the ML. Regards Neville On Wed, 28 Apr 2021 at 17:31, Jorge Cardoso Leitão

Re: [DISCUSS] Moving the format directory to arrow-format repository

2021-04-28 Thread Jorge Cardoso Leitão
Hi, imo the time-scale of changes in the format is too large to justify the complexity. I also think that we should not force users to clone or submodule the repo to even compile the crate. What if we just do not have the format files there at all, and instead just keep the generated code?

Re: [DISCUSS] Moving the format directory to arrow-format repository

2021-04-28 Thread Andrew Lamb
I also think manually copying the format .fbs files to arrow-rs is probably ok for the time being. Once Arrow gets to the point where many implementations that need format.fbs live in many different repos, pulling out the format files into their own repo might be worth reconsidering. Andrew On

Re: [DISCUSS] Moving the format directory to arrow-format repository

2021-04-27 Thread Wes McKinney
I wouldn't be too excited about this. Here are my thoughts: 1. Having the format/ directory in apache/arrow be a submodule would be cumbersome and error-prone for developers. The only submodules we have right now are optional testing dependencies — not having these initialized and updated does

[DISCUSS] Moving the format directory to arrow-format repository

2021-04-27 Thread Neville Dipale
Hi Arrow devs, Andy noticed that we carry a copy of the format directory in arrow-rs, which is bound to get outdated in the future. We would like to propose creating an arrow-format repository, similar to parquet-format, so that arrow-rs and other future separate repositories could add this as a