Hi Tim, Unfortunately, Drill has no version compatibility guidelines. We break compatibility all the time — often by accident. As Drill matures, we should consider defining such a policy.
Experience with other systems suggests that provide an automatic way to handle the inevitable evolution of APIs, data formats and the like. This is done for user convenience, and (in commercial products) to avoid support calls. Would have been great if we had a version number in ZK. Would solve problems not just with options, but with storage plugins when we change their classes (and hence the JSON stored in ZK.) But, we don’t… Vitalii recently added versioning for the Parquet metadata file to avoid the need for users to delete all the metadata each time we make a change. Would be great if we could follow that example for other areas. In the meantime, perhaps you can implement a way to read the old format ZK but write the new format. I believe you also changed the layout of the system table for options. These were long-needed improvements. Still, I wonder if anyone has a script that depends on the old format? Do we need a way to support the old format, while offering a new table with the new format? Jyothsna recently did that as part of her work in options; I wonder if something like that is needed here also? In fact, you may be able to simply alter the table that she added: it hasn’t see the light of a Drill release yet. Thanks, - Paul > On Sep 21, 2017, at 4:16 PM, Timothy Farkas <tfar...@mapr.com> wrote: > > Makes sense Abhishek, I'll work on making it backwards compatible then. > > Thanks, > Tim > > ________________________________ > From: Abhishek Girish <agir...@apache.org> > Sent: Thursday, September 21, 2017 3:58:55 PM > To: dev@drill.apache.org > Subject: Re: Backwards Compatibility Policy > > Hey Tim, > > Requiring users to purge Drill's ZK data is not advisable and we might not > want to go that route. We need to have a seamless upgrade path - for > instance modifying values found to be in an older format to the new one, > without explicit user interaction. > > Regards, > Abhishek > > On Thu, Sep 21, 2017 at 3:46 PM, Timothy Farkas <tfar...@mapr.com> wrote: > >> Hi All, >> >> I recently made a change to the option system which impacted the fields >> contained in OptionValues and hence the format of option information we are >> storing in zookeeper. So it is currently not backward compatible with old >> system options stored in zookeeper. Two ways to resolve the issue are to >> require old data to be purged from zookeeper when upgrading the cluster or >> to attempt to allow backward compatibility by modifying the deserializer >> for OptionValue. So my question is what is our stance on backwards >> compatibility? >> >> Thanks, >> Tim >>
