On Tue, Feb 3, 2015 at 11:01 AM, Burak Emre <[email protected]> wrote:
> @Sean thanks for the explanation. > > I have multiple writers but only one reader and the only schema migration > operation is adding a new field so I thought that I may use the same schema > for all dataset since the ordering will be same in all of them even though > some may contain extra fields which is also defined in schema definition. > > Actually I wanted to avoid using an external database for sequential > schema ids since it would make the system more complex than it should be in > my case but it seems this is the only option for now. > > > An external database isn't strictly required. The only important bit is that each schema have a unique immutable identifier. As Doug mentioned, you could do this as an enum of schemas in your source code (so long as you handled updates in reader-then-writer order). Similarly, you could do it by relying on schema fingerprints and just loading avsc files out of shared storage. -- Sean
