On Sat, Dec 31, 2011 at 7:38 AM, Ralph Goers <[email protected]>wrote:
> I guess I don't see completely how that can work. In OG the event > contained getTimestamp(), getNanos() and getHost(). The Event in OG does > not have these fields. These accessors were just sugar for poking things into the headers of the event. The equivalent would be to explicitly stash these attributes in the headers of the event so they're available down the line (i.e. in the sink to which they're sent). > We have an application that is reading legacy log data, creating an OG > event and passing it through Flume to a custom Sink to write to Cassandra. Sources and sinks obviously need to be updated to the new APIs at which time they should use headers directly. > In addition, we will have a bunch of apps that use an Appender to create > an OG Event which will pass it to an agent to forward to the same Cassandra > sink. I'm trying to figure out how we can do the upgrade to NG in a sane > fashion. > The log4j / logback appender case is a bit trickier. Basically, you do need to update the appender and the agent to which it writes at the same time or have it speak to the proposed compatibility source in an NG agent which knows how to deal with OG thrift events. I plan on writing up a compatibility proposal on the wiki as a strawman. If that makes sense, we'll render it out in JIRAs and run with it. If it doesn't, we'll work together to figure out a better plan. I was hoping to get away without spending a lot of time on compatibility with OG, but if the community as a whole feels phased upgrades are going to be critical I'm happy to help work out how it might go. > Ralph > > On Dec 31, 2011, at 1:30 AM, Eric Sammer wrote: > > > I've been running through some of the compatibility scenarios and while I > > haven't spent a ton of time on it, I think there's a solution. It's true > > that the event object isn't the same as OG. That would have been bulky as > > we would have had to include thrift just to accommodate it (NG's event is > > just a pojo to prevent RPC bleed throughout the codebase). Since users > can > > only upgrade an entire agent at once - that is, run an OG agent and > connect > > it to an NG agent or vice versa - we could introduce a compatibility > source > > and sink for NG and handle the marshaling from OG events to NG events. > > > > I don't know if that fully allays your concern, but it should get us to > > where we need to be. > > > > On Sat, Dec 31, 2011 at 12:04 AM, Ralph Goers < > [email protected]>wrote: > > > >> Was it really necessary to make the Event object in NG backward > >> incompatible. This will make upgrading systems very painful as > everything > >> needs to be upgraded at once. > >> > >> Ralph > > > > > > > > > > -- > > Eric Sammer > > twitter: esammer > > data: www.cloudera.com > > -- Eric Sammer twitter: esammer data: www.cloudera.com
