/* will leave my comments in the JIRA as well */ the mechanism proposed here - by itself - will not prevent regressions. we can create a 'registry' of legal options - but that will not prevent a software change that adds a new option, moves semantics of old option to the new option - and then merrily leaves the old option behind in the registry.
(exactly what happened here. the tree has good looking code with references to deprecated options). we would need isolated regression tests against each registered option (tall order indeed) for this to suffice. besides - even the Java interfaces were obsoleted or their semantics changed without removing old interfaces. and that's not a 'lack of mechanism' issue - it's a pure software engineering issue. -- can't complain too much though - it's one or two errant changes of many good ones. thanks to the hadoop team for all the good work! hope the team can articulate and adopt a strict policy of introducing backwards incompatibility when interfaces semantics change. -----Original Message----- From: Aaron Kimball [mailto:[EMAIL PROTECTED] Sent: Wed 2/20/2008 8:13 PM To: core-user@hadoop.apache.org Subject: Re: changes to compression interfaces in 0.15? I filed a JIRA for this issue: https://issues.apache.org/jira/browse/HADOOP-2866 - Aaron Aaron Kimball wrote: > +1 > > Ted Dunning wrote: >> Actually, it might just be good to have a warning spit out if you use ANY >> unknown key that starts with mapred.* or any of the other hadoop-specific >> parameters. >> >> That way mis-spellings would be caught as well as deprecations. >> >> If you want to set a value and not get a warning, just pick a different >> prefix. >> >> On 2/20/08 6:13 PM, "Jason Venner" <[EMAIL PROTECTED]> wrote: >> >>> I agree. I am in the midst of combing through the config files for 16 to >>> see what changes i have to retrofit into our jobs. >>> Support in the tools to inform of the use of depreciated or outright >>> removed keys would be wonderful. >>> >>> Aaron Kimball wrote: >>>> As a general follow-up suggestion : Is there a mechanism to output a >>>> warning when the user sets deprecated JobConf keys? Given that you can >>>> set any arbitrary key name and it will simply be ignored, this might >>>> be a good idea. >>>> >>>> - Aaron >>>> >>>> Joydeep Sen Sarma wrote: >>>>> In addition: >>>>> >>>>> - "mapred.output.compression.type" is now replaced with >>>>> "mapred.map.output.compression.type" >>>>> >>>>> - the old implementation of the Java interface >>>>> setMapOutputCompressorClass() used to turn on map compression on >>>>> automatically as side-effect, the 0.15 one doesn't. Looks like one has >>>>> to call setCompressMapOutput() separately. >>>>> >>>>> >>>>> >>>>> Aargh. >>>>> >>>>> >>>>> >>>>> ________________________________ >>>>> >>>>> From: [EMAIL PROTECTED] >>>>> [mailto:[EMAIL PROTECTED] On Behalf Of Joydeep >>>>> Sen >>>>> Sarma >>>>> Sent: Wednesday, February 20, 2008 5:06 PM >>>>> To: core-user@hadoop.apache.org >>>>> Subject: changes to compression interfaces in 0.15? >>>>> >>>>> >>>>> >>>>> Hi developers, >>>>> >>>>> >>>>> >>>>> In migrating to 0.15 - i am noticing that the compression interfaces >>>>> have changed: >>>>> >>>>> >>>>> >>>>> - compression type for sequencefile outputs used to be set >>>>> by: >>>>> SequenceFile.setCompressionType() >>>>> >>>>> - now it seems to be set using: >>>>> sequenceFileOutputFormat.setOutputCompressionType() >>>>> >>>>> >>>>> >>>>> The change is for the better - but would it be possible to: >>>>> >>>>> >>>>> >>>>> - remove old/dead interfaces. That would have been a >>>>> straightforward hint for applications to look for new interfaces. >>>>> (hadoop-default.xml also still has setting for old conf variable: >>>>> io.seqfile.compression.type) >>>>> >>>>> - if possible - document changed interfaces in the release >>>>> notes (there's no way we can find this out by looking at the long list >>>>> of Jiras). >>>>> >>>>> >>>>> >>>>> As u can imagine - this causes a very subtle and harmful regression in >>>>> behavior of existing apps. It does not causes failures - and in our >>>>> case >>>>> - switched from BLOCK to RECORD compression - meaning - there's no >>>>> compression at all pretty much. I caught this by *pure* chance and >>>>> now I >>>>> am living in absolute fear of what else lurks out there. >>>>> >>>>> >>>>> >>>>> i am not sure how updated the wiki is on the compression stuff (my >>>>> responsibility to update it) - but please do consider the impact of >>>>> changing interfaces on existing applications. (maybe we should have a >>>>> JIRA tag to mark out bugs that change interfaces). >>>>> >>>>> >>>>> >>>>> As always - thanks for all the fish (err .. working code), >>>>> >>>>> >>>>> >>>>> Joydeep >>>>> >>>>> >>>>> >>>>> >>
