[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685404#comment-13685404 ] Hudson commented on HADOOP-9517: Integrated in Hadoop-trunk-Commit #3951 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3951/]) HADOOP-9517. Documented various aspects of compatibility for Apache Hadoop. Contributed by Karthik Kambatla. (Revision 1493693) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493693 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/Compatibility.apt.vm * /hadoop/common/trunk/hadoop-project/src/site/site.xml Document Hadoop Compatibility - Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Priority: Blocker Fix For: 2.1.0-beta Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-v2.patch, hadoop-9517-v3.patch, hadoop-9517-v4.patch, hadoop-9517-v4-v5-diff.patch, hadoop-9517-v5.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685439#comment-13685439 ] Hudson commented on HADOOP-9517: Integrated in Hadoop-Yarn-trunk #243 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/243/]) HADOOP-9517. Documented various aspects of compatibility for Apache Hadoop. Contributed by Karthik Kambatla. (Revision 1493693) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493693 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/Compatibility.apt.vm * /hadoop/common/trunk/hadoop-project/src/site/site.xml Document Hadoop Compatibility - Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Priority: Blocker Fix For: 2.1.0-beta Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-v2.patch, hadoop-9517-v3.patch, hadoop-9517-v4.patch, hadoop-9517-v4-v5-diff.patch, hadoop-9517-v5.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685525#comment-13685525 ] Hudson commented on HADOOP-9517: Integrated in Hadoop-Hdfs-trunk #1433 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1433/]) HADOOP-9517. Documented various aspects of compatibility for Apache Hadoop. Contributed by Karthik Kambatla. (Revision 1493693) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493693 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/Compatibility.apt.vm * /hadoop/common/trunk/hadoop-project/src/site/site.xml Document Hadoop Compatibility - Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Priority: Blocker Fix For: 2.1.0-beta Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-v2.patch, hadoop-9517-v3.patch, hadoop-9517-v4.patch, hadoop-9517-v4-v5-diff.patch, hadoop-9517-v5.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685579#comment-13685579 ] Hudson commented on HADOOP-9517: Integrated in Hadoop-Mapreduce-trunk #1460 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1460/]) HADOOP-9517. Documented various aspects of compatibility for Apache Hadoop. Contributed by Karthik Kambatla. (Revision 1493693) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493693 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/Compatibility.apt.vm * /hadoop/common/trunk/hadoop-project/src/site/site.xml Document Hadoop Compatibility - Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Priority: Blocker Fix For: 2.1.0-beta Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-v2.patch, hadoop-9517-v3.patch, hadoop-9517-v4.patch, hadoop-9517-v4-v5-diff.patch, hadoop-9517-v5.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13684760#comment-13684760 ] Arun C Murthy commented on HADOOP-9517: --- I hope we can iterate on this quickly and commit fast since it's blocking 2.1.0-beta. I propose to commit this by Sun night PST if I don't see further comments - we can always file follow-on jiras. Thoughts? Document Hadoop Compatibility - Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Priority: Blocker Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-v2.patch, hadoop-9517-v3.patch, hadoop-9517-v4.patch, hadoop-9517-v4-v5-diff.patch, hadoop-9517-v5.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[
https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13684763#comment-13684763
]
Hadoop QA commented on HADOOP-9517:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588042/hadoop-9517-v5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2654//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2654//console
This message is automatically generated.
Document Hadoop Compatibility
-
Key: HADOOP-9517
URL: https://issues.apache.org/jira/browse/HADOOP-9517
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Reporter: Arun C Murthy
Assignee: Karthik Kambatla
Priority: Blocker
Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch,
hadoop-9517.patch, hadoop-9517-proposal-v1.patch,
hadoop-9517-proposal-v1.patch, hadoop-9517-v2.patch, hadoop-9517-v3.patch,
hadoop-9517-v4.patch, hadoop-9517-v4-v5-diff.patch, hadoop-9517-v5.patch
As we get ready to call hadoop-2 stable we need to better define 'Hadoop
Compatibility'.
http://wiki.apache.org/hadoop/Compatibility is a start, let's document
requirements clearly and completely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13684891#comment-13684891 ] Karthik Kambatla commented on HADOOP-9517: -- Thanks for chipping in, [~acmurthy]. The additions make sense to me. +1 to the v5 patch. Thoughts on binary compatibility: # Given that it is hard to guarantee source API compatibility in Java (to the extent that JDK itself doesn't), I wasn't sure if we should separately call out binary compatibility. # However, I really like the way ABI is described in the latest patch. It mentions all of API, wire and semantic compatibilities, and focuses on the practical implications for end-users. Thanks for including this. Document Hadoop Compatibility - Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Priority: Blocker Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-v2.patch, hadoop-9517-v3.patch, hadoop-9517-v4.patch, hadoop-9517-v4-v5-diff.patch, hadoop-9517-v5.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[
https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13684522#comment-13684522
]
Hadoop QA commented on HADOOP-9517:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12587997/hadoop-9517-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2652//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2652//console
This message is automatically generated.
Document Hadoop Compatibility
-
Key: HADOOP-9517
URL: https://issues.apache.org/jira/browse/HADOOP-9517
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Reporter: Arun C Murthy
Assignee: Karthik Kambatla
Priority: Blocker
Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch,
hadoop-9517.patch, hadoop-9517-proposal-v1.patch,
hadoop-9517-proposal-v1.patch, hadoop-9517-v2.patch, hadoop-9517-v3.patch,
hadoop-9517-v4.patch
As we get ready to call hadoop-2 stable we need to better define 'Hadoop
Compatibility'.
http://wiki.apache.org/hadoop/Compatibility is a start, let's document
requirements clearly and completely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13684528#comment-13684528 ] Vinod Kumar Vavilapalli commented on HADOOP-9517: - Still to review the doc, but one more thing that I just discovered is that the published site has apidocs which only show @Public APIs. This is different from regular javadoc that you can generate with mvn javadoc:javadoc or what your favourite IDE shows. Given that, it is confusing what is the source of truth of InterfaceAudience - Published javadoc on hadoop website or what is present in jars themselves. If it's the later, it isn't clear what the policy is for classes/interfaces with no annotations. Document Hadoop Compatibility - Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Priority: Blocker Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-v2.patch, hadoop-9517-v3.patch, hadoop-9517-v4.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680883#comment-13680883 ] Karthik Kambatla commented on HADOOP-9517: -- If no one has any comments against the newly proposed policies, I ll upload a new patch on Thursday with the (Proposal) tags removed. Document Hadoop Compatibility - Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Priority: Blocker Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-v2.patch, hadoop-9517-v3.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674683#comment-13674683 ] Karthik Kambatla commented on HADOOP-9517: -- bq. Classes that subclass a Hadoop class to provide a plugin point MAY need recompiling on each major version, possibly with the handling of changes to methods. If the Hadoop class being extended is Public and the stability is defined by the annotation, is that not sufficient indication to the user that it might need to be changed as that interface/class changes. For example, we recently added a SchedulingPolicy to FairScheduler annotated Public-Evolving: the policies written for the current version need to be updated as and when the SchedulingPolicy class changes. Once it becomes stable, we follow the standard deprecation rules for Public-Stable API that protects the policies. No? I think it is important to detail how the API compatibility rules impact the user-level code. May be I am missing something here. Otherwise, we might not need specific policies for them? Document Hadoop Compatibility - Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-proposal-v1.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675461#comment-13675461 ] Sanjay Radia commented on HADOOP-9517: -- bq. ... Optional fields can be added any time .. Fields can be renamed any time ... This is what stable means. Hence i suggest that we add a comment to the .proto files to say that the .protos are private-stable, and we could add the comment on kinds of changes allowed. Will file a jira for this. Given that are no annotations for .proto, a comment is the best that can be done. bq. Don't you think the proto files for client interfaces should be public? I was chatting with Todd about this, and it seems to us they should. I would still mark it as private till we make the rpc and data transfer protocol itself public (ie the protos being public is useless without the rpc proto being public. Todd and I occasionally disagree ;-) Document Hadoop Compatibility - Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-v2.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[
https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13672050#comment-13672050
]
Steve Loughran commented on HADOOP-9517:
As raised in hdfs-dev, we should also define compatibility for people who
develop plugins for things like filesystems, schedulers, sorting, block
placement, etc.
We should make clear that this isn't considered user-level code, and
implementors should have no expectation of forward- or backward- compatibility.
Indeed, the use of base classes rather than interfaces for many of these plugin
points is to enable new methods to be added with ease.
How about something like (based on my recent {{FileSystem}} work):
* Classes that subclass a Hadoop class to provide a plugin point MAY need
recompiling on each major version, possibly with the handling of changes to
methods.
* There is no guarantee that a plugin built for an earlier major version of
Hadoop can be directly used in a later major version - implementors of plugins
MUST expect to have to release new versions
* There is no guarantee that a plugin built on a later major version of Hadoop
can be directly used in the earlier major versions. Implementors of plugins
MUST expect to have to maintain different versions for each major release.
* We strive to maintain semantic compatibility of existing parameters and
methods across major versions. Plugin points MAY tighten their specifications,
and MAY mark methods as deprecated. Extra tests MAY be added, for both new
features and tightened specifications.
* We strive to maintain binary and semantic compatibility between minor
releases. Hopefully plugin implementations are compatible across minor
releases. However, plugin points MAY tighten their specifications -and even add
extra methods and/or enumerated options to existing methods. Extra tests MAY be
added, for both new features and tightened specifications. Implementors SHOULD
test on different releases -especially alpha and beta releases, to catch
compatibility issues early.
Document Hadoop Compatibility
-
Key: HADOOP-9517
URL: https://issues.apache.org/jira/browse/HADOOP-9517
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Reporter: Arun C Murthy
Assignee: Karthik Kambatla
Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch,
hadoop-9517.patch, hadoop-9517-proposal-v1.patch,
hadoop-9517-proposal-v1.patch
As we get ready to call hadoop-2 stable we need to better define 'Hadoop
Compatibility'.
http://wiki.apache.org/hadoop/Compatibility is a start, let's document
requirements clearly and completely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671816#comment-13671816 ] Sanjay Radia commented on HADOOP-9517: -- bq. Shouldn't the proto files themselves be classified as public and stable? If you mean the wire protocol proto files, then no. They are not public as we have not made the protocol itself public; we may do that at some point and at that time we would make them public. At this stage we have merely promised to maintain wire compatibility going forward. Right now the proto files should be marked as private-stable. Document Hadoop Compatibility - Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-proposal-v1.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671821#comment-13671821 ] Sanjay Radia commented on HADOOP-9517: -- bq. I vote to strengthen the compatibility requirements for user data file formats. ... Rather we'd like to permit both writers or readers of data files to be upgraded independently. ... Agreed. What do you mean by user data file formats? Do you mean data files that are processed by libraries in user land (as opposed to a server). Would har files, sequence files and RC files be such user data file formats? To ground this better, Doug could you please give some examples of kinds of updates one would want to do and of those which would be allowed and which would not. Document Hadoop Compatibility - Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-proposal-v1.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671838#comment-13671838 ] Alejandro Abdelnur commented on HADOOP-9517: [~sanjay.radia], on the proto files being public or not. Don't you think the proto files for client interfaces should be public? I was chatting with Todd about this, and it seems to us they should. On a different note, we should document we can change name of fields in proto files if we keep the ID/type/required|optional. Document Hadoop Compatibility - Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-proposal-v1.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671846#comment-13671846 ] Karthik Kambatla commented on HADOOP-9517: -- It might make sense to make certain (non-internal) proto files public, and leave the internal (RM-NM) protos private. Also, I think the policy on changes to proto files should be: # Optional fields can be added any time # Fields can be renamed any time # Required fields can't be added within a major release # Field order and type can't be modified within a major release Document Hadoop Compatibility - Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-proposal-v1.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Document Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13670509#comment-13670509 ] Sanjay Radia commented on HADOOP-9517: -- bq. wrt HDFS upgrade the current prose is not clear whether we permit HDFS metadata upgrades w/in minor release or not.. and explicitly say that metadata upgrades may only be required for major version upgrades? Agreed that it is not clear. Given that the conversion is automatic, do we want to allow metadata *AND data* changes in minor releases ? Or are we trying to say that rolling upgrades are always possible in minor releases and hence we don't want to allow metadata and data changes? Note although we have not figured out how to do rolling upgrades when there are metadata changes, it may be possible to do so. BTW -upgrade is often used even when there are no metadata or data changes as a safety measure. bq. Let's clearly distinguish between HDFS upgrades (ie just upgrading the HDFS bits) from an HDFS metadata upgrade, Given that this is in the data section I thought it was obvious. Lets modify the Data Section to start by saying that we are not talking about the executable bits. bq. Also, what does automatic conversion mean, that the HDFS metadata upgrade process can automatically convert the old version to the new? As opposed to requiring a user manually perform multiple such upgrades? Doesn't the word automatic clarify this? Again suggest some text improvements and we can put it in. Document Hadoop Compatibility - Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517-proposal-v1.patch, hadoop-9517-proposal-v1.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
