Re: [DISCUSS] Making 2.10 the last minor 2.x release
Some other additional items we would need: - Mark all fix-versions in YARN/HDFS/MAPREDUCE/HADOOP from 2.11.0 to 2.10.1 - Remove 2.11.0 as a version in these projects Jonathan Hung On Thu, Nov 14, 2019 at 6:51 PM Jonathan Hung wrote: > Hi folks, > > Given the release of 2.10.0, and the fact that it's intended to be a > bridge release to Hadoop 3.x [1], I'm proposing we make 2.10.x the last > minor release line in branch-2. Currently, the main issue is that there's > many fixes going into branch-2 (the theoretical 2.11.0) that's not going > into branch-2.10 (which will become 2.10.1), so the fixes in branch-2 will > likely never see the light of day unless they are backported to branch-2.10. > > To do this, I propose we: > >- Delete branch-2.10 >- Rename branch-2 to branch-2.10 >- Set version in the new branch-2.10 to 2.10.1-SNAPSHOT > > This way we get all the current branch-2 fixes into the 2.10.x release > line. Then the commit chain will look like: trunk -> branch-3.2 -> > branch-3.1 -> branch-2.10 -> branch-2.9 -> branch-2.8 > > Thoughts? > > Jonathan Hung > > [1] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg29479.html >
[DISCUSS] Making 2.10 the last minor 2.x release
Hi folks, Given the release of 2.10.0, and the fact that it's intended to be a bridge release to Hadoop 3.x [1], I'm proposing we make 2.10.x the last minor release line in branch-2. Currently, the main issue is that there's many fixes going into branch-2 (the theoretical 2.11.0) that's not going into branch-2.10 (which will become 2.10.1), so the fixes in branch-2 will likely never see the light of day unless they are backported to branch-2.10. To do this, I propose we: - Delete branch-2.10 - Rename branch-2 to branch-2.10 - Set version in the new branch-2.10 to 2.10.1-SNAPSHOT This way we get all the current branch-2 fixes into the 2.10.x release line. Then the commit chain will look like: trunk -> branch-3.2 -> branch-3.1 -> branch-2.10 -> branch-2.9 -> branch-2.8 Thoughts? Jonathan Hung [1] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg29479.html
[jira] [Created] (HADOOP-16712) Config ha.failover-controller.active-standby-elector.zk.op.retries is not in core-default.xml
Wei-Chiu Chuang created HADOOP-16712: Summary: Config ha.failover-controller.active-standby-elector.zk.op.retries is not in core-default.xml Key: HADOOP-16712 URL: https://issues.apache.org/jira/browse/HADOOP-16712 Project: Hadoop Common Issue Type: Improvement Reporter: Wei-Chiu Chuang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16711) With S3Guard + authmode, consider skipping "verifyBuckets" check in S3A fs init()
Rajesh Balamohan created HADOOP-16711: - Summary: With S3Guard + authmode, consider skipping "verifyBuckets" check in S3A fs init() Key: HADOOP-16711 URL: https://issues.apache.org/jira/browse/HADOOP-16711 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Rajesh Balamohan When authoritative mode is enabled with s3guard, it would be good to skip verifyBuckets call during S3A filesystem init(). This would save call to S3 during init method. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: Questions about Apache Hadoop
Please note that Hadoop is essentially an on-prem data infrastructure these days. To fully leverage Hadoop you typically run other compute engines such as Hive, Spark, HBase and so on. So YMMV. Folks on the mailing list, please feel free to add/correct my comments. • What type of secure connection will there be between Apache Hadoop and VA systems in terms of secure protocols implemented? Hadoop implements SSL/TLS up to TLS 1.2 as of Hadoop 3.2 for HTTP-based connections. SPNEGO is supported too. Separately, Hadoop's RPC protocol leverages Java SASL APIs to authenticate/encrypt. Kerberos is supported via Java SASL. Hadoop's data transfer leverages OpenSSL for data encryption. There are additional custom protocols (Hadoop Delegation Token) that authenticates users in a cluster. • To what extent does Apache Hadoop use a FIPS 140-2 validated cryptographic module, and what is the certification number? Hadoop KMS is the cryptographic module in Hadoop. As far as I know it is not FIPS 140-2 certified. You may need a commercial vendor for that purpose. • What is the most recent version of Apache Hadoop and its release date? We maintain several branches. https://hadoop.apache.org/releases.html • Is there a Voluntary Product Accessibility Template (VPAT) program in place to assess Section 508 compliance? I looked it up at Wikipedia and I still don't know what this is. You may need to hire a consultant to help. I suspect we don't, otherwise we wouldn't have used light green as the header of HDFS NameNode web UI. • What are the main features of Apache Hadoop? https://hadoop.apache.org/ The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing. The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures. • What Cloud Service Provider (CSP) agreements have been set for Apache Hadoop to be used securely through the cloud? Apache Hadoop is an open source project. You need a commercial vendor to answer this question for you. • Does Apache Hadoop offer an Application Program Interface (API)? Yes -- we offer Java API, C API and RESTFUL API. • What other apps does Apache Hadoop integrate with? Many -- HBase, Hive, Spark, Presto and many others. Search for "Hadoop ecosystem" • What level of support does Apache Hadoop offer? The project itself provides community support. Commercial support is available via commercial vendors. Cloudera, for example, offers several levels of technical support and services. • Does Apache Hadoop leverage other database products? No. However, if you use Hive on top of Hadoop, Hive requires a metastore that runs on a database. • Is Apache Hadoop available for on-premise deployment? Yes. • Does Apache Hadoop reside on user network? Not sure what this means. Thank you for your willingness to help by answering these questions. Please note that I am working on a tight deadline, and must have my research completed within three business days. If you would please acknowledge your initial receipt of my email, it would be greatly appreciated. Please contact me if you have any questions or concerns. Best Regards, Foday B. Fofanah (Contractor) Senior Security Analyst (Prosphere) Solution Delivery (Station 116) (005OPB14) Office of Information and Technology, IT Operations and Services (ITOPS) Office: (202) 461-4424 On Wed, Nov 13, 2019 at 2:07 PM Fofanah, Foday B. (Prosphere) wrote: > Hello, > > I am reaching out to you from the Department of Veterans Affairs (VA) > where I am part of the team that reviews various information-based products > from an information security perspective for use within VA. I am reviewing > information regarding Apache Hadoop and have several questions listed > below; please respond to the best of your ability so that I may use your > answers to reach a final determination. > > • What type of secure connection will there be between Apache > Hadoop and VA systems in terms of secure protocols implemented? > > > > • To what extent does Apache Hadoop use a FIPS 140-2 validated > cryptographic module, and what is the certification number? > > > > • What is the most recent version of Apache Hadoop and its > release date? > > > > • Is there a Voluntary Product Accessibility Template (VPAT) >
[jira] [Created] (HADOOP-16710) testing_azure.md documentation is misleading
Andras Bokor created HADOOP-16710:
-
Summary: testing_azure.md documentation is misleading
Key: HADOOP-16710
URL: https://issues.apache.org/jira/browse/HADOOP-16710
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure
Affects Versions: 3.2.0
Reporter: Andras Bokor
Assignee: Andras Bokor
testing_azure.md states that "-Dparallel-tests" will run all the integration
tests in parallel.
But in fact using -Dparallel-tests without any value actually skips the
integration tests and runs only the unit tests.
The reason is that to activate a profile which is able to run ITs in parallel
requires parallel-tests property to have a value (abfs, wasb or 'both').
sequential-tests profile defines !parallel-tests as value which means that the
property should not even be mentioned.
Please check the output of help:active-profiles command:
{code:java}
cd hadoop-tools/hadoop-azure
andrasbokor$ mvn help:active-profiles -Dparallel-tests=abfs
- parallel-tests-abfs (source: org.apache.hadoop:hadoop-azure:3.3.0-SNAPSHOT)
- os.mac (source: org.apache.hadoop:hadoop-project:3.3.0-SNAPSHOT)
- hbase1 (source: org.apache.hadoop:hadoop-project:3.3.0-SNAPSHOT) {code}
{code:java}
andrasbokor$ mvn help:active-profiles -Dparallel-tests
- os.mac (source: org.apache.hadoop:hadoop-project:3.3.0-SNAPSHOT)
- hbase1 (source: org.apache.hadoop:hadoop-project:3.3.0-SNAPSHOT)
{code}
{code:java}
mvn help:active-profiles
- sequential-tests (source: org.apache.hadoop:hadoop-azure:3.3.0-SNAPSHOT)
- os.mac (source: org.apache.hadoop:hadoop-project:3.3.0-SNAPSHOT)
- hbase1 (source: org.apache.hadoop:hadoop-project:3.3.0-SNAPSHOT){code}
The help:active-profiles shows that -Dparallel-tests does not add any IT
related profiles and results in skipping all the integration tests during
verify phrase.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[ https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal reopened HADOOP-16683: - Let's backport this issue to lower branches. > Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped > AccessControlException > -- > > Key: HADOOP-16683 > URL: https://issues.apache.org/jira/browse/HADOOP-16683 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16683.001.patch, HADOOP-16683.002.patch, > HADOOP-16683.003.patch > > > Follow up patch on HADOOP-16580. > We successfully disabled the retry in case of an AccessControlException which > has resolved some of the cases, but in other cases AccessControlException is > wrapped inside another IOException and you can only get the original > exception by calling getCause(). > Let's add this extra case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/ No changes -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:[line 335] Failed junit tests : hadoop.util.TestReadWriteDiskValidator hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.TestDistributedFileSystem hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.registry.secure.TestSecureLogins hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 hadoop.yarn.client.api.impl.TestAMRMClient cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt [328K] cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/diff-compile-cc-root-jdk1.8.0_222.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/diff-compile-javac-root-jdk1.8.0_222.txt [308K] checkstyle: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/diff-checkstyle-root.txt [16M] hadolint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/diff-patch-shellcheck.txt [72K] shelldocs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/diff-patch-shelldocs.txt [8.0K] whitespace: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/whitespace-tabs.txt [1.3M] xml: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/xml.txt [12K] findbugs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/diff-javadoc-javadoc-root-jdk1.7.0_95.txt [16K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/diff-javadoc-javadoc-root-jdk1.8.0_222.txt [1.1M] unit: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [160K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [232K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/505/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K]
