This is an automated email from the ASF dual-hosted git repository. vinayakumarb pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git
The following commit(s) were added to refs/heads/trunk by this push: new fd78dcf HADOOP-16595. [pb-upgrade] Create hadoop-thirdparty artifact to have shaded protobuf. Contributed by Vinayakumar B. (#1) fd78dcf is described below commit fd78dcf990adde4d09dc1c9dfbf46a83f710027b Author: Vinayakumar B <vinayakum...@apache.org> AuthorDate: Mon Jan 13 10:56:24 2020 +0530 HADOOP-16595. [pb-upgrade] Create hadoop-thirdparty artifact to have shaded protobuf. Contributed by Vinayakumar B. (#1) --- .github/pull_request_template.md | 6 + .gitignore | 9 + LICENSE-binary | 241 +++++++ LICENSE.txt | 224 ++++++ NOTICE-binary | 780 +++++++++++++++++++++ NOTICE.txt | 34 + dev-support/bin/create-release | 641 +++++++++++++++++ dev-support/bin/releasedocmaker | 18 + dev-support/bin/yetus-wrapper | 188 +++++ dev-support/docker/Dockerfile | 219 ++++++ dev-support/docker/hadoop_env_checks.sh | 117 ++++ hadoop-shaded-protobuf_3_7/pom.xml | 115 +++ licenses-binary/LICENSE-protobuf.txt | 32 + pom.xml | 438 ++++++++++++ .../resources/assemblies/hadoop-thirdparty-src.xml | 62 ++ src/site/markdown/index.md.vm | 45 ++ src/site/resources/css/site.css | 30 + src/site/site.xml | 59 ++ 18 files changed, 3258 insertions(+) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000..2b5014b --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,6 @@ +## NOTICE + +Please create an issue in ASF JIRA before opening a pull request, +and you need to set the title of the pull request which starts with +the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.) +For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..ed49e7c --- /dev/null +++ b/.gitignore @@ -0,0 +1,9 @@ +.idea +**/target/* +*.patch +*.iml +.project +.classpath +.settings +patchprocess +**/dependency-reduced-pom.xml diff --git a/LICENSE-binary b/LICENSE-binary new file mode 100644 index 0000000..6c668ef --- /dev/null +++ b/LICENSE-binary @@ -0,0 +1,241 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +-------------------------------------------------------------------------------- +This project bundles some components that are also licensed under the Apache +License Version 2.0: + + + +-------------------------------------------------------------------------------- +This product bundles various third-party components under other open source +licenses. This section summarizes those components and their licenses. +See licenses-binary/ for text of these licenses. + + +BSD 2-Clause +------------ + + +BSD 3-Clause +------------ + +com.google.protobuf:protobuf-java:3.7.1 + + +MIT License +----------- + + +CDDL 1.1 + GPLv2 with classpath exception +----------------------------------------- + +Eclipse Public License 1.0 +-------------------------- + +junit:junit:4.12 + + +Public Domain +------------- + diff --git a/LICENSE.txt b/LICENSE.txt new file mode 100644 index 0000000..643e3e7 --- /dev/null +++ b/LICENSE.txt @@ -0,0 +1,224 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +-------------------------------------------------------------------------------- +This product bundles various third-party components under other open source +licenses. This section summarizes those components and their licenses. +See licenses/ for text of these licenses. + + +Apache Software Foundation License 2.0 +-------------------------------------- + + +BSD 2-Clause +------------ + + +BSD 3-Clause +------------ + + +MIT License +----------- + diff --git a/NOTICE-binary b/NOTICE-binary new file mode 100644 index 0000000..8268c52 --- /dev/null +++ b/NOTICE-binary @@ -0,0 +1,780 @@ +Apache Hadoop +Copyright 2006 and onwards The Apache Software Foundation. + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +Export Control Notice +--------------------- + +This distribution includes cryptographic software. The country in +which you currently reside may have restrictions on the import, +possession, use, and/or re-export to another country, of +encryption software. BEFORE using any encryption software, please +check your country's laws, regulations and policies concerning the +import, possession, or use, and re-export of encryption software, to +see if this is permitted. See <http://www.wassenaar.org/> for more +information. + +The U.S. Government Department of Commerce, Bureau of Industry and +Security (BIS), has classified this software as Export Commodity +Control Number (ECCN) 5D002.C.1, which includes information security +software using or performing cryptographic functions with asymmetric +algorithms. The form and manner of this Apache Software Foundation +distribution makes it eligible for export under the License Exception +ENC Technology Software Unrestricted (TSU) exception (see the BIS +Export Administration Regulations, Section 740.13) for both object +code and source code. + +The following provides more details on the included cryptographic software: + +This software uses the SSL libraries from the Jetty project written +by mortbay.org. +Hadoop Yarn Server Web Proxy uses the BouncyCastle Java +cryptography APIs written by the Legion of the Bouncy Castle Inc. + +// ------------------------------------------------------------------ +// NOTICE file corresponding to the section 4d of The Apache License, +// Version 2.0, in this case for +// ------------------------------------------------------------------ + + +Apache Yetus +Copyright 2008-2017 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +--- +Additional licenses for the Apache Yetus Source/Website: +--- + + +See LICENSE for terms. + + + +Apache Avro +Copyright 2010 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +C JSON parsing provided by Jansson and +written by Petri Lehtinen. The original software is +available from http://www.digip.org/jansson/. + + +********************** +THIRD PARTY COMPONENTS +********************** +This software includes third party software subject to the following copyrights: +- XML parsing and utility functions from JetS3t - Copyright 2006-2009 James Murty. +- PKCS#1 PEM encoded private key parsing and utility functions from oauth.googlecode.com - Copyright 1998-2010 AOL Inc. + +The licenses for these third party components are included in LICENSE.txt + + +Apache Commons BeanUtils +Copyright 2000-2016 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons CLI +Copyright 2001-2009 The Apache Software Foundation + +This product includes software developed by +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons Codec +Copyright 2002-2017 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java +contains test data from http://aspell.net/test/orig/batch0.tab. +Copyright (C) 2002 Kevin Atkinson (kev...@gnu.org) + +=============================================================================== + +The content of package org.apache.commons.codec.language.bm has been translated +from the original php source code available at http://stevemorse.org/phoneticinfo.htm +with permission from the original authors. +Original source copyright: +Copyright (c) 2008 Alexander Beider & Stephen P. Morse. + + +Apache Commons Collections +Copyright 2001-2018 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons Compress +Copyright 2002-2018 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (https://www.apache.org/). + +The files in the package org.apache.commons.compress.archivers.sevenz +were derived from the LZMA SDK, version 9.20 (C/ and CPP/7zip/), +which has been placed in the public domain: + +"LZMA SDK is placed in the public domain." (http://www.7-zip.org/sdk.html) + + +Apache Commons Configuration +Copyright 2001-2017 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons CSV +Copyright 2005-2014 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +Apache Commons Daemon +Copyright 1999-2013 The Apache Software Foundation + +This product includes software developed by +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons Digester +Copyright 2001-2008 The Apache Software Foundation + +This product includes software developed by +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons IO +Copyright 2002-2016 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons Lang +Copyright 2001-2017 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +This product includes software from the Spring Framework, +under the Apache License 2.0 (see: StringUtils.containsWhitespace()) + + +Apache Commons Logging +Copyright 2003-2013 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons Math +Copyright 2001-2012 The Apache Software Foundation + +This product includes software developed by +The Apache Software Foundation (http://www.apache.org/). + +=============================================================================== + +The BracketFinder (package org.apache.commons.math3.optimization.univariate) +and PowellOptimizer (package org.apache.commons.math3.optimization.general) +classes are based on the Python code in module "optimize.py" (version 0.5) +developed by Travis E. Oliphant for the SciPy library (http://www.scipy.org/) +Copyright © 2003-2009 SciPy Developers. +=============================================================================== + +The LinearConstraint, LinearObjectiveFunction, LinearOptimizer, +RelationShip, SimplexSolver and SimplexTableau classes in package +org.apache.commons.math3.optimization.linear include software developed by +Benjamin McCann (http://www.benmccann.com) and distributed with +the following copyright: Copyright 2009 Google Inc. +=============================================================================== + +This product includes software developed by the +University of Chicago, as Operator of Argonne National +Laboratory. +The LevenbergMarquardtOptimizer class in package +org.apache.commons.math3.optimization.general includes software +translated from the lmder, lmpar and qrsolv Fortran routines +from the Minpack package +Minpack Copyright Notice (1999) University of Chicago. All rights reserved +=============================================================================== + +The GraggBulirschStoerIntegrator class in package +org.apache.commons.math3.ode.nonstiff includes software translated +from the odex Fortran routine developed by E. Hairer and G. Wanner. +Original source copyright: +Copyright (c) 2004, Ernst Hairer +=============================================================================== + +The EigenDecompositionImpl class in package +org.apache.commons.math3.linear includes software translated +from some LAPACK Fortran routines. Original source copyright: +Copyright (c) 1992-2008 The University of Tennessee. All rights reserved. +=============================================================================== + +The MersenneTwister class in package org.apache.commons.math3.random +includes software translated from the 2002-01-26 version of +the Mersenne-Twister generator written in C by Makoto Matsumoto and Takuji +Nishimura. Original source copyright: +Copyright (C) 1997 - 2002, Makoto Matsumoto and Takuji Nishimura, +All rights reserved +=============================================================================== + +The LocalizedFormatsTest class in the unit tests is an adapted version of +the OrekitMessagesTest class from the orekit library distributed under the +terms of the Apache 2 licence. Original source copyright: +Copyright 2010 CS Systèmes d'Information +=============================================================================== + +The HermiteInterpolator class and its corresponding test have been imported from +the orekit library distributed under the terms of the Apache 2 licence. Original +source copyright: +Copyright 2010-2012 CS Systèmes d'Information +=============================================================================== + +The creation of the package "o.a.c.m.analysis.integration.gauss" was inspired +by an original code donated by Sébastien Brisard. +=============================================================================== + + +The complete text of licenses and disclaimers associated with the the original +sources enumerated above at the time of code translation are in the LICENSE.txt +file. + + +Apache Commons Net +Copyright 2001-2017 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons Text +Copyright 2014-2018 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache Commons Validator +Copyright 2001-2017 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache Curator +Copyright 2013-2014 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Ehcache V3 +Copyright 2014-2016 Terracotta, Inc. + +The product includes software from the Apache Commons Lang project, +under the Apache License 2.0 (see: org.ehcache.impl.internal.classes.commonslang) + + +Apache Geronimo +Copyright 2003-2018 The Apache Software Foundation + +This product includes software developed by +The Apache Software Foundation (http://www.apache.org/). + + +Copyright 2014 The gRPC Authors + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +----------------------------------------------------------------------- + +This product contains a modified portion of 'OkHttp', an open source +HTTP & SPDY client for Android and Java applications, which can be obtained +at: + + * LICENSE: + * okhttp/third_party/okhttp/LICENSE (Apache License 2.0) + * HOMEPAGE: + * https://github.com/square/okhttp + * LOCATION_IN_GRPC: + * okhttp/third_party/okhttp + +This product contains a modified portion of 'Netty', an open source +networking library, which can be obtained at: + + * LICENSE: + * netty/third_party/netty/LICENSE.txt (Apache License 2.0) + * HOMEPAGE: + * https://netty.io + * LOCATION_IN_GRPC: + * netty/third_party/netty + +-- +This product incorporates portions of the 'Hadoop' project + +Copyright 2007-2009 The Apache Software Foundation + +Licensed under the Apache License v2.0 +-- +Our Orca logo we got here: http://www.vectorfree.com/jumping-orca +It is licensed Creative Commons Attribution 3.0. +See https://creativecommons.org/licenses/by/3.0/us/ +We changed the logo by stripping the colored background, inverting +it and then rotating it some. + +Later we found that vectorfree.com image is not properly licensed. +The original is owned by vectorportal.com. The original was +relicensed so we could use it as Creative Commons Attribution 3.0. +The license is bundled with the download available here: +http://www.vectorportal.com/subcategory/205/KILLER-WHALE-FREE-VECTOR.eps/ifile/9136/detailtest.asp +-- +This product includes portions of the Bootstrap project v3.0.0 + +Copyright 2013 Twitter, Inc. + +Licensed under the Apache License v2.0 + +This product uses the Glyphicons Halflings icon set. + +http://glyphicons.com/ + +Copyright Jan Kovařík + +Licensed under the Apache License v2.0 as a part of the Bootstrap project. + +-- +This product includes portions of the Guava project v14 and v21, specifically +'hbase-common/src/main/java/org/apache/hadoop/hbase/io/LimitInputStream.java' +'hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java' + +Copyright (C) 2007 The Guava Authors + +Licensed under the Apache License, Version 2.0 + + +Apache HttpComponents Client +Copyright 1999-2018 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache HttpComponents Core +Copyright 2005-2018 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +============================================================== + Jetty Web Container + Copyright 1995-2017 Mort Bay Consulting Pty Ltd. +============================================================== + +The Jetty Web Container is Copyright Mort Bay Consulting Pty Ltd +unless otherwise noted. + +Jetty is dual licensed under both + + * The Apache 2.0 License + http://www.apache.org/licenses/LICENSE-2.0.html + + and + + * The Eclipse Public 1.0 License + http://www.eclipse.org/legal/epl-v10.html + +Jetty may be distributed under either license. + +------ +Eclipse + +The following artifacts are EPL. + * org.eclipse.jetty.orbit:org.eclipse.jdt.core + +The following artifacts are EPL and ASL2. + * org.eclipse.jetty.orbit:javax.security.auth.message + + +The following artifacts are EPL and CDDL 1.0. + * org.eclipse.jetty.orbit:javax.mail.glassfish + + +------ +Oracle + +The following artifacts are CDDL + GPLv2 with classpath exception. +https://glassfish.dev.java.net/nonav/public/CDDL+GPL.html + + * javax.servlet:javax.servlet-api + * javax.annotation:javax.annotation-api + * javax.transaction:javax.transaction-api + * javax.websocket:javax.websocket-api + +------ +Oracle OpenJDK + +If ALPN is used to negotiate HTTP/2 connections, then the following +artifacts may be included in the distribution or downloaded when ALPN +module is selected. + + * java.sun.security.ssl + +These artifacts replace/modify OpenJDK classes. The modififications +are hosted at github and both modified and original are under GPL v2 with +classpath exceptions. +http://openjdk.java.net/legal/gplv2+ce.html + + +------ +OW2 + +The following artifacts are licensed by the OW2 Foundation according to the +terms of http://asm.ow2.org/license.html + +org.ow2.asm:asm-commons +org.ow2.asm:asm + + +------ +Apache + +The following artifacts are ASL2 licensed. + +org.apache.taglibs:taglibs-standard-spec +org.apache.taglibs:taglibs-standard-impl + + +------ +MortBay + +The following artifacts are ASL2 licensed. Based on selected classes from +following Apache Tomcat jars, all ASL2 licensed. + +org.mortbay.jasper:apache-jsp + org.apache.tomcat:tomcat-jasper + org.apache.tomcat:tomcat-juli + org.apache.tomcat:tomcat-jsp-api + org.apache.tomcat:tomcat-el-api + org.apache.tomcat:tomcat-jasper-el + org.apache.tomcat:tomcat-api + org.apache.tomcat:tomcat-util-scan + org.apache.tomcat:tomcat-util + +org.mortbay.jasper:apache-el + org.apache.tomcat:tomcat-jasper-el + org.apache.tomcat:tomcat-el-api + + +------ +Mortbay + +The following artifacts are CDDL + GPLv2 with classpath exception. + +https://glassfish.dev.java.net/nonav/public/CDDL+GPL.html + +org.eclipse.jetty.toolchain:jetty-schemas + +------ +Assorted + +The UnixCrypt.java code implements the one way cryptography used by +Unix systems for simple password protection. Copyright 1996 Aki Yoshida, +modified April 2001 by Iris Van den Broeke, Daniel Deville. +Permission to use, copy, modify and distribute UnixCrypt +for non-commercial or commercial purposes and without fee is +granted provided that the copyright notice appears in all copies. + + +Apache Kerby +Copyright 2015-2017 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache log4j +Copyright 2010 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Metrics +Copyright 2010-2013 Coda Hale and Yammer, Inc. + +This product includes software developed by Coda Hale and Yammer, Inc. + +This product includes code derived from the JSR-166 project (ThreadLocalRandom, Striped64, +LongAdder), which was released with the following comments: + + Written by Doug Lea with assistance from members of JCP JSR-166 + Expert Group and released to the public domain, as explained at + http://creativecommons.org/publicdomain/zero/1.0/ + + + + The Netty Project + ================= + +Please visit the Netty web site for more information: + + * http://netty.io/ + +Copyright 2014 The Netty Project + +The Netty Project licenses this file to you under the Apache License, +version 2.0 (the "License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at: + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +License for the specific language governing permissions and limitations +under the License. + +Also, please refer to each LICENSE.<component>.txt file, which is located in +the 'license' directory of the distribution file, for the license terms of the +components that this product depends on. + +------------------------------------------------------------------------------- +This product contains the extensions to Java Collections Framework which has +been derived from the works by JSR-166 EG, Doug Lea, and Jason T. Greene: + + * LICENSE: + * license/LICENSE.jsr166y.txt (Public Domain) + * HOMEPAGE: + * http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/ + * http://viewvc.jboss.org/cgi-bin/viewvc.cgi/jbosscache/experimental/jsr166/ + +This product contains a modified version of Robert Harder's Public Domain +Base64 Encoder and Decoder, which can be obtained at: + + * LICENSE: + * license/LICENSE.base64.txt (Public Domain) + * HOMEPAGE: + * http://iharder.sourceforge.net/current/java/base64/ + +This product contains a modified portion of 'Webbit', an event based +WebSocket and HTTP server, which can be obtained at: + + * LICENSE: + * license/LICENSE.webbit.txt (BSD License) + * HOMEPAGE: + * https://github.com/joewalnes/webbit + +This product contains a modified portion of 'SLF4J', a simple logging +facade for Java, which can be obtained at: + + * LICENSE: + * license/LICENSE.slf4j.txt (MIT License) + * HOMEPAGE: + * http://www.slf4j.org/ + +This product contains a modified portion of 'Apache Harmony', an open source +Java SE, which can be obtained at: + + * NOTICE: + * license/NOTICE.harmony.txt + * LICENSE: + * license/LICENSE.harmony.txt (Apache License 2.0) + * HOMEPAGE: + * http://archive.apache.org/dist/harmony/ + +This product contains a modified portion of 'jbzip2', a Java bzip2 compression +and decompression library written by Matthew J. Francis. It can be obtained at: + + * LICENSE: + * license/LICENSE.jbzip2.txt (MIT License) + * HOMEPAGE: + * https://code.google.com/p/jbzip2/ + +This product contains a modified portion of 'libdivsufsort', a C API library to construct +the suffix array and the Burrows-Wheeler transformed string for any input string of +a constant-size alphabet written by Yuta Mori. It can be obtained at: + + * LICENSE: + * license/LICENSE.libdivsufsort.txt (MIT License) + * HOMEPAGE: + * https://github.com/y-256/libdivsufsort + +This product contains a modified portion of Nitsan Wakart's 'JCTools', Java Concurrency Tools for the JVM, + which can be obtained at: + + * LICENSE: + * license/LICENSE.jctools.txt (ASL2 License) + * HOMEPAGE: + * https://github.com/JCTools/JCTools + +This product optionally depends on 'JZlib', a re-implementation of zlib in +pure Java, which can be obtained at: + + * LICENSE: + * license/LICENSE.jzlib.txt (BSD style License) + * HOMEPAGE: + * http://www.jcraft.com/jzlib/ + +This product optionally depends on 'Compress-LZF', a Java library for encoding and +decoding data in LZF format, written by Tatu Saloranta. It can be obtained at: + + * LICENSE: + * license/LICENSE.compress-lzf.txt (Apache License 2.0) + * HOMEPAGE: + * https://github.com/ning/compress + +This product optionally depends on 'lz4', a LZ4 Java compression +and decompression library written by Adrien Grand. It can be obtained at: + + * LICENSE: + * license/LICENSE.lz4.txt (Apache License 2.0) + * HOMEPAGE: + * https://github.com/jpountz/lz4-java + +This product optionally depends on 'lzma-java', a LZMA Java compression +and decompression library, which can be obtained at: + + * LICENSE: + * license/LICENSE.lzma-java.txt (Apache License 2.0) + * HOMEPAGE: + * https://github.com/jponge/lzma-java + +This product contains a modified portion of 'jfastlz', a Java port of FastLZ compression +and decompression library written by William Kinney. It can be obtained at: + + * LICENSE: + * license/LICENSE.jfastlz.txt (MIT License) + * HOMEPAGE: + * https://code.google.com/p/jfastlz/ + +This product contains a modified portion of and optionally depends on 'Protocol Buffers', Google's data +interchange format, which can be obtained at: + + * LICENSE: + * license/LICENSE.protobuf.txt (New BSD License) + * HOMEPAGE: + * https://github.com/google/protobuf + +This product optionally depends on 'Bouncy Castle Crypto APIs' to generate +a temporary self-signed X.509 certificate when the JVM does not provide the +equivalent functionality. It can be obtained at: + + * LICENSE: + * license/LICENSE.bouncycastle.txt (MIT License) + * HOMEPAGE: + * http://www.bouncycastle.org/ + +This product optionally depends on 'Snappy', a compression library produced +by Google Inc, which can be obtained at: + + * LICENSE: + * license/LICENSE.snappy.txt (New BSD License) + * HOMEPAGE: + * https://github.com/google/snappy + +This product optionally depends on 'JBoss Marshalling', an alternative Java +serialization API, which can be obtained at: + + * LICENSE: + * license/LICENSE.jboss-marshalling.txt (GNU LGPL 2.1) + * HOMEPAGE: + * http://www.jboss.org/jbossmarshalling + +This product optionally depends on 'Caliper', Google's micro- +benchmarking framework, which can be obtained at: + + * LICENSE: + * license/LICENSE.caliper.txt (Apache License 2.0) + * HOMEPAGE: + * https://github.com/google/caliper + +This product optionally depends on 'Apache Commons Logging', a logging +framework, which can be obtained at: + + * LICENSE: + * license/LICENSE.commons-logging.txt (Apache License 2.0) + * HOMEPAGE: + * http://commons.apache.org/logging/ + +This product optionally depends on 'Apache Log4J', a logging framework, which +can be obtained at: + + * LICENSE: + * license/LICENSE.log4j.txt (Apache License 2.0) + * HOMEPAGE: + * http://logging.apache.org/log4j/ + +This product optionally depends on 'Aalto XML', an ultra-high performance +non-blocking XML processor, which can be obtained at: + + * LICENSE: + * license/LICENSE.aalto-xml.txt (Apache License 2.0) + * HOMEPAGE: + * http://wiki.fasterxml.com/AaltoHome + +This product contains a modified version of 'HPACK', a Java implementation of +the HTTP/2 HPACK algorithm written by Twitter. It can be obtained at: + + * LICENSE: + * license/LICENSE.hpack.txt (Apache License 2.0) + * HOMEPAGE: + * https://github.com/twitter/hpack + +This product contains a modified portion of 'Apache Commons Lang', a Java library +provides utilities for the java.lang API, which can be obtained at: + + * LICENSE: + * license/LICENSE.commons-lang.txt (Apache License 2.0) + * HOMEPAGE: + * https://commons.apache.org/proper/commons-lang/ + + +This product contains the Maven wrapper scripts from 'Maven Wrapper', that provides an easy way to ensure a user has everything necessary to run the Maven build. + + * LICENSE: + * license/LICENSE.mvn-wrapper.txt (Apache License 2.0) + * HOMEPAGE: + * https://github.com/takari/maven-wrapper + + +This product includes software developed by Google + Snappy: http://code.google.com/p/snappy/ (New BSD License) + +This product includes software developed by Apache + PureJavaCrc32C from apache-hadoop-common http://hadoop.apache.org/ + (Apache 2.0 license) + +This library containd statically linked libstdc++. This inclusion is allowed by +"GCC RUntime Library Exception" +http://gcc.gnu.org/onlinedocs/libstdc++/manual/license.html + +== Contributors == + * Tatu Saloranta + * Providing benchmark suite + * Alec Wysoker + * Performance and memory usage improvement diff --git a/NOTICE.txt b/NOTICE.txt new file mode 100644 index 0000000..f6715f7 --- /dev/null +++ b/NOTICE.txt @@ -0,0 +1,34 @@ +Apache Hadoop +Copyright 2006 and onwards The Apache Software Foundation. + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +Export Control Notice +--------------------- + +This distribution includes cryptographic software. The country in +which you currently reside may have restrictions on the import, +possession, use, and/or re-export to another country, of +encryption software. BEFORE using any encryption software, please +check your country's laws, regulations and policies concerning the +import, possession, or use, and re-export of encryption software, to +see if this is permitted. See <http://www.wassenaar.org/> for more +information. + +The U.S. Government Department of Commerce, Bureau of Industry and +Security (BIS), has classified this software as Export Commodity +Control Number (ECCN) 5D002.C.1, which includes information security +software using or performing cryptographic functions with asymmetric +algorithms. The form and manner of this Apache Software Foundation +distribution makes it eligible for export under the License Exception +ENC Technology Software Unrestricted (TSU) exception (see the BIS +Export Administration Regulations, Section 740.13) for both object +code and source code. + +The following provides more details on the included cryptographic software: + +This software uses the SSL libraries from the Jetty project written +by mortbay.org. +Hadoop Yarn Server Web Proxy uses the BouncyCastle Java +cryptography APIs written by the Legion of the Bouncy Castle Inc. diff --git a/dev-support/bin/create-release b/dev-support/bin/create-release new file mode 100755 index 0000000..db767e2 --- /dev/null +++ b/dev-support/bin/create-release @@ -0,0 +1,641 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +if [[ -z "${BASH_VERSINFO[0]}" ]] \ + || [[ "${BASH_VERSINFO[0]}" -lt 3 ]] \ + || [[ "${BASH_VERSINFO[0]}" -eq 3 && "${BASH_VERSINFO[1]}" -lt 2 ]]; then + echo "bash v3.2+ is required. Sorry." + exit 1 +fi + +function centered_text +{ + local text="$*" + local spacing=$(( (75+${#text}) /2 )) + printf "%*s\n" ${spacing} "${text}" +} + +function big_console_header +{ + printf "\n\n" + echo "****************************************************************************" + centered_text "${@}" + echo "****************************************************************************" + printf "\n\n" +} + +## @description Given a filename or dir, return the absolute version of it +## @audience public +## @stability stable +## @param directory +## @replaceable no +## @return 0 success +## @return 1 failure +## @return stdout abspath +function hadoop_abs +{ + declare obj=$1 + declare dir + declare fn + declare ret + + if [[ ! -e ${obj} ]]; then + return 1 + elif [[ -d ${obj} ]]; then + dir=${obj} + else + dir=$(dirname -- "${obj}") + fn=$(basename -- "${obj}") + fn="/${fn}" + fi + + dir=$(cd -P -- "${dir}" >/dev/null 2>/dev/null && pwd -P) + ret=$? + if [[ ${ret} = 0 ]]; then + echo "${dir}${fn}" + return 0 + fi + return 1 +} + +## @description Print a message to stderr +## @audience public +## @stability stable +## @replaceable no +## @param string +function hadoop_error +{ + echo "$*" 1>&2 +} + + +function run_and_redirect +{ + declare logfile=$1 + shift + declare res + + echo "\$ ${*} > ${logfile} 2>&1" + # to the log + { + date + echo "cd $(pwd)" + echo "${*}" + } > "${logfile}" + # run the actual command + "${@}" >> "${logfile}" 2>&1 + res=$? + if [[ ${res} != 0 ]]; then + echo + echo "Failed!" + echo + exit "${res}" + fi +} + +# Function to probe the exit code of the script commands, +# and stop in the case of failure with an contextual error +# message. +function run() +{ + declare res + declare logfile + + echo "\$ ${*}" + "${@}" + res=$? + if [[ ${res} != 0 ]]; then + echo + echo "Failed!" + echo + exit "${res}" + fi +} + +function header() +{ + echo + printf "\n\n" + echo "============================================================================" + echo "============================================================================" + centered_text "Hadoop Thirdparty Release Creator" + echo "============================================================================" + echo "============================================================================" + printf "\n\n" + echo "Version to create : ${HADOOP_THIRDPARTY_VERSION}" + echo "Release Candidate Label: ${RC_LABEL##-}" + echo "Source Version : ${DEFAULT_HADOOP_THIRDPARTY_VERSION}" + printf "\n\n" +} + +function set_defaults +{ + BINDIR=$(dirname "${BIN}") + BASEDIR=$(hadoop_abs "${BINDIR}/../..") + + ARTIFACTS_DIR="${BASEDIR}/target/artifacts" + + # Extract Hadoop version from ${BASEDIR}/pom.xml + DEFAULT_HADOOP_THIRDPARTY_VERSION=$(grep "<version>" "${BASEDIR}/pom.xml" \ + | head -1 \ + | sed -e 's|^ *<version>||' -e 's|</version>.*$||') + + DEPLOY=false + + DOCKER=false + DOCKERCACHE=false + DOCKERFILE="${BASEDIR}/dev-support/docker/Dockerfile" + DOCKERRAN=false + + # Extract Java version from ${BASEDIR}/pom.xml + # doing this outside of maven means we can do this before + # the docker container comes up... + JVM_VERSION=$(grep "<javac.version>" "${BASEDIR}/pom.xml" \ + | head -1 \ + | sed -e 's|^ *<javac.version>||' -e 's|</javac.version>.*$||' -e 's|..||') + + GIT=$(command -v git) + + GPG=$(command -v gpg) + GPGAGENT=$(command -v gpg-agent) + + HADOOP_THIRDPARTY_VERSION="${DEFAULT_HADOOP_THIRDPARTY_VERSION}" + + INDOCKER=false + + LOGDIR="${BASEDIR}/patchprocess" + + if [[ -z "${MVN}" ]]; then + if [[ -n "${MAVEN_HOME}" ]]; then + MVN=${MAVEN_HOME}/bin/mvn + else + MVN=$(command -v mvn) + fi + fi + + OSNAME=$(uname -s) + + PUBKEYFILE="https://dist.apache.org/repos/dist/release/hadoop/common/KEYS" + + SIGN=false +} + +function startgpgagent +{ + if [[ "${SIGN}" = true ]]; then + if [[ -n "${GPGAGENT}" && -z "${GPG_AGENT_INFO}" ]]; then + echo "starting gpg agent" + echo "default-cache-ttl 36000" > "${LOGDIR}/gpgagent.conf" + echo "max-cache-ttl 36000" >> "${LOGDIR}/gpgagent.conf" + # shellcheck disable=2046 + eval $("${GPGAGENT}" --daemon \ + --options "${LOGDIR}/gpgagent.conf" \ + --log-file="${LOGDIR}/create-release-gpgagent.log") + GPGAGENTPID=$(pgrep "${GPGAGENT}") + export GPG_AGENT_INFO="$HOME/.gnupg/S.gpg-agent:$(pgrep gpg-agent):1" + fi + + if [[ -n "${GPG_AGENT_INFO}" ]]; then + echo "Warming the gpg-agent cache prior to calling maven" + # warm the agent's cache: + touch "${LOGDIR}/warm" + ${GPG} --use-agent --armor --output "${LOGDIR}/warm.asc" --detach-sig "${LOGDIR}/warm" + rm "${LOGDIR}/warm.asc" "${LOGDIR}/warm" + else + SIGN=false + hadoop_error "ERROR: Unable to launch or acquire gpg-agent. Disable signing." + fi + fi +} + +function stopgpgagent +{ + if [[ -n "${GPGAGENTPID}" ]]; then + kill "${GPGAGENTPID}" + fi +} + +function usage +{ + echo "--artifactsdir=[path] Path to use to store release bits" + echo "--asfrelease Make an ASF release" + echo "--deploy Deploy Maven artifacts using ~/.m2/settings.xml" + echo "--docker Use Hadoop's Dockerfile for guaranteed environment" + echo "--dockercache Use a Docker-private maven cache" + echo "--logdir=[path] Path to store logs" + echo "--mvncache=[path] Path to the maven cache to use" + echo "--rc-label=[label] Add this label to the builds" + echo "--security Emergency security release" + echo "--sign Use .gnupg dir to sign the artifacts and jars" + echo "--version=[version] Use an alternative version string" +} + +function option_parse +{ + declare i + + for i in "$@"; do + case ${i} in + --asfrelease) + ASFRELEASE=true + SIGN=true + DEPLOY=true + ;; + --artifactsdir=*) + ARTIFACTS_DIR=${i#*=} + ;; + --deploy) + DEPLOY=true + ;; + --docker) + DOCKER=true + ;; + --dockercache) + DOCKERCACHE=true + ;; + --help) + usage + exit + ;; + --indocker) + INDOCKER=true + ;; + --logdir=*) + LOGDIR=${i#*=} + ;; + --mvncache=*) + MVNCACHE=${i#*=} + ;; + --rc-label=*) + RC_LABEL=${i#*=} + ;; + --security) + SECURITYRELEASE=true + ;; + --sign) + SIGN=true + ;; + --version=*) + HADOOP_THIRDPARTY_VERSION=${i#*=} + ;; + esac + done + + if [[ ! -d "${HOME}/.gnupg" ]]; then + hadoop_error "ERROR: No .gnupg dir. Disabling signing capability." + SIGN=false + fi + + if [[ "${SIGN}" = true ]]; then + if [[ -n "${GPG_AGENT_INFO}" ]]; then + echo "NOTE: Using existing gpg-agent. If the default-cache-ttl" + echo "is set to less than ~20 mins, maven commands will fail." + elif [[ -z "${GPGAGENT}" ]]; then + hadoop_error "ERROR: No gpg-agent. Disabling signing capability." + SIGN=false + fi + fi + + if [[ "${DEPLOY}" = true && ! -f "${HOME}/.m2/settings.xml" ]]; then + hadoop_error "ERROR: No ~/.m2/settings.xml file, cannot deploy Maven artifacts." + exit 1 + fi + + DOCKERCMD=$(command -v docker) + if [[ "${DOCKER}" = true && -z "${DOCKERCMD}" ]]; then + hadoop_error "ERROR: docker binary not found. Disabling docker mode." + DOCKER=false + fi + + if [[ "${DOCKERCACHE}" = true && "${DOCKER}" = false ]]; then + if [[ "${INDOCKER}" = false ]]; then + hadoop_error "ERROR: docker mode not enabled. Disabling dockercache." + fi + DOCKERCACHE=false + fi + + if [[ "${DOCKERCACHE}" = true && -n "${MVNCACHE}" ]]; then + hadoop_error "ERROR: Cannot set --mvncache and --dockercache simultaneously." + exit 1 + else + MVNCACHE=${MVNCACHE:-"${HOME}/.m2/repository"} + fi + + if [[ "${ASFRELEASE}" = true ]]; then + if [[ "${SIGN}" = false ]]; then + hadoop_error "ERROR: --asfrelease requires --sign. Exiting." + exit 1 + fi + + if [[ "${OSNAME}" = Linux ]]; then + if [[ "${DOCKER}" = false && "${INDOCKER}" = false ]]; then + hadoop_error "ERROR: --asfrelease requires --docker on Linux. Exiting." + exit 1 + elif [[ "${DOCKERCACHE}" = false && "${INDOCKER}" = false ]]; then + hadoop_error "ERROR: --asfrelease on Linux requires --dockercache. Exiting." + exit 1 + fi + fi + fi + + if [[ -n "${MVNCACHE}" ]]; then + mkdir -p "${MVNCACHE}" + if [[ -d "${MVNCACHE}" ]]; then + MVN_ARGS=("-Dmaven.repo.local=${MVNCACHE}") + fi + fi + + if [[ "${SECURITYRELEASE}" = true ]]; then + if [[ ! -d "${BASEDIR}/src/site/markdown/release/${HADOOP_THIRDPARTY_VERSION}" ]]; then + hadoop_error "ERROR: ${BASEDIR}/src/site/markdown/release/${HADOOP_THIRDPARTY_VERSION} does not exist." + hadoop_error "ERROR: This directory and its contents are required to be manually created for a security release." + exit 1 + fi + fi +} + +function dockermode +{ + declare lines + declare -a modp + declare imgname + declare -a extrad + declare user_name + declare group_id + + if [[ "${DOCKER}" != true ]]; then + return + fi + + user_name=${SUDO_USER:=$USER} + user_id=$(id -u "${user_name}") + group_id=$(id -g "${user_name}") + + imgname="hadoop/createrelease:${HADOOP_THIRDPARTY_VERSION}_${RANDOM}" + + if [[ -d "${HOME}/.gnupg" ]]; then + extrad+=("-v" "${HOME}/.gnupg:/home/${user_name}/.gnupg") + fi + + if [[ -n "${LOGDIR}" ]]; then + if [[ ! -d "${LOGDIR}" ]]; then + mkdir -p "${LOGDIR}" + fi + lines=$(hadoop_abs "${LOGDIR}") + extrad+=("-v" "${lines}:${lines}") + fi + + if [[ -n "${ARTIFACTS_DIR}" ]]; then + if [[ ! -d "${ARTIFACTS_DIR}" ]]; then + mkdir -p "${ARTIFACTS_DIR}" + fi + lines=$(hadoop_abs "${ARTIFACTS_DIR}") + extrad+=("-v" "${lines}:${lines}") + fi + + if [[ "${DEPLOY}" = true ]]; then + modp+=("--deploy") + extrad+=("-v" "${HOME}/.m2/settings.xml:/home/${user_name}/.m2/settings.xml") + fi + + if [[ "${DOCKERCACHE}" = true ]]; then + modp+=("--mvncache=/maven") + else + lines=$(hadoop_abs "${MVNCACHE}") + extrad+=("-v" "${lines}:${lines}") + fi + + for lines in "${PARAMS[@]}"; do + if [[ "${lines}" != "--docker" ]]; then + modp+=("$lines") + fi + done + + modp+=("--indocker") + + ( + lines=$(grep -n 'YETUS CUT HERE' "${DOCKERFILE}" | cut -f1 -d:) + if [[ -z "${lines}" ]]; then + cat "${DOCKERFILE}" + else + head -n "${lines}" "${DOCKERFILE}" + fi + # make sure we put some space between, just in case last + # line isn't an empty line or whatever + printf "\n\n" + + # force a new image for every run to make it easier to remove later + echo "LABEL org.apache.hadoop.create-release=\"cr-${RANDOM}\"" + + # setup ownerships, etc + echo "RUN groupadd --non-unique -g ${group_id} ${user_name}" + echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name}" + echo "RUN chown -R ${user_name} /home/${user_name}" + echo "ENV HOME /home/${user_name}" + echo "RUN mkdir -p /maven" + echo "RUN chown -R ${user_name} /maven" + + # we always force build with the OpenJDK JDK + # but with the correct version + echo "ENV JAVA_HOME /usr/lib/jvm/java-${JVM_VERSION}-openjdk-amd64" + echo "USER ${user_name}" + printf "\n\n" + ) | docker build -t "${imgname}" - + + run docker run -i -t \ + --privileged \ + "${extrad[@]}" \ + -v "${BASEDIR}:/build/source" \ + -u "${user_name}" \ + -w "/build/source" \ + "${imgname}" \ + "/build/source/dev-support/bin/create-release" "${modp[@]}" + + DOCKERRAN=true +} + +function makearelease +{ + # let's start at the root + run cd "${BASEDIR}" + + big_console_header "Cleaning the Source Tree" + + # git clean to clear any remnants from previous build + run "${GIT}" clean -xdf -e /patchprocess + + mkdir -p "${LOGDIR}" + + # mvn clean for sanity + run_and_redirect "${LOGDIR}/mvn_clean.log" "${MVN}" "${MVN_ARGS[@]}" clean + + # Create staging dir for release artifacts + run mkdir -p "${ARTIFACTS_DIR}" + + big_console_header "Apache RAT Check" + + # Create RAT report + run_and_redirect "${LOGDIR}/mvn_apache_rat.log" "${MVN}" "${MVN_ARGS[@]}" apache-rat:check + + big_console_header "Maven Build and Install" + + if [[ "${SIGN}" = true ]]; then + signflags=("-Psign" "-Dgpg.useagent=true" "-Dgpg.executable=${GPG}") + fi + + local target="install" + if [[ "${DEPLOY}" = true ]]; then + target="deploy" + fi + + # Create SRC and BIN tarballs for release, + # shellcheck disable=SC2046 + run_and_redirect "${LOGDIR}/mvn_${target}.log" \ + "${MVN}" "${MVN_ARGS[@]}" ${target} \ + -Psrc \ + "${signflags[@]}" \ + -DskipTests + + # Stage SRC tarball + run mv \ + "${BASEDIR}/target/hadoop-thirdparty-${HADOOP_THIRDPARTY_VERSION}-src.tar.gz" \ + "${ARTIFACTS_DIR}/hadoop-thirdparty-${HADOOP_THIRDPARTY_VERSION}${RC_LABEL}-src.tar.gz" + + big_console_header "Maven Site" + + if [[ "${SECURITYRELEASE}" = true ]]; then + DOCFLAGS="" + hadoop_error "WARNING: Skipping automatic changelog and release notes generation due to --security" + else + DOCFLAGS="-Preleasedocs" + fi + + # Create site for release + # we need to do install again so that jdiff and + # a few other things get registered in the maven + # universe correctly + run_and_redirect "${LOGDIR}/mvn_site.log" \ + "${MVN}" "${MVN_ARGS[@]}" install \ + site site:stage \ + -DskipTests \ + -Psrc \ + "${DOCFLAGS}" + + # Create the site tarball + run mv "${BASEDIR}/target/staging" "${BASEDIR}/target/r${HADOOP_THIRDPARTY_VERSION}/" + run cd "${BASEDIR}/target/" + run tar czpf "hadoop-thirdparty-site-${HADOOP_THIRDPARTY_VERSION}.tar.gz" "r${HADOOP_THIRDPARTY_VERSION}"/* + run cd "${BASEDIR}" + + # Stage SITE tarball + run mv \ + "${BASEDIR}/target/hadoop-thirdparty-site-${HADOOP_THIRDPARTY_VERSION}.tar.gz" \ + "${ARTIFACTS_DIR}/hadoop-thirdparty-${HADOOP_THIRDPARTY_VERSION}${RC_LABEL}-site.tar.gz" + + # Stage RAT report + #shellcheck disable=SC2038 + find . -name rat.txt | xargs -I% cat % > "${ARTIFACTS_DIR}/hadoop-thirdparty-${HADOOP_THIRDPARTY_VERSION}${RC_LABEL}-rat.txt" + + # Stage CHANGELOG and RELEASENOTES files + for i in CHANGELOG RELEASENOTES; do + if [[ $(ls -l "${BASEDIR}/src/site/markdown/release/${HADOOP_THIRDPARTY_VERSION}"/${i}*.md | wc -l) == 0 ]]; then + echo "No ${i} found. Continuing..." + continue; + fi + run cp -p \ + "${BASEDIR}/src/site/markdown/release/${HADOOP_THIRDPARTY_VERSION}"/${i}*.md \ + "${ARTIFACTS_DIR}/${i}.md" + done + +} + +function signartifacts +{ + declare i + declare ret + + if [[ "${SIGN}" = false ]]; then + echo "" + echo "Remember to sign the artifacts before staging them on the open" + echo "" + return + fi + + big_console_header "Signing the release" + + for i in ${ARTIFACTS_DIR}/*; do + ${GPG} --use-agent --armor --output "${i}.asc" --detach-sig "${i}" + sha512sum --tag "${i}" > "${i}.sha512" + done + + if [[ "${ASFRELEASE}" = true ]]; then + echo "Fetching the Apache Hadoop KEYS file..." + curl -L "${PUBKEYFILE}" -o "${BASEDIR}/target/KEYS" + ${GPG} --import --trustdb "${BASEDIR}/target/testkeysdb" "${BASEDIR}/target/KEYS" + ${GPG} --verify --trustdb "${BASEDIR}/target/testkeysdb" \ + "${ARTIFACTS_DIR}/hadoop-thirdparty-${HADOOP_THIRDPARTY_VERSION}${RC_LABEL}-src.tar.gz.asc" \ + "${ARTIFACTS_DIR}/hadoop-thirdparty-${HADOOP_THIRDPARTY_VERSION}${RC_LABEL}-src.tar.gz" + ret=$? + if [[ ${ret} != 0 ]]; then + hadoop_error "ERROR: GPG key is not present in ${PUBKEYFILE}." + hadoop_error "ERROR: This MUST be fixed. Exiting." + exit 1 + fi + fi +} + +# find root of the source tree +BIN=$(hadoop_abs "${BASH_SOURCE:-$0}") +PARAMS=("$@") + +set_defaults + +option_parse "${PARAMS[@]}" + +dockermode + +header + +if [[ -n ${RC_LABEL} ]]; then + RC_LABEL="-${RC_LABEL}" +fi + +if [[ "${INDOCKER}" = true || "${DOCKERRAN}" = false ]]; then + + startgpgagent + + makearelease + releaseret=$? + + signartifacts + + stopgpgagent +fi + +if [[ "${INDOCKER}" = true ]]; then + exit $? +fi + +if [[ ${releaseret} == 0 ]]; then + echo + echo "Congratulations, you have successfully built the release" + echo "artifacts for Apache Hadoop Thirdparty ${HADOOP_THIRDPARTY_VERSION}${RC_LABEL}" + echo + echo "The artifacts for this run are available at ${ARTIFACTS_DIR}:" + run ls -1 "${ARTIFACTS_DIR}" + + echo +fi diff --git a/dev-support/bin/releasedocmaker b/dev-support/bin/releasedocmaker new file mode 100755 index 0000000..16e4c7e --- /dev/null +++ b/dev-support/bin/releasedocmaker @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P) +exec "${BINDIR}/yetus-wrapper" releasedocmaker "$@" diff --git a/dev-support/bin/yetus-wrapper b/dev-support/bin/yetus-wrapper new file mode 100755 index 0000000..b0f71f1 --- /dev/null +++ b/dev-support/bin/yetus-wrapper @@ -0,0 +1,188 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# you must be this high to ride the ride +if [[ -z "${BASH_VERSINFO[0]}" ]] \ + || [[ "${BASH_VERSINFO[0]}" -lt 3 ]] \ + || [[ "${BASH_VERSINFO[0]}" -eq 3 && "${BASH_VERSINFO[1]}" -lt 2 ]]; then + echo "bash v3.2+ is required. Sorry." + exit 1 +fi + +set -o pipefail + +## @description Print a message to stderr +## @audience public +## @stability stable +## @replaceable no +## @param string +function yetus_error +{ + echo "$*" 1>&2 +} + +## @description Given a filename or dir, return the absolute version of it +## @audience public +## @stability stable +## @param directory +## @replaceable no +## @return 0 success +## @return 1 failure +## @return stdout abspath +function yetus_abs +{ + declare obj=$1 + declare dir + declare fn + declare dirret + + if [[ ! -e ${obj} ]]; then + return 1 + elif [[ -d ${obj} ]]; then + dir=${obj} + else + dir=$(dirname -- "${obj}") + fn=$(basename -- "${obj}") + fn="/${fn}" + fi + + dir=$(cd -P -- "${dir}" >/dev/null 2>/dev/null && pwd -P) + dirret=$? + if [[ ${dirret} = 0 ]]; then + echo "${dir}${fn}" + return 0 + fi + return 1 +} + +function version_ge() +{ + test "$(echo "$@" | tr " " "\n" | sort -rV | head -n 1)" == "$1"; +} + +WANTED="$1" +shift +ARGV=("$@") + +HADOOP_YETUS_VERSION=${HADOOP_YETUS_VERSION:-0.10.0} +BIN=$(yetus_abs "${BASH_SOURCE-$0}") +BINDIR=$(dirname "${BIN}") + +## HADOOP_YETUS_VERSION >= 0.9.0 the tarball named with apache-yetus prefix +if version_ge "${HADOOP_YETUS_VERSION}" "0.9.0"; then + YETUS_PREFIX=apache-yetus +else + YETUS_PREFIX=yetus +fi + +### +### if YETUS_HOME is set, then try to use it +### +if [[ -n "${YETUS_HOME}" && -x "${YETUS_HOME}/bin/${WANTED}" ]]; then + exec "${YETUS_HOME}/bin/${WANTED}" "${ARGV[@]}" +fi + +# +# this directory is ignored by git and maven +# +HADOOP_PATCHPROCESS=${HADOOP_PATCHPROCESS:-"${BINDIR}/../../patchprocess"} + +if [[ ! -d "${HADOOP_PATCHPROCESS}" ]]; then + mkdir -p "${HADOOP_PATCHPROCESS}" +fi + +mytmpdir=$(yetus_abs "${HADOOP_PATCHPROCESS}") +ret=$? +if [[ ${ret} != 0 ]]; then + yetus_error "yetus-dl: Unable to cwd to ${HADOOP_PATCHPROCESS}" + exit 1 +fi +HADOOP_PATCHPROCESS=${mytmpdir} + +## +## if we've already DL'd it, then short cut +## +if [[ -x "${HADOOP_PATCHPROCESS}/${YETUS_PREFIX}-${HADOOP_YETUS_VERSION}/bin/${WANTED}" ]]; then + exec "${HADOOP_PATCHPROCESS}/${YETUS_PREFIX}-${HADOOP_YETUS_VERSION}/bin/${WANTED}" "${ARGV[@]}" +fi + +## +## need to DL, etc +## + +BASEURL="https://archive.apache.org/dist/yetus/${HADOOP_YETUS_VERSION}/" +TARBALL="${YETUS_PREFIX}-${HADOOP_YETUS_VERSION}-bin.tar" + +GPGBIN=$(command -v gpg) +CURLBIN=$(command -v curl) + +if ! pushd "${HADOOP_PATCHPROCESS}" >/dev/null; then + yetus_error "ERROR: yetus-dl: Cannot pushd to ${HADOOP_PATCHPROCESS}" + exit 1 +fi + +if [[ -n "${CURLBIN}" ]]; then + if ! "${CURLBIN}" -f -s -L -O "${BASEURL}/${TARBALL}.gz"; then + yetus_error "ERROR: yetus-dl: unable to download ${BASEURL}/${TARBALL}.gz" + exit 1 + fi +else + yetus_error "ERROR: yetus-dl requires curl." + exit 1 +fi + +if [[ -n "${GPGBIN}" ]]; then + if ! mkdir -p .gpg; then + yetus_error "ERROR: yetus-dl: Unable to create ${HADOOP_PATCHPROCESS}/.gpg" + exit 1 + fi + if ! chmod -R 700 .gpg; then + yetus_error "ERROR: yetus-dl: Unable to chmod ${HADOOP_PATCHPROCESS}/.gpg" + exit 1 + fi + if ! "${CURLBIN}" -s -L -o KEYS_YETUS https://dist.apache.org/repos/dist/release/yetus/KEYS; then + yetus_error "ERROR: yetus-dl: unable to fetch https://dist.apache.org/repos/dist/release/yetus/KEYS" + exit 1 + fi + if ! "${CURLBIN}" -s -L -O "${BASEURL}/${TARBALL}.gz.asc"; then + yetus_error "ERROR: yetus-dl: unable to fetch ${BASEURL}/${TARBALL}.gz.asc" + exit 1 + fi + if ! "${GPGBIN}" --homedir "${HADOOP_PATCHPROCESS}/.gpg" --import "${HADOOP_PATCHPROCESS}/KEYS_YETUS" >/dev/null 2>&1; then + yetus_error "ERROR: yetus-dl: gpg unable to import ${HADOOP_PATCHPROCESS}/KEYS_YETUS" + exit 1 + fi + if ! "${GPGBIN}" --homedir "${HADOOP_PATCHPROCESS}/.gpg" --verify "${TARBALL}.gz.asc" >/dev/null 2>&1; then + yetus_error "ERROR: yetus-dl: gpg verify of tarball in ${HADOOP_PATCHPROCESS} failed" + exit 1 + fi +fi + +if ! (gunzip -c "${TARBALL}.gz" | tar xpf -); then + yetus_error "ERROR: ${TARBALL}.gz is corrupt. Investigate and then remove ${HADOOP_PATCHPROCESS} to try again." + exit 1 +fi + +if [[ -x "${HADOOP_PATCHPROCESS}/${YETUS_PREFIX}-${HADOOP_YETUS_VERSION}/bin/${WANTED}" ]]; then + popd >/dev/null + exec "${HADOOP_PATCHPROCESS}/${YETUS_PREFIX}-${HADOOP_YETUS_VERSION}/bin/${WANTED}" "${ARGV[@]}" +fi + +## +## give up +## +yetus_error "ERROR: ${WANTED} is not part of Apache Yetus ${HADOOP_YETUS_VERSION}" +exit 1 diff --git a/dev-support/docker/Dockerfile b/dev-support/docker/Dockerfile new file mode 100644 index 0000000..657c223 --- /dev/null +++ b/dev-support/docker/Dockerfile @@ -0,0 +1,219 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Dockerfile for installing the necessary dependencies for building Hadoop. +# See BUILDING.txt. + +FROM ubuntu:xenial + +WORKDIR /root + +SHELL ["/bin/bash", "-o", "pipefail", "-c"] + +##### +# Disable suggests/recommends +##### +RUN echo APT::Install-Recommends "0"\; > /etc/apt/apt.conf.d/10disableextras +RUN echo APT::Install-Suggests "0"\; >> /etc/apt/apt.conf.d/10disableextras + +ENV DEBIAN_FRONTEND noninteractive +ENV DEBCONF_TERSE true + +###### +# Install common dependencies from packages. Versions here are either +# sufficient or irrelevant. +# +# WARNING: DO NOT PUT JAVA APPS HERE! Otherwise they will install default +# Ubuntu Java. See Java section below! +###### +# hadolint ignore=DL3008 +RUN apt-get -q update \ + && apt-get -q install -y --no-install-recommends \ + apt-utils \ + build-essential \ + bzip2 \ + clang \ + curl \ + doxygen \ + fuse \ + g++ \ + gcc \ + git \ + gnupg-agent \ + libbz2-dev \ + libcurl4-openssl-dev \ + libfuse-dev \ + libprotobuf-dev \ + libprotoc-dev \ + libsasl2-dev \ + libsnappy-dev \ + libssl-dev \ + libtool \ + libzstd1-dev \ + locales \ + make \ + pinentry-curses \ + pkg-config \ + python \ + python2.7 \ + python-pip \ + python-pkg-resources \ + python-setuptools \ + python-wheel \ + rsync \ + software-properties-common \ + snappy \ + sudo \ + valgrind \ + zlib1g-dev \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + + +####### +# OpenJDK 8 +####### +# hadolint ignore=DL3008 +RUN apt-get -q update \ + && apt-get -q install -y --no-install-recommends openjdk-8-jdk libbcprov-java \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + + +###### +# Install cmake 3.1.0 (3.5.1 ships with Xenial) +###### +RUN mkdir -p /opt/cmake \ + && curl -L -s -S \ + https://cmake.org/files/v3.1/cmake-3.1.0-Linux-x86_64.tar.gz \ + -o /opt/cmake.tar.gz \ + && tar xzf /opt/cmake.tar.gz --strip-components 1 -C /opt/cmake +ENV CMAKE_HOME /opt/cmake +ENV PATH "${PATH}:/opt/cmake/bin" + +###### +# Install Google Protobuf 3.7.1 (2.6.0 ships with Xenial) +###### +# hadolint ignore=DL3003 +RUN mkdir -p /opt/protobuf-src \ + && curl -L -s -S \ + https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz \ + -o /opt/protobuf.tar.gz \ + && tar xzf /opt/protobuf.tar.gz --strip-components 1 -C /opt/protobuf-src \ + && cd /opt/protobuf-src \ + && ./configure --prefix=/opt/protobuf \ + && make install \ + && cd /root \ + && rm -rf /opt/protobuf-src +ENV PROTOBUF_HOME /opt/protobuf +ENV PATH "${PATH}:/opt/protobuf/bin" + +###### +# Install Apache Maven 3.3.9 (3.3.9 ships with Xenial) +###### +# hadolint ignore=DL3008 +RUN apt-get -q update \ + && apt-get -q install -y --no-install-recommends maven \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* +ENV MAVEN_HOME /usr + +###### +# Install findbugs 3.0.1 (3.0.1 ships with Xenial) +# Ant is needed for findbugs +###### +# hadolint ignore=DL3008 +RUN apt-get -q update \ + && apt-get -q install -y --no-install-recommends findbugs ant \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* +ENV FINDBUGS_HOME /usr + +#### +# Install shellcheck (0.4.6, the latest as of 2017-09-26) +#### +# hadolint ignore=DL3008 +RUN add-apt-repository -y ppa:jonathonf/ghc-8.0.2 \ + && apt-get -q update \ + && apt-get -q install -y --no-install-recommends shellcheck \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + +#### +# Install bats (0.4.0, the latest as of 2017-09-26, ships with Xenial) +#### +# hadolint ignore=DL3008 +RUN apt-get -q update \ + && apt-get -q install -y --no-install-recommends bats \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + +#### +# Install pylint at fixed version (2.0.0 removed python2 support) +# https://github.com/PyCQA/pylint/issues/2294 +#### +RUN pip2 install pylint==1.9.2 + +#### +# Install dateutil.parser +#### +RUN pip2 install python-dateutil==2.7.3 + +### +# Install node.js for web UI framework (4.2.6 ships with Xenial) +### +# hadolint ignore=DL3008, DL3016 +RUN apt-get -q update \ + && apt-get install -y --no-install-recommends nodejs npm \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* \ + && ln -s /usr/bin/nodejs /usr/bin/node \ + && npm install npm@latest -g \ + && npm install -g jshint + +### +# Install hadolint +#### +RUN curl -L -s -S \ + https://github.com/hadolint/hadolint/releases/download/v1.11.1/hadolint-Linux-x86_64 \ + -o /bin/hadolint \ + && chmod a+rx /bin/hadolint \ + && shasum -a 512 /bin/hadolint | \ + awk '$1!="734e37c1f6619cbbd86b9b249e69c9af8ee1ea87a2b1ff71dccda412e9dac35e63425225a95d71572091a3f0a11e9a04c2fc25d9e91b840530c26af32b9891ca" {exit(1)}' + +### +# Avoid out of memory errors in builds +### +ENV MAVEN_OPTS -Xms256m -Xmx1536m + +### +# Everything past this point is either not needed for testing or breaks Yetus. +# So tell Yetus not to read the rest of the file: +# YETUS CUT HERE +### + +# Hugo static website generator (for new hadoop site and Ozone docs) +RUN curl -L -o hugo.deb https://github.com/gohugoio/hugo/releases/download/v0.30.2/hugo_0.30.2_Linux-64bit.deb \ + && dpkg --install hugo.deb \ + && rm hugo.deb + + +# Add a welcome message and environment checks. +COPY hadoop_env_checks.sh /root/hadoop_env_checks.sh +RUN chmod 755 /root/hadoop_env_checks.sh +# hadolint ignore=SC2016 +RUN echo '${HOME}/hadoop_env_checks.sh' >> /root/.bashrc diff --git a/dev-support/docker/hadoop_env_checks.sh b/dev-support/docker/hadoop_env_checks.sh new file mode 100755 index 0000000..c9420e7 --- /dev/null +++ b/dev-support/docker/hadoop_env_checks.sh @@ -0,0 +1,117 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# SHELLDOC-IGNORE + +# ------------------------------------------------------- +function showWelcome { +cat <<Welcome-message + + _ _ _ ______ +| | | | | | | _ \\ +| |_| | __ _ __| | ___ ___ _ __ | | | |_____ __ +| _ |/ _\` |/ _\` |/ _ \\ / _ \\| '_ \\ | | | / _ \\ \\ / / +| | | | (_| | (_| | (_) | (_) | |_) | | |/ / __/\\ V / +\\_| |_/\\__,_|\\__,_|\\___/ \\___/| .__/ |___/ \\___| \\_(_) + | | + |_| + +This is the standard Hadoop Developer build environment. +This has all the right tools installed required to build +Hadoop from source. + +Welcome-message +} + +# ------------------------------------------------------- + +function showAbort { + cat <<Abort-message + + ___ _ _ _ + / _ \\| | | | (_) +/ /_\\ \\ |__ ___ _ __| |_ _ _ __ __ _ +| _ | '_ \\ / _ \\| '__| __| | '_ \\ / _\` | +| | | | |_) | (_) | | | |_| | | | | (_| | +\\_| |_/_.__/ \\___/|_| \\__|_|_| |_|\\__, | + __/ | + |___/ + +Abort-message +} + +# ------------------------------------------------------- + +function failIfUserIsRoot { + if [ "$(id -u)" -eq "0" ]; # If you are root then something went wrong. + then + cat <<End-of-message + +Apparently you are inside this docker container as the user root. +Putting it simply: + + This should not occur. + +Known possible causes of this are: +1) Running this script as the root user ( Just don't ) +2) Running an old docker version ( upgrade to 1.4.1 or higher ) + +End-of-message + + showAbort + + logout + + fi +} + +# ------------------------------------------------------- + +function warnIfLowMemory { + MINIMAL_MEMORY=2046755 + INSTALLED_MEMORY=$(grep -F MemTotal /proc/meminfo | awk '{print $2}') + if [[ $((INSTALLED_MEMORY)) -lt $((MINIMAL_MEMORY)) ]]; then + cat <<End-of-message + + _ ___ ___ +| | | \\/ | +| | _____ __ | . . | ___ _ __ ___ ___ _ __ _ _ +| | / _ \\ \\ /\\ / / | |\\/| |/ _ \\ '_ \` _ \\ / _ \\| '__| | | | +| |___| (_) \\ V V / | | | | __/ | | | | | (_) | | | |_| | +\\_____/\\___/ \\_/\\_/ \\_| |_/\\___|_| |_| |_|\\___/|_| \\__, | + __/ | + |___/ + +Your system is running on very little memory. +This means it may work but it wil most likely be slower than needed. + +If you are running this via boot2docker you can simply increase +the available memory to at least ${MINIMAL_MEMORY}KiB +(you have ${INSTALLED_MEMORY}KiB ) + +End-of-message + fi +} + +# ------------------------------------------------------- + +showWelcome +warnIfLowMemory +failIfUserIsRoot + +# ------------------------------------------------------- diff --git a/hadoop-shaded-protobuf_3_7/pom.xml b/hadoop-shaded-protobuf_3_7/pom.xml new file mode 100644 index 0000000..102625c --- /dev/null +++ b/hadoop-shaded-protobuf_3_7/pom.xml @@ -0,0 +1,115 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0" + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> + <parent> + <artifactId>hadoop-thirdparty</artifactId> + <groupId>org.apache.hadoop.thirdparty</groupId> + <version>1.0.0-SNAPSHOT</version> + <relativePath>..</relativePath> + </parent> + <modelVersion>4.0.0</modelVersion> + <artifactId>hadoop-shaded-protobuf_3_7</artifactId> + <name>Apache Hadoop shaded Protobuf 3.7</name> + <packaging>jar</packaging> + + <properties> + </properties> + + <dependencies> + <dependency> + <groupId>com.google.protobuf</groupId> + <artifactId>protobuf-java</artifactId> + <version>${protobuf_3_7.version}</version> + </dependency> + </dependencies> + + <build> + <plugins> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-shade-plugin</artifactId> + <configuration> + <createDependencyReducedPom>true</createDependencyReducedPom> + </configuration> + <executions> + <execution> + <id>shade-protobuf</id> + <phase>package</phase> + <goals> + <goal>shade</goal> + </goals> + <configuration> + <artifactSet> + <includes> + <include>com.google.protobuf:protobuf-java</include> + </includes> + </artifactSet> + <filters> + <filter> + <artifact>com.google.protobuf:*</artifact> + <includes> + <include>**/*</include> + </includes> + </filter> + </filters> + <relocations> + <relocation> + <pattern>com/google/protobuf</pattern> + <shadedPattern>${shaded.prefix}.protobuf_3_7</shadedPattern> + </relocation> + <relocation> + <pattern>google/</pattern> + <shadedPattern>${shaded.prefix}.google.</shadedPattern> + <includes> + <include>**/*.proto</include> + </includes> + </relocation> + </relocations> + <transformers> + <transformer + implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer"/> + <transformer + implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer"> + <resources> + <resource>NOTICE.txt</resource> + <resource>NOTICE</resource> + <resource>LICENSE</resource> + </resources> + </transformer> + <transformer + implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer"> + <resource>META-INF/LICENSE.txt</resource> + <file>${basedir}/../LICENSE.txt</file> + </transformer> + <transformer + implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer"> + <resource>META-INF/NOTICE.txt</resource> + <file>${basedir}/../NOTICE.txt</file> + </transformer> + </transformers> + </configuration> + </execution> + </executions> + </plugin> + </plugins> + </build> +</project> \ No newline at end of file diff --git a/licenses-binary/LICENSE-protobuf.txt b/licenses-binary/LICENSE-protobuf.txt new file mode 100644 index 0000000..97a6e3d --- /dev/null +++ b/licenses-binary/LICENSE-protobuf.txt @@ -0,0 +1,32 @@ +Copyright 2008 Google Inc. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Code generated by the Protocol Buffer compiler is owned by the owner +of the input file used when generating it. This code is not +standalone and requires a support library to be linked with it. This +support library is itself covered by the above license. \ No newline at end of file diff --git a/pom.xml b/pom.xml new file mode 100644 index 0000000..155a0a2 --- /dev/null +++ b/pom.xml @@ -0,0 +1,438 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0" + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> + <modelVersion>4.0.0</modelVersion> + <groupId>org.apache.hadoop.thirdparty</groupId> + <artifactId>hadoop-thirdparty</artifactId> + <version>1.0.0-SNAPSHOT</version> + <parent> + <groupId>org.apache</groupId> + <artifactId>apache</artifactId> + <version>21</version> + <relativePath/> + <!-- no parent resolution --> + </parent> + <name>Apache Hadoop Third-party Libs</name> + <packaging>pom</packaging> + <description> + Packaging of relocated (renamed, shaded) third-party libraries used by Hadoop. + </description> + + <distributionManagement> + <repository> + <id>${distMgmtStagingId}</id> + <name>${distMgmtStagingName}</name> + <url>${distMgmtStagingUrl}</url> + </repository> + <snapshotRepository> + <id>${distMgmtSnapshotsId}</id> + <name>${distMgmtSnapshotsName}</name> + <url>${distMgmtSnapshotsUrl}</url> + </snapshotRepository> + <site> + <id>apache.website</id> + <url>scpexe://people.apache.org/www/hadoop.apache.org/docs/rthirdparty-${project.version}</url> + </site> + </distributionManagement> + + <repositories> + <repository> + <id>${distMgmtSnapshotsId}</id> + <name>${distMgmtSnapshotsName}</name> + <url>${distMgmtSnapshotsUrl}</url> + </repository> + <repository> + <id>repository.jboss.org</id> + <url>https://repository.jboss.org/nexus/content/groups/public/</url> + <snapshots> + <enabled>false</enabled> + </snapshots> + </repository> + </repositories> + + <licenses> + <license> + <name>Apache License, Version 2.0</name> + <url>https://www.apache.org/licenses/LICENSE-2.0.txt</url> + </license> + </licenses> + + <properties> + <!-- define the Java language version used by the compiler --> + <javac.version>1.8</javac.version> + + <distMgmtSnapshotsId>apache.snapshots.https</distMgmtSnapshotsId> + <distMgmtSnapshotsName>Apache Development Snapshot Repository</distMgmtSnapshotsName> + <distMgmtSnapshotsUrl>https://repository.apache.org/content/repositories/snapshots</distMgmtSnapshotsUrl> + <distMgmtStagingId>apache.staging.https</distMgmtStagingId> + <distMgmtStagingName>Apache Release Distribution Repository</distMgmtStagingName> + <distMgmtStagingUrl>https://repository.apache.org/service/local/staging/deploy/maven2</distMgmtStagingUrl> + + <!-- platform encoding override --> + <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> + <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> + + <!--thirdparty dependency versions--> + <shaded.prefix>org.apache.hadoop.thirdparty</shaded.prefix> + <protobuf_3_7.version>3.7.1</protobuf_3_7.version> + + <!-- maven plugin versions --> + <maven-deploy-plugin.version>2.8.1</maven-deploy-plugin.version> + <maven-site-plugin.version>3.6</maven-site-plugin.version> + <maven-stylus-skin.version>1.5</maven-stylus-skin.version> + <maven-antrun-plugin.version>1.7</maven-antrun-plugin.version> + <maven-assembly-plugin.version>2.5</maven-assembly-plugin.version> + <maven-dependency-plugin.version>3.0.2</maven-dependency-plugin.version> + <maven-enforcer-plugin.version>3.0.0-M1</maven-enforcer-plugin.version> + <maven-javadoc-plugin.version>3.0.1</maven-javadoc-plugin.version> + <maven-gpg-plugin.version>1.5</maven-gpg-plugin.version> + <apache-rat-plugin.version>0.12</apache-rat-plugin.version> + <wagon-ssh.version>2.4</wagon-ssh.version> + <maven-bundle-plugin.version>2.5.0</maven-bundle-plugin.version> + <maven-checkstyle-plugin.version>3.0.0</maven-checkstyle-plugin.version> + <checkstyle.version>8.19</checkstyle.version> + <dependency-check-maven.version>1.4.3</dependency-check-maven.version> + <exec-maven-plugin.version>1.3.1</exec-maven-plugin.version> + </properties> + + <organization> + <name>Apache Software Foundation</name> + <url>https://www.apache.org</url> + </organization> + + <modules> + <module>hadoop-shaded-protobuf_3_7</module> + </modules> + + <build> + <pluginManagement> + <plugins> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-dependency-plugin</artifactId> + <version>${maven-dependency-plugin.version}</version> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-enforcer-plugin</artifactId> + <version>${maven-enforcer-plugin.version}</version> + <configuration> + <rules> + <requireMavenVersion> + <version>[3.0.2,)</version> + </requireMavenVersion> + <requireJavaVersion> + <version>[1.8,)</version> + </requireJavaVersion> + </rules> + </configuration> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-assembly-plugin</artifactId> + <version>${maven-assembly-plugin.version}</version> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-deploy-plugin</artifactId> + <version>${maven-deploy-plugin.version}</version> + </plugin> + <plugin> + <groupId>org.apache.rat</groupId> + <artifactId>apache-rat-plugin</artifactId> + <version>${apache-rat-plugin.version}</version> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-antrun-plugin</artifactId> + <version>${maven-antrun-plugin.version}</version> + </plugin> + <plugin> + <groupId>org.codehaus.mojo</groupId> + <artifactId>exec-maven-plugin</artifactId> + <version>${exec-maven-plugin.version}</version> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-site-plugin</artifactId> + <version>${maven-site-plugin.version}</version> + <dependencies> + <dependency><!-- add support for ssh/scp --> + <groupId>org.apache.maven.wagon</groupId> + <artifactId>wagon-ssh</artifactId> + <version>${wagon-ssh.version}</version> + </dependency> + <dependency> + <groupId>org.apache.maven.doxia</groupId> + <artifactId>doxia-module-markdown</artifactId> + <version>1.8</version> + </dependency> + </dependencies> + </plugin> + <plugin> + <groupId>org.apache.felix</groupId> + <artifactId>maven-bundle-plugin</artifactId> + <version>${maven-bundle-plugin.version}</version> + </plugin> + <plugin> + <groupId>org.owasp</groupId> + <artifactId>dependency-check-maven</artifactId> + <version>${dependency-check-maven.version}</version> + </plugin> + </plugins> + </pluginManagement> + + <plugins> + <plugin> + <artifactId>maven-clean-plugin</artifactId> + <configuration> + <filesets> + <fileset> + <directory>${basedir}</directory> + <includes> + <include>dependency-reduced-pom.xml</include> + </includes> + </fileset> + </filesets> + </configuration> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-enforcer-plugin</artifactId> + <inherited>false</inherited> + <executions> + <execution> + <id>clean</id> + <goals> + <goal>enforce</goal> + </goals> + <phase>pre-clean</phase> + </execution> + <execution> + <id>default</id> + <goals> + <goal>enforce</goal> + </goals> + <phase>validate</phase> + </execution> + <execution> + <id>site</id> + <goals> + <goal>enforce</goal> + </goals> + <phase>pre-site</phase> + </execution> + </executions> + </plugin> + <plugin> + <groupId>org.apache.rat</groupId> + <artifactId>apache-rat-plugin</artifactId> + <configuration> + <excludes> + <exclude>.gitattributes</exclude> + <exclude>.gitignore</exclude> + <exclude>.git/**</exclude> + <exclude>.github/pull_request_template.md</exclude> + <exclude>.idea/**</exclude> + <exclude>README.md</exclude> + <exclude>**/build/**</exclude> + <exclude>**/patchprocess/**</exclude> + <exclude>**/*.js</exclude> + <exclude>licenses/**</exclude> + <exclude>licenses-binary/**</exclude> + <exclude>**/dependency-reduced-pom.xml</exclude> + </excludes> + </configuration> + </plugin> + <plugin> + <artifactId>maven-site-plugin</artifactId> + <executions> + <execution> + <id>attach-descriptor</id> + <goals> + <goal>attach-descriptor</goal> + </goals> + </execution> + </executions> + </plugin> + <plugin> + <groupId>org.apache.felix</groupId> + <artifactId>maven-bundle-plugin</artifactId> + <inherited>true</inherited> + <extensions>true</extensions> + </plugin> + <plugin> + <!-- OWASP's dependency-check plugin will scan the third party + dependencies of this project for known CVEs (security + vulnerabilities against them). It will produce a report + in target/dependency-check-report.html. To invoke, run + 'mvn dependency-check:aggregate'. Note that this plugin + requires maven 3.1.1 or greater. + --> + <groupId>org.owasp</groupId> + <artifactId>dependency-check-maven</artifactId> + <version>${dependency-check-maven.version}</version> + </plugin> + </plugins> + </build> + + <reporting> + <excludeDefaults>true</excludeDefaults> + </reporting> + + <profiles> + <profile> + <id>src</id> + <activation> + <activeByDefault>false</activeByDefault> + </activation> + <build> + <plugins> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-assembly-plugin</artifactId> + <inherited>false</inherited> + <executions> + <execution> + <id>src-dist</id> + <phase>package</phase> + <goals> + <goal>single</goal> + </goals> + <configuration> + <tarLongFileMode>posix</tarLongFileMode> + <appendAssemblyId>false</appendAssemblyId> + <attach>false</attach> + <finalName>hadoop-thirdparty-${project.version}-src</finalName> + <outputDirectory>target</outputDirectory> + <descriptors> + <descriptor>src/main/resources/assemblies/hadoop-thirdparty-src.xml</descriptor> + </descriptors> + </configuration> + </execution> + </executions> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-antrun-plugin</artifactId> + <inherited>false</inherited> + <executions> + <execution> + <id>src-dist-msg</id> + <phase>package</phase> + <goals> + <goal>run</goal> + </goals> + <configuration> + <target> + <echo/> + <echo>Hadoop Thirdparty source tar available at: ${basedir}/target/hadoop-thirdparty-${project.version}-src.tar.gz</echo> + <echo/> + </target> + </configuration> + </execution> + </executions> + </plugin> + </plugins> + </build> + </profile> + <profile> + <id>sign</id> + <build> + <plugins> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-gpg-plugin</artifactId> + <version>${maven-gpg-plugin.version}</version> + <executions> + <execution> + <id>sign-artifacts</id> + <phase>verify</phase> + <goals> + <goal>sign</goal> + </goals> + </execution> + </executions> + </plugin> + </plugins> + </build> + </profile> + <profile> + <id>releasedocs</id> + <activation> + <activeByDefault>false</activeByDefault> + </activation> + <build> + <plugins> + <plugin> + <groupId>org.codehaus.mojo</groupId> + <artifactId>exec-maven-plugin</artifactId> + <inherited>false</inherited> + <executions> + <execution> + <id>releasedocs</id> + <phase>pre-site</phase> + <goals> + <goal>exec</goal> + </goals> + <configuration> + <executable>${basedir}/dev-support/bin/releasedocmaker</executable> + <arguments> + <argument>--index</argument> + <argument>--license</argument> + <argument>--outputdir</argument> + <argument>${basedir}/src/site/markdown/release</argument> + <argument>--project</argument> + <argument>HADOOP</argument> + <argument>--projecttitle</argument> + <argument>Apache Hadoop Third-party Libs</argument> + <argument>--usetoday</argument> + <argument>--fileversions</argument> + <argument>--dirversions</argument> + <argument>--version</argument> + <argument>thirdparty-${project.version}</argument> + </arguments> + </configuration> + </execution> + </executions> + </plugin> + <plugin> + <artifactId>maven-clean-plugin</artifactId> + <configuration> + <filesets> + <fileset> + <directory>src/site/markdown/release</directory> + <includes> + <include>${project.version}/</include> + <include>index.md</include> + </includes> + <followSymlinks>false</followSymlinks> + </fileset> + </filesets> + </configuration> + </plugin> + </plugins> + </build> + </profile> + </profiles> + +</project> \ No newline at end of file diff --git a/src/main/resources/assemblies/hadoop-thirdparty-src.xml b/src/main/resources/assemblies/hadoop-thirdparty-src.xml new file mode 100644 index 0000000..ec3aceb --- /dev/null +++ b/src/main/resources/assemblies/hadoop-thirdparty-src.xml @@ -0,0 +1,62 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3" + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3 http://maven.apache.org/xsd/assembly-1.1.3.xsd"> + <id>hadoop-thirdparty-src</id> + <formats> + <format>tar.gz</format> + </formats> + <includeBaseDirectory>true</includeBaseDirectory> + <fileSets> + <fileSet> + <directory>.</directory> + <includes> + <include>LICENCE.txt</include> + <include>README.txt</include> + <include>NOTICE.txt</include> + </includes> + </fileSet> + <fileSet> + <directory>./licenses</directory> + <includes> + <include>*</include> + </includes> + </fileSet> + <fileSet> + <directory>.</directory> + <useDefaultExcludes>true</useDefaultExcludes> + <excludes> + <exclude>.git/**</exclude> + <exclude>**/.gitignore</exclude> + <exclude>**/.svn</exclude> + <exclude>**/*.iws</exclude> + <exclude>**/*.ipr</exclude> + <exclude>**/*.iml</exclude> + <exclude>**/.classpath</exclude> + <exclude>**/.project</exclude> + <exclude>**/.settings</exclude> + <exclude>**/target/**</exclude> + <!-- until the code that does this is fixed --> + <exclude>**/*.log</exclude> + <exclude>**/build/**</exclude> + <exclude>**/file:/**</exclude> + <exclude>**/SecurityAuth.audit*</exclude> + </excludes> + </fileSet> + </fileSets> +</assembly> diff --git a/src/site/markdown/index.md.vm b/src/site/markdown/index.md.vm new file mode 100644 index 0000000..adafd02 --- /dev/null +++ b/src/site/markdown/index.md.vm @@ -0,0 +1,45 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +Apache Hadoop Thirdparty Libs ${project.version} +================================ + +Apache Hadoop Thirdparty packages relocated third-party libraries used by +Apache Hadoop. + +**DISCLAIMER : This project is for Apache Hadoop internal use. Included libs +and/or their versions are subject to change at the dictate of hadoop without +regard to the concern of others!** + + +All shading is done using the same relocation offset of +**org.apache.hadoop.thirdparty**. We add this prefix to the relocated thirdparty +library class names. + +Apache Hadoop Thirdparty ${project.version} introduces following major changes. + +Overview +======== + +Users are encouraged to read the full set of release notes. +This page provides an overview of the major changes. + +Protobuf-java +------------- +Google Protobuf's 3.7.1 jar is available as *org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7* artifact. diff --git a/src/site/resources/css/site.css b/src/site/resources/css/site.css new file mode 100644 index 0000000..f830baa --- /dev/null +++ b/src/site/resources/css/site.css @@ -0,0 +1,30 @@ +/* +* Licensed to the Apache Software Foundation (ASF) under one or more +* contributor license agreements. See the NOTICE file distributed with +* this work for additional information regarding copyright ownership. +* The ASF licenses this file to You under the Apache License, Version 2.0 +* (the "License"); you may not use this file except in compliance with +* the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +*/ +#banner { + height: 93px; + background: none; +} + +#bannerLeft img { + margin-left: 30px; + margin-top: 10px; +} + +#bannerRight img { + margin: 17px; +} + diff --git a/src/site/site.xml b/src/site/site.xml new file mode 100644 index 0000000..aa6c2cb --- /dev/null +++ b/src/site/site.xml @@ -0,0 +1,59 @@ +<?xml version="1.0" encoding="ISO-8859-1"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<project name="Hadoop"> + <bannerLeft> + <name>Hadoop</name> + <src>http://hadoop.apache.org/images/hadoop-logo.jpg</src> + <href>http://hadoop.apache.org/</href> + </bannerLeft> + <bannerRight> + <src>http://www.apache.org/images/asf_logo_wide.png</src> + <href>http://www.apache.org/</href> + </bannerRight> + + <publishDate position="right"/> + <version position="right"/> + + <skin> + <groupId>org.apache.maven.skins</groupId> + <artifactId>maven-stylus-skin</artifactId> + <version>${maven-stylus-skin.version}</version> + </skin> + + <body> + <links position="left"> + <item name="Wiki" href="http://wiki.apache.org/hadoop" /> + <item name="git" href="https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git" /> + </links> + + <breadcrumbs position="left"> + <item name="Apache" href="http://www.apache.org/" /> + <item name="Hadoop" href="http://hadoop.apache.org/"/> + </breadcrumbs> + + <menu name="General" inherit="top"> + <item name="Overview" href="index.html"/> + </menu> + + <menu name="Reference" inherit="top"> + <item name="Changelog and Release Notes" href="release/index.html"/> + </menu> + + </body> + +</project> --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org