This is an automated email from the ASF dual-hosted git repository. elek pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit c8f14a560beb9a83c7d98388614a5ba36d7638f6 Author: Márton Elek <e...@apache.org> AuthorDate: Sat Oct 12 09:54:15 2019 +0000 HDDS-2287. Import common utility scripts and txt files from Hadoop without history. --- BUILDING.txt | 511 ++++++++++++++++++++++++++++++++++++++ LICENSE.txt | 258 +++++++++++++++++++ NOTICE.txt | 34 +++ README.txt | 7 + dev-support/bin/qbt | 18 ++ dev-support/bin/smart-apply-patch | 18 ++ dev-support/bin/test-patch | 18 ++ dev-support/bin/yetus-wrapper | 188 ++++++++++++++ dev-support/byteman/README.md | 31 +++ dev-support/byteman/hadooprpc.btm | 44 ++++ 10 files changed, 1127 insertions(+) diff --git a/BUILDING.txt b/BUILDING.txt new file mode 100644 index 0000000..d3c9a1a --- /dev/null +++ b/BUILDING.txt @@ -0,0 +1,511 @@ +Build instructions for Hadoop + +---------------------------------------------------------------------------------- +Requirements: + +* Unix System +* JDK 1.8 +* Maven 3.3 or later +* Protocol Buffers 3.7.1 (if compiling native code) +* CMake 3.1 or newer (if compiling native code) +* Zlib devel (if compiling native code) +* Cyrus SASL devel (if compiling native code) +* One of the compilers that support thread_local storage: GCC 4.8.1 or later, Visual Studio, + Clang (community version), Clang (version for iOS 9 and later) (if compiling native code) +* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance) +* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs) +* Doxygen ( if compiling libhdfspp and generating the documents ) +* Internet connection for first build (to fetch all Maven and Hadoop dependencies) +* python (for releasedocs) +* bats (for shell code testing) +* Node.js / bower / Ember-cli (for YARN UI v2 building) + +---------------------------------------------------------------------------------- +The easiest way to get an environment with all the appropriate tools is by means +of the provided Docker config. +This requires a recent version of docker (1.4.1 and higher are known to work). + +On Linux / Mac: + Install Docker and run this command: + + $ ./start-build-env.sh + +The prompt which is then presented is located at a mounted version of the source tree +and all required tools for testing and building have been installed and configured. + +Note that from within this docker environment you ONLY have access to the Hadoop source +tree from where you started. So if you need to run + dev-support/bin/test-patch /path/to/my.patch +then the patch must be placed inside the hadoop source tree. + +Known issues: +- On Mac with Boot2Docker the performance on the mounted directory is currently extremely slow. + This is a known problem related to boot2docker on the Mac. + See: + https://github.com/boot2docker/boot2docker/issues/593 + This issue has been resolved as a duplicate, and they point to a new feature for utilizing NFS mounts + as the proposed solution: + https://github.com/boot2docker/boot2docker/issues/64 + An alternative solution to this problem is to install Linux native inside a virtual machine + and run your IDE and Docker etc inside that VM. + +---------------------------------------------------------------------------------- +Installing required packages for clean install of Ubuntu 14.04 LTS Desktop: + +* Oracle JDK 1.8 (preferred) + $ sudo apt-get purge openjdk* + $ sudo apt-get install software-properties-common + $ sudo add-apt-repository ppa:webupd8team/java + $ sudo apt-get update + $ sudo apt-get install oracle-java8-installer +* Maven + $ sudo apt-get -y install maven +* Native libraries + $ sudo apt-get -y install build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev libsasl2-dev +* Protocol Buffers 3.7.1 (required to build native code) + $ mkdir -p /opt/protobuf-3.7-src \ + && curl -L -s -S \ + https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz \ + -o /opt/protobuf-3.7.1.tar.gz \ + && tar xzf /opt/protobuf-3.7.1.tar.gz --strip-components 1 -C /opt/protobuf-3.7-src \ + && cd /opt/protobuf-3.7-src \ + && ./configure\ + && make install \ + && rm -rf /opt/protobuf-3.7-src + +Optional packages: + +* Snappy compression + $ sudo apt-get install snappy libsnappy-dev +* Intel ISA-L library for erasure coding + Please refer to https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version + (OR https://github.com/01org/isa-l) +* Bzip2 + $ sudo apt-get install bzip2 libbz2-dev +* Linux FUSE + $ sudo apt-get install fuse libfuse-dev +* ZStandard compression + $ sudo apt-get install zstd +* PMDK library for storage class memory(SCM) as HDFS cache backend + Please refer to http://pmem.io/ and https://github.com/pmem/pmdk + +---------------------------------------------------------------------------------- +Maven main modules: + + hadoop (Main Hadoop project) + - hadoop-project (Parent POM for all Hadoop Maven modules. ) + (All plugins & dependencies versions are defined here.) + - hadoop-project-dist (Parent POM for modules that generate distributions.) + - hadoop-annotations (Generates the Hadoop doclet used to generated the Javadocs) + - hadoop-assemblies (Maven assemblies used by the different modules) + - hadoop-maven-plugins (Maven plugins used in project) + - hadoop-build-tools (Build tools like checkstyle, etc.) + - hadoop-common-project (Hadoop Common) + - hadoop-hdfs-project (Hadoop HDFS) + - hadoop-yarn-project (Hadoop YARN) + - hadoop-mapreduce-project (Hadoop MapReduce) + - hadoop-ozone (Hadoop Ozone) + - hadoop-hdds (Hadoop Distributed Data Store) + - hadoop-tools (Hadoop tools like Streaming, Distcp, etc.) + - hadoop-dist (Hadoop distribution assembler) + - hadoop-client-modules (Hadoop client modules) + - hadoop-minicluster (Hadoop minicluster artifacts) + - hadoop-cloud-storage-project (Generates artifacts to access cloud storage like aws, azure, etc.) + +---------------------------------------------------------------------------------- +Where to run Maven from? + + It can be run from any module. The only catch is that if not run from utrunk + all modules that are not part of the build run must be installed in the local + Maven cache or available in a Maven repository. + +---------------------------------------------------------------------------------- +Maven build goals: + + * Clean : mvn clean [-Preleasedocs] + * Compile : mvn compile [-Pnative] + * Run tests : mvn test [-Pnative] [-Pshelltest] + * Create JAR : mvn package + * Run findbugs : mvn compile findbugs:findbugs + * Run checkstyle : mvn compile checkstyle:checkstyle + * Install JAR in M2 cache : mvn install + * Deploy JAR to Maven repo : mvn deploy + * Run clover : mvn test -Pclover [-DcloverLicenseLocation=${user.name}/.clover.license] + * Run Rat : mvn apache-rat:check + * Build javadocs : mvn javadoc:javadoc + * Build distribution : mvn package [-Pdist][-Pdocs][-Psrc][-Pnative][-Dtar][-Preleasedocs][-Pyarn-ui] + * Change Hadoop version : mvn versions:set -DnewVersion=NEWVERSION + + Build options: + + * Use -Pnative to compile/bundle native code + * Use -Pdocs to generate & bundle the documentation in the distribution (using -Pdist) + * Use -Psrc to create a project source TAR.GZ + * Use -Dtar to create a TAR with the distribution (using -Pdist) + * Use -Preleasedocs to include the changelog and release docs (requires Internet connectivity) + * Use -Pyarn-ui to build YARN UI v2. (Requires Internet connectivity) + * Use -DskipShade to disable client jar shading to speed up build times (in + development environments only, not to build release artifacts) + + YARN Application Timeline Service V2 build options: + + YARN Timeline Service v.2 chooses Apache HBase as the primary backing storage. The supported + versions of Apache HBase are 1.2.6 (default) and 2.0.0-beta1. + + * HBase 1.2.6 is used by default to build Hadoop. The official releases are ready to use if you + plan on running Timeline Service v2 with HBase 1.2.6. + + * Use -Dhbase.profile=2.0 to build Hadoop with HBase 2.0.0-beta1. Provide this option if you plan + on running Timeline Service v2 with HBase 2.0. + + + Snappy build options: + + Snappy is a compression library that can be utilized by the native code. + It is currently an optional component, meaning that Hadoop can be built with + or without this dependency. + + * Use -Drequire.snappy to fail the build if libsnappy.so is not found. + If this option is not specified and the snappy library is missing, + we silently build a version of libhadoop.so that cannot make use of snappy. + This option is recommended if you plan on making use of snappy and want + to get more repeatable builds. + + * Use -Dsnappy.prefix to specify a nonstandard location for the libsnappy + header files and library files. You do not need this option if you have + installed snappy using a package manager. + * Use -Dsnappy.lib to specify a nonstandard location for the libsnappy library + files. Similarly to snappy.prefix, you do not need this option if you have + installed snappy using a package manager. + * Use -Dbundle.snappy to copy the contents of the snappy.lib directory into + the final tar file. This option requires that -Dsnappy.lib is also given, + and it ignores the -Dsnappy.prefix option. If -Dsnappy.lib isn't given, the + bundling and building will fail. + + + ZStandard build options: + + ZStandard is a compression library that can be utilized by the native code. + It is currently an optional component, meaning that Hadoop can be built with + or without this dependency. + + * Use -Drequire.zstd to fail the build if libzstd.so is not found. + If this option is not specified and the zstd library is missing. + + * Use -Dzstd.prefix to specify a nonstandard location for the libzstd + header files and library files. You do not need this option if you have + installed zstandard using a package manager. + + * Use -Dzstd.lib to specify a nonstandard location for the libzstd library + files. Similarly to zstd.prefix, you do not need this option if you have + installed using a package manager. + + * Use -Dbundle.zstd to copy the contents of the zstd.lib directory into + the final tar file. This option requires that -Dzstd.lib is also given, + and it ignores the -Dzstd.prefix option. If -Dzstd.lib isn't given, the + bundling and building will fail. + + OpenSSL build options: + + OpenSSL includes a crypto library that can be utilized by the native code. + It is currently an optional component, meaning that Hadoop can be built with + or without this dependency. + + * Use -Drequire.openssl to fail the build if libcrypto.so is not found. + If this option is not specified and the openssl library is missing, + we silently build a version of libhadoop.so that cannot make use of + openssl. This option is recommended if you plan on making use of openssl + and want to get more repeatable builds. + * Use -Dopenssl.prefix to specify a nonstandard location for the libcrypto + header files and library files. You do not need this option if you have + installed openssl using a package manager. + * Use -Dopenssl.lib to specify a nonstandard location for the libcrypto library + files. Similarly to openssl.prefix, you do not need this option if you have + installed openssl using a package manager. + * Use -Dbundle.openssl to copy the contents of the openssl.lib directory into + the final tar file. This option requires that -Dopenssl.lib is also given, + and it ignores the -Dopenssl.prefix option. If -Dopenssl.lib isn't given, the + bundling and building will fail. + + Tests options: + + * Use -DskipTests to skip tests when running the following Maven goals: + 'package', 'install', 'deploy' or 'verify' + * -Dtest=<TESTCLASSNAME>,<TESTCLASSNAME#METHODNAME>,.... + * -Dtest.exclude=<TESTCLASSNAME> + * -Dtest.exclude.pattern=**/<TESTCLASSNAME1>.java,**/<TESTCLASSNAME2>.java + * To run all native unit tests, use: mvn test -Pnative -Dtest=allNative + * To run a specific native unit test, use: mvn test -Pnative -Dtest=<test> + For example, to run test_bulk_crc32, you would use: + mvn test -Pnative -Dtest=test_bulk_crc32 + + Intel ISA-L build options: + + Intel ISA-L is an erasure coding library that can be utilized by the native code. + It is currently an optional component, meaning that Hadoop can be built with + or without this dependency. Note the library is used via dynamic module. Please + reference the official site for the library details. + https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version + (OR https://github.com/01org/isa-l) + + * Use -Drequire.isal to fail the build if libisal.so is not found. + If this option is not specified and the isal library is missing, + we silently build a version of libhadoop.so that cannot make use of ISA-L and + the native raw erasure coders. + This option is recommended if you plan on making use of native raw erasure + coders and want to get more repeatable builds. + * Use -Disal.prefix to specify a nonstandard location for the libisal + library files. You do not need this option if you have installed ISA-L to the + system library path. + * Use -Disal.lib to specify a nonstandard location for the libisal library + files. + * Use -Dbundle.isal to copy the contents of the isal.lib directory into + the final tar file. This option requires that -Disal.lib is also given, + and it ignores the -Disal.prefix option. If -Disal.lib isn't given, the + bundling and building will fail. + + Special plugins: OWASP's dependency-check: + + OWASP's dependency-check plugin will scan the third party dependencies + of this project for known CVEs (security vulnerabilities against them). + It will produce a report in target/dependency-check-report.html. To + invoke, run 'mvn dependency-check:aggregate'. Note that this plugin + requires maven 3.1.1 or greater. + + PMDK library build options: + + The Persistent Memory Development Kit (PMDK), formerly known as NVML, is a growing + collection of libraries which have been developed for various use cases, tuned, + validated to production quality, and thoroughly documented. These libraries are built + on the Direct Access (DAX) feature available in both Linux and Windows, which allows + applications directly load/store access to persistent memory by memory-mapping files + on a persistent memory aware file system. + + It is currently an optional component, meaning that Hadoop can be built without + this dependency. Please Note the library is used via dynamic module. For getting + more details please refer to the official sites: + http://pmem.io/ and https://github.com/pmem/pmdk. + + * -Drequire.pmdk is used to build the project with PMDK libraries forcibly. With this + option provided, the build will fail if libpmem library is not found. If this option + is not given, the build will generate a version of Hadoop with libhadoop.so. + And storage class memory(SCM) backed HDFS cache is still supported without PMDK involved. + Because PMDK can bring better caching write/read performance, it is recommended to build + the project with this option if user plans to use SCM backed HDFS cache. + * -Dpmdk.lib is used to specify a nonstandard location for PMDK libraries if they are not + under /usr/lib or /usr/lib64. + * -Dbundle.pmdk is used to copy the specified libpmem libraries into the distribution tar + package. This option requires that -Dpmdk.lib is specified. With -Dbundle.pmdk provided, + the build will fail if -Dpmdk.lib is not specified. + +---------------------------------------------------------------------------------- +Building components separately + +If you are building a submodule directory, all the hadoop dependencies this +submodule has will be resolved as all other 3rd party dependencies. This is, +from the Maven cache or from a Maven repository (if not available in the cache +or the SNAPSHOT 'timed out'). +An alternative is to run 'mvn install -DskipTests' from Hadoop source top +level once; and then work from the submodule. Keep in mind that SNAPSHOTs +time out after a while, using the Maven '-nsu' will stop Maven from trying +to update SNAPSHOTs from external repos. + +---------------------------------------------------------------------------------- +Importing projects to eclipse + +When you import the project to eclipse, install hadoop-maven-plugins at first. + + $ cd hadoop-maven-plugins + $ mvn install + +Then, generate eclipse project files. + + $ mvn eclipse:eclipse -DskipTests + +At last, import to eclipse by specifying the root directory of the project via +[File] > [Import] > [Existing Projects into Workspace]. + +---------------------------------------------------------------------------------- +Building distributions: + +Create binary distribution without native code and without documentation: + + $ mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true + +Create binary distribution with native code and with documentation: + + $ mvn package -Pdist,native,docs -DskipTests -Dtar + +Create source distribution: + + $ mvn package -Psrc -DskipTests + +Create source and binary distributions with native code and documentation: + + $ mvn package -Pdist,native,docs,src -DskipTests -Dtar + +Create a local staging version of the website (in /tmp/hadoop-site) + + $ mvn clean site -Preleasedocs; mvn site:stage -DstagingDirectory=/tmp/hadoop-site + +Note that the site needs to be built in a second pass after other artifacts. + +---------------------------------------------------------------------------------- +Installing Hadoop + +Look for these HTML files after you build the document by the above commands. + + * Single Node Setup: + hadoop-project-dist/hadoop-common/SingleCluster.html + + * Cluster Setup: + hadoop-project-dist/hadoop-common/ClusterSetup.html + +---------------------------------------------------------------------------------- + +Handling out of memory errors in builds + +---------------------------------------------------------------------------------- + +If the build process fails with an out of memory error, you should be able to fix +it by increasing the memory used by maven which can be done via the environment +variable MAVEN_OPTS. + +Here is an example setting to allocate between 256 MB and 1.5 GB of heap space to +Maven + +export MAVEN_OPTS="-Xms256m -Xmx1536m" + +---------------------------------------------------------------------------------- + +Building on macOS (without Docker) + +---------------------------------------------------------------------------------- +Installing required dependencies for clean install of macOS 10.14: + +* Install Xcode Command Line Tools + $ xcode-select --install +* Install Homebrew + $ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" +* Install OpenJDK 8 + $ brew tap AdoptOpenJDK/openjdk + $ brew cask install adoptopenjdk8 +* Install maven and tools + $ brew install maven autoconf automake cmake wget +* Install native libraries, only openssl is required to compile native code, +you may optionally install zlib, lz4, etc. + $ brew install openssl +* Protocol Buffers 3.7.1 (required to compile native code) + $ wget https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz + $ mkdir -p protobuf-3.7 && tar zxvf protobuf-java-3.7.1.tar.gz --strip-components 1 -C protobuf-3.7 + $ cd protobuf-3.7 + $ ./configure + $ make + $ make check + $ make install + $ protoc --version + +Note that building Hadoop 3.1.1/3.1.2/3.2.0 native code from source is broken +on macOS. For 3.1.1/3.1.2, you need to manually backport YARN-8622. For 3.2.0, +you need to backport both YARN-8622 and YARN-9487 in order to build native code. + +---------------------------------------------------------------------------------- +Building command example: + +* Create binary distribution with native code but without documentation: + $ mvn package -Pdist,native -DskipTests -Dmaven.javadoc.skip \ + -Dopenssl.prefix=/usr/local/opt/openssl + +Note that the command above manually specified the openssl library and include +path. This is necessary at least for Homebrewed OpenSSL. + +---------------------------------------------------------------------------------- + +Building on Windows + +---------------------------------------------------------------------------------- +Requirements: + +* Windows System +* JDK 1.8 +* Maven 3.0 or later +* Protocol Buffers 3.7.1 +* CMake 3.1 or newer +* Visual Studio 2010 Professional or Higher +* Windows SDK 8.1 (if building CPU rate control for the container executor) +* zlib headers (if building native code bindings for zlib) +* Internet connection for first build (to fetch all Maven and Hadoop dependencies) +* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These + tools must be present on your PATH. +* Python ( for generation of docs using 'mvn site') + +Unix command-line tools are also included with the Windows Git package which +can be downloaded from http://git-scm.com/downloads + +If using Visual Studio, it must be Professional level or higher. +Do not use Visual Studio Express. It does not support compiling for 64-bit, +which is problematic if running a 64-bit system. + +The Windows SDK 8.1 is available to download at: + +http://msdn.microsoft.com/en-us/windows/bg162891.aspx + +Cygwin is not required. + +---------------------------------------------------------------------------------- +Building: + +Keep the source code tree in a short path to avoid running into problems related +to Windows maximum path length limitation (for example, C:\hdc). + +There is one support command file located in dev-support called win-paths-eg.cmd. +It should be copied somewhere convenient and modified to fit your needs. + +win-paths-eg.cmd sets up the environment for use. You will need to modify this +file. It will put all of the required components in the command path, +configure the bit-ness of the build, and set several optional components. + +Several tests require that the user must have the Create Symbolic Links +privilege. + +All Maven goals are the same as described above with the exception that +native code is built by enabling the 'native-win' Maven profile. -Pnative-win +is enabled by default when building on Windows since the native components +are required (not optional) on Windows. + +If native code bindings for zlib are required, then the zlib headers must be +deployed on the build machine. Set the ZLIB_HOME environment variable to the +directory containing the headers. + +set ZLIB_HOME=C:\zlib-1.2.7 + +At runtime, zlib1.dll must be accessible on the PATH. Hadoop has been tested +with zlib 1.2.7, built using Visual Studio 2010 out of contrib\vstudio\vc10 in +the zlib 1.2.7 source tree. + +http://www.zlib.net/ + +---------------------------------------------------------------------------------- +Building distributions: + + * Build distribution with native code : mvn package [-Pdist][-Pdocs][-Psrc][-Dtar][-Dmaven.javadoc.skip=true] + +---------------------------------------------------------------------------------- +Running compatibility checks with checkcompatibility.py + +Invoke `./dev-support/bin/checkcompatibility.py` to run Java API Compliance Checker +to compare the public Java APIs of two git objects. This can be used by release +managers to compare the compatibility of a previous and current release. + +As an example, this invocation will check the compatibility of interfaces annotated as Public or LimitedPrivate: + +./dev-support/bin/checkcompatibility.py --annotation org.apache.hadoop.classification.InterfaceAudience.Public --annotation org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate --include "hadoop.*" branch-2.7.2 trunk + +---------------------------------------------------------------------------------- +Changing the Hadoop version declared returned by VersionInfo + +If for compatibility reasons the version of Hadoop has to be declared as a 2.x release in the information returned by +org.apache.hadoop.util.VersionInfo, set the property declared.hadoop.version to the desired version. +For example: mvn package -Pdist -Ddeclared.hadoop.version=2.11 + +If unset, the project version declared in the POM file is used. diff --git a/LICENSE.txt b/LICENSE.txt new file mode 100644 index 0000000..d0d5746 --- /dev/null +++ b/LICENSE.txt @@ -0,0 +1,258 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +-------------------------------------------------------------------------------- +This product bundles various third-party components under other open source +licenses. This section summarizes those components and their licenses. +See licenses/ for text of these licenses. + + +Apache Software Foundation License 2.0 +-------------------------------------- + +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/nvd3-1.8.5.* (css and js files) +hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java +hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/TimeoutFuture.java + + +BSD 2-Clause +------------ + +hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/{lz4.h,lz4.c,lz4hc.h,lz4hc.c} +hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/util/tree.h +hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/compat/{fstatat|openat|unlinkat}.h + + +BSD 3-Clause +------------ + +hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/bloom/* +hadoop-common-project/hadoop-common/src/main/native/gtest/gtest-all.cc +hadoop-common-project/hadoop-common/src/main/native/gtest/include/gtest/gtest.h +hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_x86.c +hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/d3.v3.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/d3-3.5.17.min.js + + +MIT License +----------- + +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/angular-1.6.4.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/angular-nvd3-1.0.9.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/angular-route-1.6.4.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/bootstrap-3.4.1 +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/dataTables.bootstrap.css +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/dataTables.bootstrap.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/dust-full-2.0.0.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/dust-helpers-1.1.1.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/jquery-3.4.1.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/jquery.dataTables.min.js +hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/moment.min.js +hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/bootstrap.min.js +hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/jquery.js +hadoop-tools/hadoop-sls/src/main/html/css/bootstrap.min.css +hadoop-tools/hadoop-sls/src/main/html/css/bootstrap-responsive.min.css +hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.18/* +hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jquery +hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jt/jquery.jstree.js +hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL diff --git a/NOTICE.txt b/NOTICE.txt new file mode 100644 index 0000000..f6715f7 --- /dev/null +++ b/NOTICE.txt @@ -0,0 +1,34 @@ +Apache Hadoop +Copyright 2006 and onwards The Apache Software Foundation. + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +Export Control Notice +--------------------- + +This distribution includes cryptographic software. The country in +which you currently reside may have restrictions on the import, +possession, use, and/or re-export to another country, of +encryption software. BEFORE using any encryption software, please +check your country's laws, regulations and policies concerning the +import, possession, or use, and re-export of encryption software, to +see if this is permitted. See <http://www.wassenaar.org/> for more +information. + +The U.S. Government Department of Commerce, Bureau of Industry and +Security (BIS), has classified this software as Export Commodity +Control Number (ECCN) 5D002.C.1, which includes information security +software using or performing cryptographic functions with asymmetric +algorithms. The form and manner of this Apache Software Foundation +distribution makes it eligible for export under the License Exception +ENC Technology Software Unrestricted (TSU) exception (see the BIS +Export Administration Regulations, Section 740.13) for both object +code and source code. + +The following provides more details on the included cryptographic software: + +This software uses the SSL libraries from the Jetty project written +by mortbay.org. +Hadoop Yarn Server Web Proxy uses the BouncyCastle Java +cryptography APIs written by the Legion of the Bouncy Castle Inc. diff --git a/README.txt b/README.txt new file mode 100644 index 0000000..8d37cc9 --- /dev/null +++ b/README.txt @@ -0,0 +1,7 @@ +For the latest information about Hadoop, please visit our website at: + + http://hadoop.apache.org/ + +and our wiki, at: + + https://cwiki.apache.org/confluence/display/HADOOP/ diff --git a/dev-support/bin/qbt b/dev-support/bin/qbt new file mode 100755 index 0000000..fe5e6f6 --- /dev/null +++ b/dev-support/bin/qbt @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P) +exec "${BINDIR}/yetus-wrapper" qbt --project=hadoop --skip-dir=dev-support "$@" diff --git a/dev-support/bin/smart-apply-patch b/dev-support/bin/smart-apply-patch new file mode 100755 index 0000000..3fd469f --- /dev/null +++ b/dev-support/bin/smart-apply-patch @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P) +exec "${BINDIR}/yetus-wrapper" smart-apply-patch --project=hadoop "$@" diff --git a/dev-support/bin/test-patch b/dev-support/bin/test-patch new file mode 100755 index 0000000..8ff8119 --- /dev/null +++ b/dev-support/bin/test-patch @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P) +exec "${BINDIR}/yetus-wrapper" test-patch --project=hadoop --skip-dir=dev-support "$@" diff --git a/dev-support/bin/yetus-wrapper b/dev-support/bin/yetus-wrapper new file mode 100755 index 0000000..b0f71f1 --- /dev/null +++ b/dev-support/bin/yetus-wrapper @@ -0,0 +1,188 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# you must be this high to ride the ride +if [[ -z "${BASH_VERSINFO[0]}" ]] \ + || [[ "${BASH_VERSINFO[0]}" -lt 3 ]] \ + || [[ "${BASH_VERSINFO[0]}" -eq 3 && "${BASH_VERSINFO[1]}" -lt 2 ]]; then + echo "bash v3.2+ is required. Sorry." + exit 1 +fi + +set -o pipefail + +## @description Print a message to stderr +## @audience public +## @stability stable +## @replaceable no +## @param string +function yetus_error +{ + echo "$*" 1>&2 +} + +## @description Given a filename or dir, return the absolute version of it +## @audience public +## @stability stable +## @param directory +## @replaceable no +## @return 0 success +## @return 1 failure +## @return stdout abspath +function yetus_abs +{ + declare obj=$1 + declare dir + declare fn + declare dirret + + if [[ ! -e ${obj} ]]; then + return 1 + elif [[ -d ${obj} ]]; then + dir=${obj} + else + dir=$(dirname -- "${obj}") + fn=$(basename -- "${obj}") + fn="/${fn}" + fi + + dir=$(cd -P -- "${dir}" >/dev/null 2>/dev/null && pwd -P) + dirret=$? + if [[ ${dirret} = 0 ]]; then + echo "${dir}${fn}" + return 0 + fi + return 1 +} + +function version_ge() +{ + test "$(echo "$@" | tr " " "\n" | sort -rV | head -n 1)" == "$1"; +} + +WANTED="$1" +shift +ARGV=("$@") + +HADOOP_YETUS_VERSION=${HADOOP_YETUS_VERSION:-0.10.0} +BIN=$(yetus_abs "${BASH_SOURCE-$0}") +BINDIR=$(dirname "${BIN}") + +## HADOOP_YETUS_VERSION >= 0.9.0 the tarball named with apache-yetus prefix +if version_ge "${HADOOP_YETUS_VERSION}" "0.9.0"; then + YETUS_PREFIX=apache-yetus +else + YETUS_PREFIX=yetus +fi + +### +### if YETUS_HOME is set, then try to use it +### +if [[ -n "${YETUS_HOME}" && -x "${YETUS_HOME}/bin/${WANTED}" ]]; then + exec "${YETUS_HOME}/bin/${WANTED}" "${ARGV[@]}" +fi + +# +# this directory is ignored by git and maven +# +HADOOP_PATCHPROCESS=${HADOOP_PATCHPROCESS:-"${BINDIR}/../../patchprocess"} + +if [[ ! -d "${HADOOP_PATCHPROCESS}" ]]; then + mkdir -p "${HADOOP_PATCHPROCESS}" +fi + +mytmpdir=$(yetus_abs "${HADOOP_PATCHPROCESS}") +ret=$? +if [[ ${ret} != 0 ]]; then + yetus_error "yetus-dl: Unable to cwd to ${HADOOP_PATCHPROCESS}" + exit 1 +fi +HADOOP_PATCHPROCESS=${mytmpdir} + +## +## if we've already DL'd it, then short cut +## +if [[ -x "${HADOOP_PATCHPROCESS}/${YETUS_PREFIX}-${HADOOP_YETUS_VERSION}/bin/${WANTED}" ]]; then + exec "${HADOOP_PATCHPROCESS}/${YETUS_PREFIX}-${HADOOP_YETUS_VERSION}/bin/${WANTED}" "${ARGV[@]}" +fi + +## +## need to DL, etc +## + +BASEURL="https://archive.apache.org/dist/yetus/${HADOOP_YETUS_VERSION}/" +TARBALL="${YETUS_PREFIX}-${HADOOP_YETUS_VERSION}-bin.tar" + +GPGBIN=$(command -v gpg) +CURLBIN=$(command -v curl) + +if ! pushd "${HADOOP_PATCHPROCESS}" >/dev/null; then + yetus_error "ERROR: yetus-dl: Cannot pushd to ${HADOOP_PATCHPROCESS}" + exit 1 +fi + +if [[ -n "${CURLBIN}" ]]; then + if ! "${CURLBIN}" -f -s -L -O "${BASEURL}/${TARBALL}.gz"; then + yetus_error "ERROR: yetus-dl: unable to download ${BASEURL}/${TARBALL}.gz" + exit 1 + fi +else + yetus_error "ERROR: yetus-dl requires curl." + exit 1 +fi + +if [[ -n "${GPGBIN}" ]]; then + if ! mkdir -p .gpg; then + yetus_error "ERROR: yetus-dl: Unable to create ${HADOOP_PATCHPROCESS}/.gpg" + exit 1 + fi + if ! chmod -R 700 .gpg; then + yetus_error "ERROR: yetus-dl: Unable to chmod ${HADOOP_PATCHPROCESS}/.gpg" + exit 1 + fi + if ! "${CURLBIN}" -s -L -o KEYS_YETUS https://dist.apache.org/repos/dist/release/yetus/KEYS; then + yetus_error "ERROR: yetus-dl: unable to fetch https://dist.apache.org/repos/dist/release/yetus/KEYS" + exit 1 + fi + if ! "${CURLBIN}" -s -L -O "${BASEURL}/${TARBALL}.gz.asc"; then + yetus_error "ERROR: yetus-dl: unable to fetch ${BASEURL}/${TARBALL}.gz.asc" + exit 1 + fi + if ! "${GPGBIN}" --homedir "${HADOOP_PATCHPROCESS}/.gpg" --import "${HADOOP_PATCHPROCESS}/KEYS_YETUS" >/dev/null 2>&1; then + yetus_error "ERROR: yetus-dl: gpg unable to import ${HADOOP_PATCHPROCESS}/KEYS_YETUS" + exit 1 + fi + if ! "${GPGBIN}" --homedir "${HADOOP_PATCHPROCESS}/.gpg" --verify "${TARBALL}.gz.asc" >/dev/null 2>&1; then + yetus_error "ERROR: yetus-dl: gpg verify of tarball in ${HADOOP_PATCHPROCESS} failed" + exit 1 + fi +fi + +if ! (gunzip -c "${TARBALL}.gz" | tar xpf -); then + yetus_error "ERROR: ${TARBALL}.gz is corrupt. Investigate and then remove ${HADOOP_PATCHPROCESS} to try again." + exit 1 +fi + +if [[ -x "${HADOOP_PATCHPROCESS}/${YETUS_PREFIX}-${HADOOP_YETUS_VERSION}/bin/${WANTED}" ]]; then + popd >/dev/null + exec "${HADOOP_PATCHPROCESS}/${YETUS_PREFIX}-${HADOOP_YETUS_VERSION}/bin/${WANTED}" "${ARGV[@]}" +fi + +## +## give up +## +yetus_error "ERROR: ${WANTED} is not part of Apache Yetus ${HADOOP_YETUS_VERSION}" +exit 1 diff --git a/dev-support/byteman/README.md b/dev-support/byteman/README.md new file mode 100644 index 0000000..9a17fc5 --- /dev/null +++ b/dev-support/byteman/README.md @@ -0,0 +1,31 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +This folder contains example byteman scripts (http://byteman.jboss.org/) to help +Hadoop debuging. + +As the startup script of the hadoop-runner docker image supports byteman +instrumentation it's enough to set the URL of a script to a specific environment +variable to activate it with the docker runs: + + +``` +BYTEMAN_SCRIPT_URL=https://raw.githubusercontent.com/apache/hadoop/trunk/dev-support/byteman/hadooprpc.btm +``` + +For more info see HADOOP-15656 and HDDS-342 + diff --git a/dev-support/byteman/hadooprpc.btm b/dev-support/byteman/hadooprpc.btm new file mode 100644 index 0000000..13894fe --- /dev/null +++ b/dev-support/byteman/hadooprpc.btm @@ -0,0 +1,44 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# This script instruments hadoop rpc layer to print out all the request/response messages to the standard output. +# + +RULE Hadoop RPC request +INTERFACE ^com.google.protobuf.BlockingService +METHOD callBlockingMethod +IF true +DO traceln("--> RPC message request: " + $3.getClass().getSimpleName() + " from " + linked(Thread.currentThread(), "source")); + traceln($3.toString()) +ENDRULE + + +RULE Hadoop RPC response +INTERFACE ^com.google.protobuf.BlockingService +METHOD callBlockingMethod +AT EXIT +IF true +DO traceln("--> RPC message response: " + $3.getClass().getSimpleName() + " to " + unlink(Thread.currentThread(), "source")); + traceln($!.toString()) +ENDRULE + + +RULE Hadoop RPC source IP +CLASS org.apache.hadoop.ipc.Server$RpcCall +METHOD run +IF true +DO link(Thread.currentThread(), "source", $0.connection.toString()) +ENDRULE --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-commits-h...@hadoop.apache.org