This is an automated email from the ASF dual-hosted git repository. xyao pushed a change to branch HDDS-4 in repository https://gitbox.apache.org/repos/asf/hadoop.git.
discard 57ca7be HDDS-960. Add cli command option for getS3Secret. Contributed by Dinesh Chitlangia. discard 1675e97 HDDS-597. Ratis: Support secure gRPC endpoint with mTLS for Ratis. Contributed by Ajay Kumar. discard b0f2a33 HDDS-967. Fix failures in TestOzoneConfigurationFields. Contributed by Ajay Kumar. discard 12ef5c8 HDDS-970. Fix classnotfound error for bouncy castle classes in OM,SCM init. Contributed by Ajay Kumar. discard 493f4a3 HDDS-964. Fix test failure in TestOmMetrics. Contributed by Ajay Kumar. discard a80cb21 HDDS-945. Fix generics warnings in delegation token. Contributed by Ajay Kumar. discard 54ef9c6 HDDS-963. Fix failure in TestOzoneShell due to null check in SecurityConfig. Contributed by Ajay Kumar. discard 1f78b04 HDDS-938. Add Client APIs for using S3 Auth interface. Contributed by Dinesh Chitlangia. discard 0fb151d HDDS-955. SCM CA: Add CA to SCM. Contributed by Anu Engineer. discard 23fdd5b HDDS-102. SCM CA: SCM CA server signs certificate for approved CSR. Contributed by Anu Engineer. discard dbdd77e HDDS-937. Create an S3 Auth Table. Contributed by Dinesh Chitlangia. discard 38861eb HDDS-805. Block token: Client api changes for block token. Contributed by Ajay Kumar. discard f6f75dd HDDS-929. Remove ozone.max.key.len property. Contributed by Ajay Kumar. discard 614bcda HDDS-115. GRPC: Support secure gRPC endpoint with mTLS. Contributed by Xiaoyu Yao. discard 5227f75 HDDS-884. Fix merge issue that causes NPE OzoneManager#httpServer. Contributed by Xiaoyu Yao. discard da89591 HDDS-804. Block token: Add secret token manager. Contributed by Ajay Kumar. discard abe3e64 HDDS-696. Bootstrap genesis SCM(CA) with self-signed certificate. Contributed by Anu Engineer. discard c4ffb58 HDDS-873. Fix TestSecureOzoneContainer NPE after HDDS-837. Contributed by Xiaoyu Yao. discard 74a038c Fix HDDS-4 after HDDS-751. discard b1dd4c7 Fix HDDS-4 after HDDS-759 discard a3b78ec HDDS-9. Add GRPC protocol interceptors for Ozone Block Token. Contributed by Xiaoyu Yao. discard e88f55a HDDS-8. Add OzoneManager Delegation Token support. Contributed by Ajay Kumar. discard 8fdce12 HDDS-836. Add TokenIdentifier Ozone for delegation token and block token. Contributed by Ajay Kumar. discard b94abf7 HDDS-778. Add an interface for CA and Clients for Certificate operations Contributed by Anu Engineer. discard 95e9f9a HDDS-592. Fix ozone-secure.robot test. Contributed by Ajay Kumar. discard 65d1662 HDDS-753. SCM security protocol server is not starting. Contributed by Ajay Kumar. discard b9a0d5d HDDS-760. Add asf license to TestCertificateSignRequest. Contributed by Ajay Kumar. discard eb6b8b2 HDDS-103. SCM CA: Add new security protocol for SCM to expose security related functions. Contributed by Ajay Kumar. discard 8c1eedd HDDS-101. SCM CA: generate CSR for SCM CA clients. Contributed by Xiaoyu Yao. discard 881ea2f HDDS-684. Fix HDDS-4 branch after HDDS-490 and HADOOP-15832. Contributed by Xiaoyu Yao. discard 0b248a5 HDDS-704. Fix the Dependency convergence issue on HDDS-4. Contributed by Xiaoyu Yao. discard a4624ea HDDS-591. Adding ASF license header to kadm5.acl. Contributed by Ajay Kumar. discard f43dac5 HDDS-588. SelfSignedCertificate#generateCertificate should sign the certificate the configured security provider. Contributed by Xiaoyu Yao. discard 1565894 HDDS-10. Add kdc docker image for secure ozone cluster. Contributed by Ajay Kumar. discard 04c32d6 HDDS-566. Move OzoneSecure docker-compose after HDDS-447. Contributed by Xiaoyu Yao. discard 9d78aa9 HDDS-547. Fix secure docker and configs. Contributed by Xiaoyu Yao. discard 03cdce3 HDDS-548. Create a Self-Signed Certificate. Contributed by Anu Engineer. discard c718db7 HDDS-546. Resolve bouncy castle dependency for hadoop-hdds-common. Contributed by Ajay Kumar. discard 2235ccb Fix merge conflicts discard b77e347 HDDS-100. SCM CA: generate public/private key pair for SCM/OM/DNs. Contributed by Ajay Kumar. discard 068c42d HDDS-70. Fix config names for secure ksm and scm. Contributed by Ajay Kumar. discard eca97c2 HDDS-7. Enable kerberos auth for Ozone client in hadoop rpc. Contributed by Ajay Kumar. discard ae4dbb6 Revert "Bad merge with 996a627b289947af3894bf83e7b63ec702a665cd" discard b27d6e2 HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar. discard 759b504 HDDS-5. Enable OzoneManager kerberos auth. Contributed by Ajay Kumar. discard a74dd65 HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar. add e89941f YARN-9041. Performance Optimization of method FSPreemptionThread#identifyContainersToPreempt. Contributed by Wanqiang Ji. add aa89492 Revert "HADOOP-15852. Refactor QuotaUsage. Contributed by Beluga Behr." add 9287ab3 HADOOP-15974. Upgrade Curator version to 2.13.0 to fix ZK tests. Contributed by Akira Ajisaka add 228156c Revert "YARN-8870. [Submarine] Add submarine installation scripts. (Xun Liu via wangda)" add 1dabb31 YARN-9057. Removed third party class bundle from CSI jar file. Contributed by Weiwei Yang add 1b790f4 YARN-9071. Improved status update for reinitialized containers. Contributed by Chandni Singh add 2bba329 YARN-8994. Fix race condition between move app and queue cleanup in Fair Scheduler. (Contributed by Wilfred Spiegelenburg) add b3c75c1 YARN-8985. Improve debug log in FSParentQueue when assigning container. (Contributed by Wilfred Spiegelenburg) add 912b1f9 YARN-9019. Ratio calculation of ResourceCalculator implementations could return NaN. (Contributed by Szilard Nemeth) add 5d4a432 YARN-9025. TestFairScheduler#testChildMaxResources is flaky. (Contributed by Szilard Nemeth) add c03024a HDFS-14113. EC : Add Configuration to restrict UserDefined Policies. Contributed by Ayush Saxena. add 343aaea HDDS-864. Use strongly typed codec implementations for the tables of the OmMetadataManager. Contributed by Elek Marton. add 9886c27 MAPREDUCE-7159. FrameworkUploader: ensure proper permissions of generated framework tar.gz if restrictive umask is used. Contributed by Peter Bacsko add 8d882c3 HDDS-880. Create api for ACL handling in Ozone. (Contributed by Ajay Kumar) add 019836b HDDS-858. Start a Standalone Ratis Server on OM add 6c852f2 HDDS-892. Parse aws v2 headers without spaces in Ozone s3 gateway. Contributed by Elek Marton. add cc51607 YARN-9085. Add Guaranteed and MaxCapacity to CSQueueMetrics add 154449f YARN-8914. Add xtermjs to YARN UI2. Contributed by Eric Yang and Akhil PB add 8fc0d04 HDFS-14001. [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap. Contributed by Virajith Jalaparti. add 5e773ef HDDS-909 : Default implementation for Ozone acls. Contributed by Ajay Kumar. add 1afba83 HDDS-870. Avoid creating block sized buffer in ChunkGroupOutputStream. Contributed by Shashikant Banerjee. add 1c09a10 YARN-9009. Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs. Contributed by OrDTesters. add 17a8708 HDDS-879. MultipartUpload: Add InitiateMultipartUpload in ozone. Contributed by Bharat Viswanadham. add 64411a6 YARN-8738. FairScheduler should not parse negative maxResources or minResources values as positive. (Contributed by Szilard Nemeth) add ac578c0 YARN-9087. Improve logging for initialization of Resource plugins. (Contributed by Szilard Nemeth) add 80e59e7 HDFS-14101. Fixing underflow error in test. Contributed by Zsolt Venczel. add 3ff8580 HADOOP-15428. s3guard bucket-info will create s3guard table if FS is set to do this automatically. (Contributed by Gabor Bota) add 39dc734 HDFS-14124. EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs. Contributed by Ayush Saxena. add c35de95 HADOOP-15987. ITestDynamoDBMetadataStore should check if table configured properly. Contributed by Gabor Bota. add 21c9080 HDDS-872. Add Dockerfile and skaffold config to deploy ozone dev build to k8s. Contributed by Elek, Marton. add 1a25bbe HADOOP-15845. Require explicit URI on CLI for s3guard init and destroy. Contributed by Gabor Bota. add 202926a HADOOP-15808. Harden Token service loader use. add 60af851 HDFS-14137. TestMaintenanceState fails with ArrayIndexOutOfBound Exception. Contributed by Ayush Saxena. add 881230d YARN-9051. Integrate multiple CustomResourceTypesConfigurationProvider implementations into one. (Contributed by Szilard Nemeth) add fb55e52 YARN-9008. Extend YARN distributed shell with file localization feature. (Contributed by Peter Bacsko) add 5a03c8b YARN-9114. [UI2] log service redirect url need to support user name. Contributed by Akhil PB add d44b37d HDDS-889. MultipartUpload: Support uploading a part file in ozone. Contributed by Bharat Viswanadham. add 82b7985 HADOOP-15988. DynamoDBMetadataStore#innerGet should support empty directory flag when using authoritative listings. Contributed by Gabor Bota. add 76efeac HADOOP-15995. Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping. Contributed by Lukas Majercak. add 561b61c HDFS-14144. TestPred fails in Trunk. Contributed by Ayush Saxena. add 8a53611 YARN-9078. [Submarine] Clean up the code of CliUtils#parseResourcesString. (Zhankun Tang via wangda) add c771fe6 YARN-8714. [Submarine] Support files/tarballs to be localized for a training job. (Zhankun Tang via wangda) add 61bdcb7 YARN-9015. [DevicePlugin] Add an interface for device plugin to provide customized scheduler. (Zhankun Tang via wangda) add 9fba6cc YARN-9112. [Submarine] Support polling applicationId when it's not ready in cluster. (Zhankun Tang via wangda) add 37eb919 YARN-8885. [DevicePlugin] Support NM APIs to query device resource allocation. (Zhankun Tang via wangda) add 72e7c6a YARN-8962. Add ability to use interactive shell with normal yarn container. Contributed by Eric Yang add e0d7508 MAPREDUCE-7170. Doc typo in PluggableShuffleAndPluggableSort.md. Contributed by Zhaohui Xin. add 92d44b2 HDDS-876. Add blockade tests for flaky network. Contributed by Nilotpal Nandi. add 4aa0609 HDFS-13985. Clearer error message for ReplicaNotFoundException (Contributed by Adam Antal via Daniel Templeton) add ca379e1 HDFS-13970. Use MultiMap for CacheManager Directives to simplify the code. Contributed by BELUGA BEHR. add aa12859 HDFS-14121. Log message about the old hosts file format is misleading (Contributed by Zsolt Venczel via Daniel Templeton) add cb3382f HDFS-14145. TestBlockStorageMovementAttemptedItems#testNoBlockMovementAttemptFinishedReportAdded fails sporadically in Trunk. Contributed by Ayush Saxena. add f858f18 HDFS-14088. RequestHedgingProxyProvider can throw NullPointerException when failover due to no lock on currentUsedProxy. Contributed by Yuxuan Wang. add 00d5e63 HDFS-14006. Refactor name node to allow different token verification implementations. Contributed by CR Hota. add ca90b24 YARN-9032. Support sh shell for interactive container shell at command line. Contributed by Eric Yang add b2d7204 YARN-9125. Fixed Carriage Return detection in Docker container launch command. Contributed by Billie Rinaldi add 26e55d4 YARN-9122. Add table of contents to YARN Service API document. Contributed by Zhankun Tang. add 1ea29b7 HADOOP-16000. Remove TLSv1 and SSLv2Hello from the default value of hadoop.ssl.enabled.protocols add aab3109 YARN-9089. Add Terminal Link to Service component instance page for UI2. Contributed by Eric Yang add 42297f7 YARN-8963. Add flag to disable interactive shell. Contributed by Eric Yang add 4f62233 YARN-9091. Improve terminal message when connection is refused. Contributed by Eric Yang add 04c0347 YARN-9095. Removed Unused field from Resource: NUM_MANDATORY_RESOURCES add 788e747 HDFS-14096. [SPS] : Add Support for Storage Policy Satisfier in ViewFs. Contributed by Ayush Saxena. add b2523d8 HADOOP-15969. ABFS: getNamespaceEnabled can fail blocking user access thru ACLs. add 62df607 HADOOP-15972 ABFS: reduce list page size to to 500. add 346c0c8 HADOOP-16004. ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException. add 71e0b0d YARN-9040. Fixed memory leak in LevelDBCacheTimelineStore and DBIterator. Contributed by Tarun Parimi add 5426653 HDDS-908: NPE in TestOzoneRpcClient. Contributed by Ajay Kumar. add 94b368f HDDS-99. Adding SCM Audit log. Contributed by Dinesh Chitlangia. add ee10ba2 HDDS-539. Ozone datanode command ignores the invalid options. Contributed by Vinicius Higa Murakami. add 4ff1c46 HDDS-925. Rename ChunkGroupOutputStream to KeyOutputStream and ChunkOutputStream to BlockOutputStream. Contributed by Shashikant Banerjee. add b51e9e4 HDDS-912. Update ozone to latest ratis snapshot build (0.4.0-3b0be02-SNAPSHOT). Contributed by Lokesh Jain. add 99b5061 HDDS-914. Add Grafana support to ozoneperf docker container. Contributed by Dinesh Chitlangia. add c5c7318 YARN-9072. Send exit command to terminate docker exec on connection close. Contributed by Eric Yang add 2f6bc25 HDDS-911. Make TestOzoneManager unit tests independent. Contributed by Elek Marton. add 12c139d HDDS-933. Add documentation for genconf tool under Tools section. Contributed by Dinesh Chitlangia. add a6e9d27 YARN-9117. Add a check for insecure setup for container terminal. Contributed by Eric Yang add c7a5a44 YARN-9075. Add ability to register/remove auxiliary service without restart node manager. Contributed by Billie Rinaldi add ccdd982 YARN-9084. Reset container state and defer readiness check for upgrade. Contributed by Chandni Singh add b1ce9aa HDDS-940. Remove dead store to local variable in OmMetadataManagerImpl. Contributed by Dinesh Chitlangia. add cf57113 HDDS-893. pipeline status is ALLOCATED in scmcli listPipelines command. Contributed by Lokesh Jain. add e815fd9 YARN-9126. Fix container clean up for reinitialization. Contributed by Chandni Singh add 2e544dc YARN-9129. Ensure flush after printing to log plus additional cleanup. Contributed by Eric Yang add d963575 HDFS-13661. Ls command with e option fails when the filesystem is not HDFS. add 499c70e MAPREDUCE-7166. map-only job should ignore node lost event when task is already succeeded. Contributed by Lei Li. add 5df9fb1 YARN-9130. Add Bind_HOST configuration for Yarn Web Proxy. Contributed by Rong Tang. add 2af90cc HADOOP-15935. [JDK 11] Update maven.plugin-tools.version to 3.6.0. Contributed by Dinesh Chitlangia. add ea621fa HDDS-934. Freon run hung on non-functional pipeline. Contributed by Nanda kumar. add 3961690 HADOOP-15973. Configuration: Included properties are not cached if resource is a stream. Contributed by Eric Payne add 13d3f99 HDDS-881.009. Encapsulate all client to OM requests into one request message. Contributed by Hanisha Koneru. add 784ea3a HADOOP-15965. Upgrade to ADLS SDK which has major performance improvement for ingress/egress. add a668f8e HADOOP-16015. Add bouncycastle jars to hadoop-aws as test dependencies. add 7affa30 YARN-9131. Updated document usage for dynamic auxiliary service. Contributed by Billie Rinaldi add a80d321 YARN-9152. Improved AuxServices REST API output. Contributed by Billie Rinaldi add f659485 YARN-8925. Updating distributed node attributes only when necessary. Contributed by Tao Yang. add ea72418 YARN-9132. Added file permission check for auxiliary services manifest file. Contributed by Billie Rinaldi add 2499435 HADOOP-16014. Fix test, checkstyle and javadoc issues in TestKerberosAuthenticationHandler. Contributed by Dinesh Chitlangia. add f82922d YARN-5168. Added exposed port information for Docker container. Contributed by Xun Liu add e98e963 YARN-9154. Fix itemization in YARN service quickstart document. Contributed by Ayush Saxena. add 763e96e HDFS-14166. Ls with -e option not giving the result in proper format. Contributed by Shubham Dewan. add feb2664 HDFS-13946. Log longest FSN write/read lock held stack trace. add 657aa43 YARN 9108. Fix FederationIntercepter merge home and secondary allocate response typo. Contributed by Abhishek Modi. add d944d5e HDFS-14165. In NameNode UI under DataNode tab ,the Capacity column is Non-Aligned. Contributed by Shubham Dewan. add 26e4be7 HDDS-921. Add JVM pause monitor to Ozone Daemons (OM, SCM and Datanodes). Contributed by Bharat Viswanadham. add 686fcd4 HDFS-14046. In-Maintenance ICON is missing in datanode info page. Contributed by Ranith Sardar. add 3a78e5f HDFS-12975. [SBN read] Changes to the NameNode to support reads from standby. Contributed by Chao Sun. add c9d7343 HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov. add 9bf0696 HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov. add 091ad97 HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. add a109f2b HDFS-13578. [SBN read] Add ReadOnly annotation to methods in ClientProtocol. Contributed by Chao Sun. add e880660 HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov. add c81ac2f HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen. add 151c8dd HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen. add 00e99c6 HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen. add 0105d7f HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko. add 64b7cf5 HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun. add e27708c HDFS-13665. [SBN read] Move RPC response serialization into Server.doResponse(). Contributed by Plamen Jeliazkov. add 1e22f2b HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup. Integration test, documentation, remove unnecessary dummy sync, minors fixups. Contributed by Erik Krogen. add eae0a5d HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang. add 4c9baba HDFS-13789. Reduce logging frequency of QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen. add 66ec3a6 HDFS-13767. Add msync server implementation. Contributed by Chen Liang. add 25d8e39 HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko. add f9fc01c HDFS-13782. ObserverReadProxyProvider should work with IPFailoverProxyProvider. Contributed by Konstantin Shvachko. add aa42fb0 HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen. add d2e85b0 HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang. add 28820fb HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov. add 25b63e8 HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun. add b74a7db HDFS-13898. [SBN read] Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun. add a65bb97 HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen. add b5b9b77 HDFS-13961. [SBN read] TestObserverNode refactoring. Contributed by Konstantin Shvachko. add 1c4265d HDFS-13523. Support observer nodes in MiniDFSCluster. Contributed by Konstantin Shvachko. add b6f20c3 HDFS-13925. Unit Test for transitioning between different states. Contributed by Sherwood Zheng. add 8c49135 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun. add 1a9ba96 HDFS-14016. [SBN read] ObserverReadProxyProvider should enable observer read by default. Contributed by Chen Liang. add 652b257 HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang. add a3aab48 HDFS-14017. [SBN read] ObserverReadProxyProviderWithIPFailover should work with HA configuration. Contributed by Chen Liang. add cbc2f93 HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun. add 3bb92a1 HDFS-14094. [SBN read] Fix the order of logging arguments in ObserverReadProxyProvider. Contributed by Ayush Saxena. add 71cf66e HDFS-14120. [SBN read] ORFPP should also clone DT for the virtual IP. Contributed by Chen Liang. add 2904aa3 HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun. add b8ad6c8 HDFS-14142. Move ipfailover config key out of HdfsClientConfigKeys. Contributed by Chen Liang. add b73fb70 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. add cae2796 HDFS-14138. [SBN read] Description errors in the comparison logic of transaction ID. Contributed by xiangheng. add ef3e192 HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun. add c3efc29 HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun. add fa85503 HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun. add c9d7737 HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko. add c546b12 HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in user guide. Contributed by Chao Sun. add bd2a59e HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko. add 47dd00a Merge branch 'HDFS-12943' into trunk add 742fd97 YARN-9090. [Submarine] Update submarine installation script document. Contributed by Xun Liu. add dea2c1b HDDS-951. Replace the source url of Ozone logo. Contributed by Takanobu Asanuma. add d8f670f HADOOP-15819. FileSystem cache misused in S3A integration tests. Contributed by Adam Antal. add 57198bc HADOOP-15962. The buffer size is small when unpacking tar archives. Contributed by BELUGA BEHR. add 15fe1b3 YARN-8878. Remove StringBuffer from ManagedParentQueue.java. Contributed by BELUGA BEHR. add 128f340 YARN-8894. Improve InMemoryPlan#toString. Contributed by BELUGA BEHR. add e9a005d HDFS-14171. Performance improvement in Tailing EditLog. Contributed by Kenneth Yang. add f30473f HDFS-14183. [SPS] Remove the -w parameter from the -satisfystoragepolicy usage. Contributed by Ayush Saxena. add eee29ed HADOOP-16009. Replace the url of the repository in Apache Hadoop source code. add 9e71967 MAPREDUCE-7174. Mapreduce example wordmedian should handle generic options. Contributed by Fei Hui. add 4d50ad0 HDDS-949. Update Ratis to 0.4.0-a8c4ca0-SNAPSHOT. Contributed by Xiaoyu Yao. add 13f0ee2 HADOOP-15860. ABFS: Throw exception when directory / file name ends with a period (.). Contributed by Shweta Yakkali. add 0cb3316 HDDS-482. NullPointer exception thrown on console when cli operation failed. Contributed by Nandakumar. add 996ab48 HDDS-393. Audit Parser tool for processing ozone audit logs. Contributed by Dinesh Chitlangia. add cb26f15 HADOOP-16025. Update the year to 2019. add 21fe77e HDFS-14184. [SPS] Add support for URI based path in satisfystoragepolicy command. Contributed by Ayush Saxena. add f660e5e HDFS-14163. Debug Admin Command Should Support Generic Options. Contributed by Ayush Saxena. add 040a202 HADOOP-15323. AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore. Contributed wujinhu. add cfe89e6 YARN-9164. Shutdown NM may cause NPE when opportunistic container scheduling is enabled. Contributed by lujie. add 14d232c HDDS-957. Replace incorrect use of system property user.name. Contributed by Dinesh Chitlangia. add ecdeaa7 HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh. add dfceffa YARN-9147. Rmove auxiliary services when manifest file is removed. Contributed by Billie Rinaldi add f4906ac YARN-9038. [CSI] Add ability to publish/unpublish volumes on node managers. Contributed by Weiwei Yang. add 573b158 YARN-8567. Fetching yarn logs fails for long running application if it is not present in timeline store. Contributed by Tarun Parimi. add 8c6978c YARN-6149. Allow port range to be specified while starting NM Timeline collector manager. Contributed by Abhishek Modi. add 51427cb HADOOP-15997. KMS client uses wrong UGI after HADOOP-14445. Contributed by Wei-Chiu Chuang. add ddc0a40 HDDS-896. Handle over replicated containers in SCM. Contributed by Nandakumar. add f4e1824 HADOOP-16028. Fix NetworkTopology chooseRandom function to support excluded nodes. Contributed by Sihai Ke. add 6e35f71 YARN-9166. Fix logging for preemption of Opportunistic containers for Guaranteed containers. Contributed by Abhishek Modi. add d43af8b HADOOP-15996. Improved Kerberos username mapping strategy in Hadoop. Contributed by Bolke de Bruin add 999da98 HDDS-915. Submit client request to OM Ratis server. Contributed by Hanisha Koneru. add 1f42527 Revert "HADOOP-15759. AliyunOSS: Update oss-sdk version to 3.0.0. Contributed by Jinhu Wu." add 650b9cb YARN-9178. TestRMAdminCli#testHelp is failing in trunk. Contributed by Abhishek Modi. add 8f004fe YARN-9141. [Submarine] JobStatus outputs with system UTC clock, not local clock. (Zac Zhou via wangda) add 2c02aa6 YARN-9160. [Submarine] Document 'PYTHONPATH' environment variable setting when using -localization options. (Zhankun Tang via wangda) add d3321fb Revert "YARN-9178. TestRMAdminCli#testHelp is failing in trunk. Contributed by Abhishek Modi." add f87b3b1 HADOOP-16030. AliyunOSS: bring fixes back from HADOOP-15671. Contributed by wujinhu. add 944cf87 YARN-9173. FairShare calculation broken for large values after YARN-8833. Contributed by Wilfred Spiegelenburg. add 5db7c49 YARN-9162. Fix TestRMAdminCLI#testHelp. Contributed by Ayush Saxena. add 32d5caa HADOOP-15937. [JDK 11] Update maven-shade-plugin.version to 3.2.1. Contributed by Dinesh Chitlangia. add d14c56d HDDS-916. MultipartUpload: Complete Multipart upload request. Contributed by Bharat Viswanadham. add 992dd9d HDDS-901. MultipartUpload: S3 API for Initiate multipart upload. Contributed by Bharat Viswanadham. add d66925a HDDS-902. MultipartUpload: S3 API for uploading a part file. Contributed by Bharat Viswanadham. add d715233 HADOOP-14556. S3A to support Delegation Tokens. add 802932c HADOOP-16031. Fixed TestSecureLogins unit test. Contributed by Akira Ajisaka add cdfbec4 HDDS-930. Multipart Upload: Abort multiupload request. Contributed by Bharat Viswanadham. add 06279ec HDDS-946. AuditParser - insert audit to database in batches. contributed by Dinesh Chitlangia. add 0a01d49 YARN-8822. Nvidia-docker v2 support for YARN GPU feature. (Charo Zhang via wangda) add 0f26b5e HDDS-931. Add documentation for ozone shell command providing ozone mapping for a S3Bucket. Contributed by Bharat Viswanadham. add 4894115 YARN-9169. Add metrics for queued opportunistic and guaranteed containers. Contributed by Abhishek Modi. add 7f78397 Revert "HADOOP-14556. S3A to support Delegation Tokens." add 4297e20 HDDS-926. Use Timeout rule for the the test methods in TestOzoneManager. Contributed by Dinesh Chitlangia. add 0921b70 YARN-9037. [CSI] Ignore volume resource in resource calculators based on tags. Contributed by Sunil Govindan. add 188bebb HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0. add 396ffba HDDS-968. Fix TestObjectPut failures. Contributed by Bharat Viswanadham. add 695e93c HDDS-969. Fix TestOzoneManagerRatisServer test failure. Contributed by Bharat Viswanadham. add 999f31f HDDS-924. MultipartUpload: S3 APi for complete Multipart Upload. Contributed by Bharat Viswanadham. add 1a08302 HDFS-14189. Fix intermittent failure of TestNameNodeMetrics. Contributed by Ayush Saxena. add 32cf041 HDDS-965. Ozone: checkstyle improvements and code quality scripts. Contributed by Elek, Marton. add 6a92346 YARN-6523. Optimize system credentials sent in node heartbeat responses. Contributed by Manikandan R add 4ab5260 HDFS-14132. Add BlockLocation.isStriped() to determine if block is replicated or Striped (Contributed by Shweta Yakkali via Daniel Templeton) add 709ddb1 HADOOP-15941. [JDK 11] Compilation failure: package com.sun.jndi.ldap is not visible. add 3420e26 HADOOP-16027. [DOC] Effective use of FS instances during S3A integration tests. Contributed by Gabor Bota. add 8dd11a1 HDDS-947. Implement OzoneManager State Machine. add f4617c6 Revert "HDDS-947. Implement OzoneManager State Machine." add c634589 Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." add 2091d1a HDDS-941. Rename ChunkGroupInputStream to keyInputStream and ChunkInputStream to BlockInputStream. Contributed by Shashikant Banerjee. add e8d1900 HADOOP-16040. ABFS: Bug fix for tolerateOobAppends configuration. add 7211269 HADOOP-15662. Better exception handling of DNS errors. add 852701f HADOOP-16036. WASB: Disable jetty logging configuration announcement. add 33c009a4 HADOOP-15909. KeyProvider class should implement Closeable. Contributed by Kuhu Shukla. add d4ca907 HADOOP-16016. TestSSLFactory#testServerWeakCiphers fails on Java 1.8.0_191 or upper add 9aeaaa0 HDFS-14198. Upload and Create button doesn't get enabled after getting reset. Contributed by Ayush Saxena. add dddad98 HADOOP-15975. ABFS: remove timeout check for DELETE and RENAME. add a4eefe5 HDDS-947. Implement OzoneManager State Machine. add fb8932a HADOOP-16029. Consecutive StringBuilder.append can be reused. Contributed by Ayush Saxena. add 01cb958 HADOOP-16013. DecayRpcScheduler decay thread should run as a daemon. Contributed by Erik Krogen. add bf08f4a HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian. add 35fa3bd HADOOP-16045. Don't run TestDU on Windows. Contributed by Lukas Majercak. add 3bb745d HADOOP-15994. Upgrade Jackson2 to 2.9.8. Contributed by lqjacklee. add 04fcbef HADOOP-16043. NPE in ITestDynamoDBMetadataStore when fs.s3a.s3guard.ddb.table is not set. add c4a00d1 HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found. add 6d0bffe HADOOP-14556. S3A to support Delegation Tokens. add 30863c5 HADOOP-16044. ABFS: Better exception handling of DNS errors followup add 05c84ab HDDS-977. Exclude dependency-reduced-pom.xml from ozone rat check. Contributed by Elek, Marton. add 614af50 YARN-9179. Fix NPE in AbstractYarnScheduler#updateNewContainerInfo. add ccc4362 HADOOP-16019. ZKDelegationTokenSecretManager won't log exception message occured in function setJaasConfiguration. add f280f52 HDDS-978. Fix typo in doc : Client > S3 section. Contributed by Dinesh Chitlangia. add 01405df HADOOP-15941. Addendum patch. Contributed by Takanobu Asanuma. new ff61931 HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar. new d3920ec HDDS-5. Enable OzoneManager kerberos auth. Contributed by Ajay Kumar. new 8e6743e HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar. new bfa4929 Revert "Bad merge with 996a627b289947af3894bf83e7b63ec702a665cd" new 914e93b HDDS-7. Enable kerberos auth for Ozone client in hadoop rpc. Contributed by Ajay Kumar. new e47135d HDDS-70. Fix config names for secure ksm and scm. Contributed by Ajay Kumar. new faf53f8 HDDS-100. SCM CA: generate public/private key pair for SCM/OM/DNs. Contributed by Ajay Kumar. new 570b503 Fix merge conflicts new 0b034b7 HDDS-546. Resolve bouncy castle dependency for hadoop-hdds-common. Contributed by Ajay Kumar. new 2d26944 HDDS-548. Create a Self-Signed Certificate. Contributed by Anu Engineer. new 9920506 HDDS-547. Fix secure docker and configs. Contributed by Xiaoyu Yao. new d451188 HDDS-566. Move OzoneSecure docker-compose after HDDS-447. Contributed by Xiaoyu Yao. new 0aab740 HDDS-10. Add kdc docker image for secure ozone cluster. Contributed by Ajay Kumar. new 8d7c5f4 HDDS-588. SelfSignedCertificate#generateCertificate should sign the certificate the configured security provider. Contributed by Xiaoyu Yao. new 16e0bb8 HDDS-591. Adding ASF license header to kadm5.acl. Contributed by Ajay Kumar. new e89c35a HDDS-704. Fix the Dependency convergence issue on HDDS-4. Contributed by Xiaoyu Yao. new 61e85d7 HDDS-684. Fix HDDS-4 branch after HDDS-490 and HADOOP-15832. Contributed by Xiaoyu Yao. new c260c19 HDDS-101. SCM CA: generate CSR for SCM CA clients. Contributed by Xiaoyu Yao. new 33c274e HDDS-103. SCM CA: Add new security protocol for SCM to expose security related functions. Contributed by Ajay Kumar. new 8b8a3f5 HDDS-760. Add asf license to TestCertificateSignRequest. Contributed by Ajay Kumar. new a28ad7a HDDS-753. SCM security protocol server is not starting. Contributed by Ajay Kumar. new 53120e2 HDDS-592. Fix ozone-secure.robot test. Contributed by Ajay Kumar. new 6ad794b HDDS-778. Add an interface for CA and Clients for Certificate operations Contributed by Anu Engineer. new 6d6b1a0 HDDS-836. Add TokenIdentifier Ozone for delegation token and block token. Contributed by Ajay Kumar. new bb4a26c HDDS-8. Add OzoneManager Delegation Token support. Contributed by Ajay Kumar. new 7e27706 HDDS-9. Add GRPC protocol interceptors for Ozone Block Token. Contributed by Xiaoyu Yao. new 8253106 HDDS-873. Fix TestSecureOzoneContainer NPE after HDDS-837. Contributed by Xiaoyu Yao. new 0c8829a HDDS-696. Bootstrap genesis SCM(CA) with self-signed certificate. Contributed by Anu Engineer. new 6d522dc HDDS-804. Block token: Add secret token manager. Contributed by Ajay Kumar. new 417951a HDDS-884. Fix merge issue that causes NPE OzoneManager#httpServer. Contributed by Xiaoyu Yao. new f894d86 HDDS-115. GRPC: Support secure gRPC endpoint with mTLS. Contributed by Xiaoyu Yao. new 2b11522 HDDS-929. Remove ozone.max.key.len property. Contributed by Ajay Kumar. new 50c4045 HDDS-805. Block token: Client api changes for block token. Contributed by Ajay Kumar. new ddaef67 HDDS-937. Create an S3 Auth Table. Contributed by Dinesh Chitlangia. new 924bea9 HDDS-102. SCM CA: SCM CA server signs certificate for approved CSR. Contributed by Anu Engineer. new 30bfc9c HDDS-955. SCM CA: Add CA to SCM. Contributed by Anu Engineer. new 1d5734e HDDS-938. Add Client APIs for using S3 Auth interface. Contributed by Dinesh Chitlangia. new a5d0fcf HDDS-963. Fix failure in TestOzoneShell due to null check in SecurityConfig. Contributed by Ajay Kumar. new 8978466 HDDS-945. Fix generics warnings in delegation token. Contributed by Ajay Kumar. new 0faa570 HDDS-964. Fix test failure in TestOmMetrics. Contributed by Ajay Kumar. new 0e16cf1 HDDS-970. Fix classnotfound error for bouncy castle classes in OM,SCM init. Contributed by Ajay Kumar. new 140565f HDDS-967. Fix failures in TestOzoneConfigurationFields. Contributed by Ajay Kumar. new 01a7f9e HDDS-597. Ratis: Support secure gRPC endpoint with mTLS for Ratis. Contributed by Ajay Kumar. new c0683ed HDDS-960. Add cli command option for getS3Secret. Contributed by Dinesh Chitlangia. This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (57ca7be) \ N -- N -- N refs/heads/HDDS-4 (c0683ed) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 44 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: LICENSE.txt | 33 + NOTICE.txt | 7 + .../main/resources/assemblies/hadoop-yarn-dist.xml | 12 +- .../hadoop-client-minicluster/pom.xml | 8 + .../server/KerberosAuthenticationHandler.java | 11 +- .../security/authentication/util/KerberosName.java | 87 +- .../client/TestKerberosAuthenticator.java | 1 + .../server/TestKerberosAuthenticationHandler.java | 151 +- .../authentication/util/TestKerberosName.java | 12 + .../java/org/apache/hadoop/conf/Configuration.java | 78 +- .../java/org/apache/hadoop/crypto/CipherSuite.java | 4 +- .../org/apache/hadoop/crypto/key/KeyProvider.java | 3 +- .../hadoop/crypto/key/kms/KMSClientProvider.java | 13 +- .../org/apache/hadoop/fs/AbstractFileSystem.java | 17 +- .../java/org/apache/hadoop/fs/BlockLocation.java | 13 +- .../hadoop/fs/CommonConfigurationKeysPublic.java | 7 + .../org/apache/hadoop/fs/FSDataOutputStream.java | 4 +- .../java/org/apache/hadoop/fs/FileContext.java | 18 + .../org/apache/hadoop/fs/FileEncryptionInfo.java | 32 +- .../main/java/org/apache/hadoop/fs/FileStatus.java | 34 +- .../main/java/org/apache/hadoop/fs/FileSystem.java | 10 + .../main/java/org/apache/hadoop/fs/FileUtil.java | 57 +- .../org/apache/hadoop/fs/FilterFileSystem.java | 5 + .../main/java/org/apache/hadoop/fs/FilterFs.java | 5 + .../src/main/java/org/apache/hadoop/fs/Path.java | 12 +- .../main/java/org/apache/hadoop/fs/QuotaUsage.java | 108 +- .../org/apache/hadoop/fs/StorageStatistics.java | 10 +- .../org/apache/hadoop/fs/permission/AclEntry.java | 4 +- .../java/org/apache/hadoop/fs/shell/Count.java | 4 +- .../main/java/org/apache/hadoop/fs/shell/Ls.java | 46 +- .../java/org/apache/hadoop/fs/shell/PathData.java | 6 +- .../hadoop/fs/shell/find/BaseExpression.java | 4 +- .../java/org/apache/hadoop/fs/shell/find/Find.java | 4 +- .../hadoop/fs/viewfs/ChRootedFileSystem.java | 5 + .../org/apache/hadoop/fs/viewfs/ChRootedFs.java | 5 + .../apache/hadoop/fs/viewfs/ViewFileSystem.java | 13 + .../java/org/apache/hadoop/fs/viewfs/ViewFs.java | 12 + .../org/apache/hadoop/ha/FailoverController.java | 2 +- .../main/java/org/apache/hadoop/ha/HAAdmin.java | 43 + .../org/apache/hadoop/ha/HAServiceProtocol.java | 18 + .../apache/hadoop/ha/HAServiceProtocolHelper.java | 9 + .../java/org/apache/hadoop/ha/HAServiceTarget.java | 7 + .../HAServiceProtocolClientSideTranslatorPB.java | 16 + .../HAServiceProtocolServerSideTranslatorPB.java | 20 + .../java/org/apache/hadoop/http/HttpServer2.java | 1 - .../main/java/org/apache/hadoop/io/MD5Hash.java | 4 +- .../java/org/apache/hadoop/io/SequenceFile.java | 4 +- .../io/compress/CompressionCodecFactory.java | 18 +- .../org/apache/hadoop/io/erasurecode/ECSchema.java | 8 +- .../hadoop/io/retry/RetryInvocationHandler.java | 5 + .../org/apache/hadoop/ipc/AlignmentContext.java | 96 + .../main/java/org/apache/hadoop/ipc/Client.java | 38 +- .../org/apache/hadoop/ipc/DecayRpcScheduler.java | 2 +- .../java/org/apache/hadoop/ipc/ExternalCall.java | 3 +- .../java/org/apache/hadoop/ipc/FairCallQueue.java | 32 +- .../hadoop/ipc/ObserverRetryOnActiveException.java | 37 + .../org/apache/hadoop/ipc/ProtobufRpcEngine.java | 24 +- .../src/main/java/org/apache/hadoop/ipc/RPC.java | 49 +- .../main/java/org/apache/hadoop/ipc/RpcEngine.java | 8 +- .../main/java/org/apache/hadoop/ipc/Server.java | 137 +- .../org/apache/hadoop/ipc/WritableRpcEngine.java | 59 +- .../hadoop/ipc/metrics/RpcDetailedMetrics.java | 2 +- .../org/apache/hadoop/log/LogThrottlingHelper.java | 34 + .../apache/hadoop/metrics2/sink/GraphiteSink.java | 8 +- .../apache/hadoop/metrics2/sink/StatsDSink.java | 6 +- .../hadoop/net/AbstractDNSToSwitchMapping.java | 4 +- .../main/java/org/apache/hadoop/net/NetUtils.java | 4 +- .../org/apache/hadoop/net/NetworkTopology.java | 25 +- .../apache/hadoop/security/HadoopKerberosName.java | 11 +- .../java/org/apache/hadoop/security/KDiag.java | 45 +- .../apache/hadoop/security/LdapGroupsMapping.java | 23 +- .../org/apache/hadoop/security/ProviderUtils.java | 4 +- .../hadoop/security/alias/CredentialProvider.java | 6 +- .../hadoop/security/alias/CredentialShell.java | 12 +- .../security/authorize/AccessControlList.java | 6 +- .../org/apache/hadoop/security/ssl/SSLFactory.java | 2 +- .../hadoop/security/ssl/SSLHostnameVerifier.java | 6 +- .../hadoop/security/token/DtFileOperations.java | 25 +- .../org/apache/hadoop/security/token/Token.java | 54 +- .../delegation/ZKDelegationTokenSecretManager.java | 2 +- .../service/launcher/InterruptEscalator.java | 6 +- .../org/apache/hadoop/tools/GetGroupsBase.java | 4 +- .../util/BlockingThreadPoolExecutorService.java | 6 +- .../org/apache/hadoop/util/CpuTimeTracker.java | 12 +- .../java/org/apache/hadoop/util/ProtoUtil.java | 13 + .../hadoop/util/SemaphoredDelegatingExecutor.java | 8 +- .../main/java/org/apache/hadoop/util/Shell.java | 14 +- .../java/org/apache/hadoop/util/SignalLogger.java | 4 +- .../hadoop/util/bloom/DynamicBloomFilter.java | 4 +- .../src/main/proto/HAServiceProtocol.proto | 20 + .../hadoop-common/src/main/proto/RpcHeader.proto | 2 + .../src/main/resources/core-default.xml | 84 +- .../hadoop-common/src/site/markdown/Metrics.md | 15 + .../hadoop-common/src/site/markdown/SecureMode.md | 29 +- .../org/apache/hadoop/conf/TestConfiguration.java | 100 + .../kms/TestLoadBalancingKMSClientProvider.java | 45 + .../src/test/java/org/apache/hadoop/fs/TestDU.java | 3 + .../org/apache/hadoop/fs/TestHarFileSystem.java | 2 + .../AbstractContractGetFileStatusTest.java | 4 +- .../java/org/apache/hadoop/fs/shell/TestLs.java | 67 +- .../hadoop/fs/viewfs/ViewFileSystemBaseTest.java | 5 + .../java/org/apache/hadoop/ha/DummyHAService.java | 18 +- .../java/org/apache/hadoop/ha/MiniZKFCCluster.java | 4 + .../org/apache/hadoop/ipc/TestFairCallQueue.java | 36 + .../test/java/org/apache/hadoop/ipc/TestRPC.java | 8 +- .../java/org/apache/hadoop/ipc/TestSaslRPC.java | 4 +- .../apache/hadoop/log/TestLogThrottlingHelper.java | 3 + .../org/apache/hadoop/net/TestClusterTopology.java | 35 + .../java/org/apache/hadoop/security/TestKDiag.java | 16 + .../hadoop/security/TestLdapGroupsMapping.java | 42 +- .../hadoop/security/TestUserGroupInformation.java | 36 +- .../apache/hadoop/security/ssl/TestSSLFactory.java | 3 +- .../apache/hadoop/security/token/TestToken.java | 22 +- .../org/apache/hadoop/test/LambdaTestUtils.java | 91 + .../hadoop/registry/secure/TestSecureLogins.java | 13 +- .../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 6 +- .../apache/hadoop/hdds/scm/XceiverClientRatis.java | 104 +- .../hadoop/hdds/scm/storage/BlockInputStream.java | 290 + .../hadoop/hdds/scm/storage/BlockOutputStream.java | 628 ++ .../hadoop/hdds/scm/storage/ChunkInputStream.java | 290 - .../hadoop/hdds/scm/storage/ChunkOutputStream.java | 519 -- .../java/org/apache/hadoop/hdds/HddsUtils.java | 37 +- .../org/apache/hadoop/hdds/cli/GenericCli.java | 2 +- .../hadoop/hdds/cli/HddsVersionProvider.java | 2 +- .../apache/hadoop/hdds/conf/HddsConfServlet.java | 3 + .../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 23 +- .../hadoop/hdds/scm/XceiverClientAsyncReply.java | 44 +- .../apache/hadoop/hdds/scm/XceiverClientSpi.java | 19 +- .../hadoop/hdds/scm/container/ContainerInfo.java | 1 + .../common/helpers/ContainerWithPipeline.java | 7 +- .../apache/hadoop/hdds/scm/pipeline/Pipeline.java | 38 +- .../pipeline/UnknownPipelineStateException.java | 46 + ...inerLocationProtocolClientSideTranslatorPB.java | 10 +- .../org/apache/hadoop/ozone/OzoneConfigKeys.java | 31 +- .../java/org/apache/hadoop/ozone/OzoneConsts.java | 17 + .../org/apache/hadoop/ozone/audit/SCMAction.java | 45 + .../org/apache/hadoop/ozone/common/BlockGroup.java | 8 + ...inerLocationProtocolServerSideTranslatorPB.java | 8 +- .../org/apache/hadoop/utils/HddsVersionInfo.java | 124 +- .../apache/hadoop/utils/MetadataStoreBuilder.java | 28 +- .../java/org/apache/hadoop/utils/db/Codec.java | 3 + .../org/apache/hadoop/utils/db/CodecRegistry.java | 19 + .../java/org/apache/hadoop/utils/db/DBStore.java | 13 +- .../org/apache/hadoop/utils/db/DBStoreBuilder.java | 9 +- .../java/org/apache/hadoop/utils/db/RDBStore.java | 106 +- .../org/apache/hadoop/utils/db/TypedTable.java | 3 + .../main/java/org/apache/ratis/RatisHelper.java | 2 +- hadoop-hdds/common/src/main/proto/hdds.proto | 8 +- .../common/src/main/resources/ozone-default.xml | 193 +- .../org/apache/hadoop/hdds/scm/HddsServerUtil.java | 20 +- .../apache/hadoop/ozone/HddsDatanodeService.java | 120 +- .../ozone/container/common/interfaces/Handler.java | 10 + .../common/statemachine/DatanodeStateMachine.java | 16 + .../common/statemachine/StateContext.java | 34 +- .../CloseContainerCommandHandler.java | 94 +- .../DeleteContainerCommandHandler.java | 84 + .../states/endpoint/HeartbeatEndpointTask.java | 11 + .../states/endpoint/VersionEndpointTask.java | 3 +- .../server/ratis/ContainerStateMachine.java | 10 +- .../transport/server/ratis/DispatcherContext.java | 22 +- .../transport/server/ratis/XceiverServerRatis.java | 250 +- .../ozone/container/common/volume/VolumeInfo.java | 2 +- .../ozone/container/keyvalue/KeyValueHandler.java | 62 +- .../helpers/KeyValueContainerLocationUtil.java | 13 +- .../container/ozoneimpl/ContainerController.java | 12 +- .../replication/SimpleContainerDownloader.java | 1 - .../protocol/commands/CloseContainerCommand.java | 13 +- .../commands/DeleteBlockCommandStatus.java | 12 +- .../protocol/commands/DeleteBlocksCommand.java | 6 +- .../protocol/commands/DeleteContainerCommand.java | 62 + .../commands/ReplicateContainerCommand.java | 4 - .../ozone/protocol/commands/ReregisterCommand.java | 11 +- .../hadoop/ozone/protocol/commands/SCMCommand.java | 2 +- .../hadoop/ozone/TestHddsDatanodeService.java | 3 +- .../ozone/container/common/SCMTestUtils.java | 22 +- .../container/common/interfaces/TestHandler.java | 4 +- .../TestCloseContainerCommandHandler.java | 18 +- hadoop-hdds/docs/content/AuditParser.md | 72 + hadoop-hdds/docs/content/BucketCommands.md | 9 + hadoop-hdds/docs/content/S3.md | 2 +- hadoop-hdds/docs/content/genconf.md | 28 + hadoop-hdds/docs/static/NOTES.md | 2 +- .../hadoop/hdds/server/events/EventWatcher.java | 12 +- .../hdds/scm/block/PendingDeleteHandler.java | 3 + .../scm/command/CommandStatusReportHandler.java | 12 + .../container/DeleteContainerCommandWatcher.java | 56 + .../hdds/scm/container/ReportHandlerHelper.java | 5 +- .../container/replication/ReplicationManager.java | 168 +- .../apache/hadoop/hdds/scm/events/SCMEvents.java | 14 + .../hdds/scm/node/states/Node2ObjectsMap.java | 2 + .../hdds/scm/node/states/Node2PipelineMap.java | 8 +- .../hdds/scm/pipeline/RatisPipelineUtils.java | 2 +- .../hdds/scm/pipeline/SCMPipelineManager.java | 6 +- .../hdds/scm/server/SCMBlockProtocolServer.java | 106 +- .../hdds/scm/server/SCMClientProtocolServer.java | 227 +- .../hdds/scm/server/SCMDatanodeProtocolServer.java | 111 +- .../hdds/scm/server/StorageContainerManager.java | 16 + .../java/org/apache/hadoop/hdds/scm/TestUtils.java | 1 + .../command/TestCommandStatusReportHandler.java | 8 +- .../scm/container/TestSCMContainerManager.java | 4 +- .../TestSCMContainerPlacementCapacity.java | 3 + .../TestSCMContainerPlacementRandom.java | 3 + .../replication/TestReplicationManager.java | 95 +- .../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 4 +- .../hdds/scm/node/TestNodeReportHandler.java | 3 + .../src/main/java/org/apache/hadoop/fs/Hdfs.java | 5 + .../org/apache/hadoop/fs/HdfsBlockLocation.java | 4 + .../org/apache/hadoop/hdfs/ClientGSIContext.java | 90 + .../java/org/apache/hadoop/hdfs/DFSClient.java | 27 + .../org/apache/hadoop/hdfs/DFSInputStream.java | 12 +- .../java/org/apache/hadoop/hdfs/DFSUtilClient.java | 8 +- .../apache/hadoop/hdfs/DistributedFileSystem.java | 6 +- .../apache/hadoop/hdfs/NameNodeProxiesClient.java | 23 +- .../hadoop/hdfs/protocol/ClientProtocol.java | 67 + .../apache/hadoop/hdfs/protocol/DatanodeInfo.java | 83 +- .../hadoop/hdfs/protocol/HdfsPathHandle.java | 4 +- .../hadoop/hdfs/protocol/ReencryptionStatus.java | 12 +- .../ClientNamenodeProtocolTranslatorPB.java | 39 + .../server/datanode/ReplicaNotFoundException.java | 8 +- .../ha/AbstractNNFailoverProxyProvider.java | 19 +- .../server/namenode/ha/ClientHAProxyFactory.java | 12 + .../namenode/ha/IPFailoverProxyProvider.java | 2 +- .../namenode/ha/ObserverReadProxyProvider.java | 357 + .../ObserverReadProxyProviderWithIPFailover.java | 142 + .../hadoop/hdfs/server/namenode/ha/ReadOnly.java | 57 + .../namenode/ha/RequestHedgingProxyProvider.java | 209 +- .../hadoop/hdfs/util/CombinedHostsFileReader.java | 41 +- .../apache/hadoop/hdfs/util/StripedBlockUtil.java | 10 +- .../org/apache/hadoop/hdfs/web/JsonUtilClient.java | 11 + .../apache/hadoop/hdfs/web/WebHdfsFileSystem.java | 33 + .../hadoop/hdfs/web/resources/GetOpParam.java | 2 + .../hadoop/hdfs/web/resources/PostOpParam.java | 2 + .../hadoop/hdfs/web/resources/PutOpParam.java | 1 + .../src/main/proto/ClientNamenodeProtocol.proto | 18 + .../apache/hadoop/hdfs/protocol/TestReadOnly.java | 103 + .../ha/TestRequestHedgingProxyProvider.java | 60 + .../main/native/libhdfspp/lib/proto/CMakeLists.txt | 1 + .../resolver/FederationNamenodeServiceState.java | 3 + .../server/federation/resolver/PathLocation.java | 6 +- .../federation/router/ConnectionContext.java | 10 +- .../federation/router/RouterClientProtocol.java | 13 + .../server/federation/router/RouterQuotaUsage.java | 4 +- .../server/federation/router/RouterRpcServer.java | 12 + .../java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 7 + .../hadoop/hdfs/protocol/CacheDirective.java | 3 +- ...ientNamenodeProtocolServerSideTranslatorPB.java | 49 + .../apache/hadoop/hdfs/protocolPB/PBHelper.java | 4 + .../hadoop/hdfs/qjournal/client/AsyncLogger.java | 7 + .../hdfs/qjournal/client/AsyncLoggerSet.java | 14 + .../hdfs/qjournal/client/IPCLoggerChannel.java | 14 + .../hdfs/qjournal/client/QuorumJournalManager.java | 137 +- .../hdfs/qjournal/client/QuorumOutputStream.java | 13 +- .../hdfs/qjournal/protocol/QJournalProtocol.java | 24 +- .../QJournalProtocolServerSideTranslatorPB.java | 14 + .../protocolPB/QJournalProtocolTranslatorPB.java | 20 + .../hadoop/hdfs/qjournal/server/Journal.java | 65 + .../hdfs/qjournal/server/JournalMetrics.java | 20 +- .../hdfs/qjournal/server/JournalNodeRpcServer.java | 8 + .../hdfs/qjournal/server/JournaledEditsCache.java | 417 + .../hdfs/server/aliasmap/InMemoryAliasMap.java | 216 +- .../aliasmap/InMemoryLevelDBAliasMapServer.java | 8 + .../hdfs/server/blockmanagement/BlockManager.java | 3 +- .../blockmanagement/BlockManagerSafeMode.java | 7 +- .../blockmanagement/DatanodeAdminManager.java | 3 +- .../hdfs/server/common/HdfsServerConstants.java | 4 +- .../hadoop/hdfs/server/common/JspHelper.java | 8 +- .../hadoop/hdfs/server/common/TokenVerifier.java | 35 + .../hdfs/server/datanode/BPServiceActor.java | 2 +- .../hadoop/hdfs/server/datanode/DataNode.java | 3 +- .../hadoop/hdfs/server/datanode/VolumeScanner.java | 18 +- .../server/diskbalancer/command/PlanCommand.java | 3 +- .../hadoop/hdfs/server/namenode/CacheManager.java | 38 +- .../server/namenode/EditLogFileInputStream.java | 44 + .../server/namenode/EncryptionZoneManager.java | 7 +- .../namenode/ErasureCodingPolicyManager.java | 13 + .../hadoop/hdfs/server/namenode/FSEditLog.java | 20 +- .../hdfs/server/namenode/FSEditLogLoader.java | 51 +- .../hadoop/hdfs/server/namenode/FSEditLogOp.java | 734 +- .../hadoop/hdfs/server/namenode/FSImage.java | 19 +- .../hadoop/hdfs/server/namenode/FSNamesystem.java | 66 +- .../hdfs/server/namenode/FSNamesystemLock.java | 93 +- .../hdfs/server/namenode/GlobalStateIdContext.java | 158 + .../hadoop/hdfs/server/namenode/ImageServlet.java | 32 +- .../hadoop/hdfs/server/namenode/JournalSet.java | 4 +- .../hadoop/hdfs/server/namenode/NameNode.java | 61 +- .../hdfs/server/namenode/NameNodeHttpServer.java | 21 + .../hdfs/server/namenode/NameNodeRpcServer.java | 22 +- .../hadoop/hdfs/server/namenode/NamenodeFsck.java | 10 +- .../server/namenode/QuotaByStorageTypeEntry.java | 6 +- .../namenode/RedundantEditLogInputStream.java | 19 +- .../hdfs/server/namenode/StoragePolicySummary.java | 13 +- .../hdfs/server/namenode/TransferFsImage.java | 21 +- .../hdfs/server/namenode/ha/BootstrapStandby.java | 91 +- .../hdfs/server/namenode/ha/EditLogTailer.java | 8 +- .../hdfs/server/namenode/ha/StandbyState.java | 19 +- .../server/namenode/metrics/NameNodeMetrics.java | 6 + .../web/resources/NamenodeWebHdfsMethods.java | 15 +- .../hadoop/hdfs/server/protocol/ServerCommand.java | 6 +- .../hadoop/hdfs/tools/DFSZKFailoverController.java | 4 +- .../org/apache/hadoop/hdfs/tools/DebugAdmin.java | 8 +- .../hadoop/hdfs/tools/NNHAServiceTarget.java | 5 + .../hadoop/hdfs/tools/StoragePolicyAdmin.java | 20 +- .../hadoop-hdfs/src/main/proto/HdfsServer.proto | 1 + .../src/main/proto/QJournalProtocol.proto | 18 + .../src/main/resources/hdfs-default.xml | 24 +- .../src/main/webapps/hdfs/dfshealth.html | 3 +- .../src/main/webapps/hdfs/explorer.html | 4 +- .../hadoop-hdfs/src/main/webapps/hdfs/explorer.js | 28 + .../hadoop-hdfs/src/main/webapps/static/hadoop.css | 12 +- .../hadoop-hdfs/src/site/markdown/HDFSCommands.md | 2 + .../site/markdown/HDFSHighAvailabilityWithQJM.md | 35 +- .../src/site/markdown/ObserverNameNode.md | 187 + .../hadoop-hdfs/src/site/markdown/WebHDFS.md | 89 +- .../java/org/apache/hadoop/hdfs/DFSTestUtil.java | 12 +- .../org/apache/hadoop/hdfs/MiniDFSCluster.java | 39 +- .../hadoop/hdfs/TestErasureCodingAddConfig.java | 79 + .../apache/hadoop/hdfs/TestMaintenanceState.java | 2 + .../java/org/apache/hadoop/hdfs/TestPread.java | 3 +- .../hdfs/TestStateAlignmentContextWithHA.java | 347 + .../hadoop/hdfs/qjournal/MiniQJMHACluster.java | 4 +- .../qjournal/client/TestQuorumJournalManager.java | 134 + .../client/TestQuorumJournalManagerUnit.java | 103 +- .../hdfs/qjournal/server/JournalTestUtil.java | 48 + .../hadoop/hdfs/qjournal/server/TestJournal.java | 47 + .../qjournal/server/TestJournaledEditsCache.java | 257 + .../server/aliasmap/ITestInMemoryAliasMap.java | 30 + .../hdfs/server/aliasmap/TestInMemoryAliasMap.java | 45 - .../impl/TestInMemoryLevelDBAliasMapClient.java | 9 - .../server/namenode/NNThroughputBenchmark.java | 6 +- .../hdfs/server/namenode/NameNodeAdapter.java | 13 + .../hdfs/server/namenode/TestAddStripedBlocks.java | 24 + .../namenode/TestEditLogFileInputStream.java | 18 + .../hdfs/server/namenode/TestFSEditLogLoader.java | 48 + .../hdfs/server/namenode/TestFSNamesystemLock.java | 41 +- .../hadoop/hdfs/server/namenode/TestFsck.java | 2 +- .../server/namenode/TestListCorruptFileBlocks.java | 24 +- .../hadoop/hdfs/server/namenode/ha/HATestUtil.java | 126 +- .../server/namenode/ha/TestBootstrapAliasmap.java | 99 + .../namenode/ha/TestConsistentReadsObserver.java | 235 + .../server/namenode/ha/TestMultiObserverNode.java | 159 + .../hdfs/server/namenode/ha/TestObserverNode.java | 364 + .../namenode/ha/TestObserverReadProxyProvider.java | 403 + .../namenode/ha/TestStandbyInProgressTail.java | 164 +- .../namenode/metrics/TestNameNodeMetrics.java | 4 +- .../TestBlockStorageMovementAttemptedItems.java | 1 - .../apache/hadoop/hdfs/tools/TestDFSHAAdmin.java | 19 +- .../hdfs/tools/TestDFSHAAdminMiniCluster.java | 44 + .../TestStoragePolicySatisfyAdminCommands.java | 22 + .../org/apache/hadoop/hdfs/web/TestWebHDFS.java | 18 +- .../mapreduce/v2/app/job/impl/TaskAttemptImpl.java | 15 + .../mapreduce/v2/app/job/impl/TestTaskAttempt.java | 122 +- .../v2/app/rm/TestRMContainerAllocator.java | 2 +- .../main/java/org/apache/hadoop/mapreduce/Job.java | 10 +- .../markdown/PluggableShuffleAndPluggableSort.md | 75 +- .../org/apache/hadoop/mapred/TestYARNRunner.java | 44 +- .../org/apache/hadoop/mapred/ShuffleHandler.java | 3 + .../hadoop-mapreduce-client-uploader/pom.xml | 6 + .../hadoop/mapred/uploader/FrameworkUploader.java | 52 + .../mapred/uploader/TestFrameworkUploader.java | 58 + .../org/apache/hadoop/examples/WordMedian.java | 13 +- hadoop-maven-plugins/pom.xml | 2 +- hadoop-ozone/Jenkinsfile | 105 + .../apache/hadoop/ozone/client/OzoneBucket.java | 76 + .../hadoop/ozone/client/OzoneKeyDetails.java | 1 + .../apache/hadoop/ozone/client/OzoneVolume.java | 6 +- .../ozone/client/io/BlockOutputStreamEntry.java | 359 + .../ozone/client/io/ChunkGroupInputStream.java | 338 - .../ozone/client/io/ChunkGroupOutputStream.java | 645 -- .../ozone/client/io/ChunkOutputStreamEntry.java | 352 - .../hadoop/ozone/client/io/KeyInputStream.java | 338 + .../hadoop/ozone/client/io/KeyOutputStream.java | 667 ++ .../hadoop/ozone/client/io/OzoneInputStream.java | 6 +- .../hadoop/ozone/client/io/OzoneOutputStream.java | 14 +- .../ozone/client/protocol/ClientProtocol.java | 59 + .../hadoop/ozone/client/rest/RestClient.java | 41 + .../apache/hadoop/ozone/client/rpc/RpcClient.java | 130 +- hadoop-ozone/common/src/main/bin/ozone | 6 + .../apache/hadoop/hdds/protocol/package-info.java | 20 + .../main/java/org/apache/hadoop/ozone/OmUtils.java | 52 +- .../org/apache/hadoop/ozone/audit/OMAction.java | 5 +- .../apache/hadoop/ozone/freon/OzoneGetConf.java | 2 + .../org/apache/hadoop/ozone/om/OMConfigKeys.java | 89 + .../apache/hadoop/ozone/om/OMMetadataManager.java | 58 +- .../hadoop/ozone/om/codec/OmBucketInfoCodec.java | 52 + .../hadoop/ozone/om/codec/OmKeyInfoCodec.java | 52 + .../ozone/om/codec/OmMultipartKeyInfoCodec.java | 56 + .../hadoop/ozone/om/codec/OmVolumeArgsCodec.java | 52 + .../hadoop/ozone/om/codec/VolumeListCodec.java | 51 + .../apache/hadoop/ozone/om/codec/package-info.java | 24 + .../hadoop/ozone/om/helpers/OmBucketArgs.java | 7 +- .../hadoop/ozone/om/helpers/OmBucketInfo.java | 17 +- .../apache/hadoop/ozone/om/helpers/OmKeyArgs.java | 43 +- .../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 31 +- .../helpers/OmMultipartCommitUploadPartInfo.java | 34 + .../hadoop/ozone/om/helpers/OmMultipartInfo.java | 77 + .../ozone/om/helpers/OmMultipartKeyInfo.java | 106 + .../om/helpers/OmMultipartUploadCompleteInfo.java | 70 + .../ozone/om/helpers/OmMultipartUploadList.java | 63 + .../hadoop/ozone/om/helpers/OmVolumeArgs.java | 3 +- .../ozone/om/protocol/OzoneManagerProtocol.java | 41 + ...OzoneManagerProtocolClientSideTranslatorPB.java | 692 +- .../ozone/security/acl/IAccessAuthorizer.java | 153 + .../hadoop/ozone/security/acl/IOzoneObj.java | 24 + .../ozone/security/acl/OzoneAccessAuthorizer.java | 29 + .../ozone/security/acl/OzoneAclException.java | 71 + .../apache/hadoop/ozone/security/acl/OzoneObj.java | 109 + .../hadoop/ozone/security/acl/OzoneObjInfo.java | 118 + .../hadoop/ozone/security/acl/RequestContext.java | 122 + .../hadoop/ozone/security/acl/package-info.java | 22 + .../apache/hadoop/ozone/util/OzoneVersionInfo.java | 137 +- .../org/apache/hadoop/ozone/util/package-info.java | 22 + .../hadoop/ozone/web/handlers/VolumeArgs.java | 1 + .../src/main/proto/OzoneManagerProtocol.proto | 465 +- .../om/codec/TestOmMultipartKeyInfoCodec.java | 54 + .../apache/hadoop/ozone/om/codec/package-info.java | 24 + .../ozone/security/acl/TestOzoneObjInfo.java | 88 + hadoop-ozone/dev-support/checks/acceptance.sh | 18 + hadoop-ozone/dev-support/checks/author.sh | 22 + hadoop-ozone/dev-support/checks/build.sh | 18 + hadoop-ozone/dev-support/checks/checkstyle.sh | 23 + hadoop-ozone/dev-support/checks/findbugs.sh | 35 + hadoop-ozone/dev-support/checks/isolation.sh | 24 + hadoop-ozone/dev-support/checks/rat.sh | 24 + hadoop-ozone/dev-support/checks/unit.sh | 24 + hadoop-ozone/dev-support/docker/Dockerfile | 66 + hadoop-ozone/dist/Dockerfile | 21 + hadoop-ozone/dist/README.md | 85 + .../dist/dev-support/bin/dist-layout-stitching | 9 + hadoop-ozone/dist/skaffold.yaml | 25 + hadoop-ozone/dist/src/main/blockade/README.md | 26 + .../src/main/blockade/blockadeUtils/__init__.py | 14 + .../src/main/blockade/blockadeUtils/blockade.py | 59 + .../src/main/blockade/clusterUtils/__init__.py | 0 .../main/blockade/clusterUtils/cluster_utils.py | 75 + hadoop-ozone/dist/src/main/blockade/conftest.py | 65 + .../dist/src/main/blockade/test_blockade.py | 54 + .../main/compose/common/grafana/conf/grafana.ini | 20 + .../grafana/dashboards/Ozone - Object Metrics.json | 1344 +++ .../grafana/dashboards/Ozone - RPC Metrics.json | 875 ++ .../grafana/provisioning/dashboards/dashboards.yml | 22 + .../provisioning/datasources/datasources.yml | 25 + .../prometheus}/prometheus.yml | 0 .../dist/src/main/compose/ozoneperf/README.md | 16 +- .../src/main/compose/ozoneperf/docker-compose.yaml | 11 +- .../dist/src/main/compose/ozoneperf/docker-config | 3 +- .../dist/src/main/conf/scm-audit-log4j2.properties | 90 + .../dist/src/main/k8s/ozone/config-configmap.yaml | 34 + .../main/k8s/ozone/datanode-public-service.yaml | 29 + .../dist/src/main/k8s/ozone/datanode-service.yaml | 29 + .../src/main/k8s/ozone/datanode-statefulset.yaml | 52 + .../dist/src/main/k8s/ozone/om-public-service.yaml | 29 + .../dist/src/main/k8s/ozone/om-service.yaml | 29 + .../dist/src/main/k8s/ozone/om-statefulset.yaml | 72 + .../src/main/k8s/ozone/s3g-public-service.yaml | 29 + .../dist/src/main/k8s/ozone/s3g-service.yaml | 29 + .../dist/src/main/k8s/ozone/s3g-statefulset.yaml | 49 + .../src/main/k8s/ozone/scm-public-service.yaml | 29 + .../dist/src/main/k8s/ozone/scm-service.yaml | 29 + .../dist/src/main/k8s/ozone/scm-statefulset.yaml | 67 + .../dist/src/main/k8s/prometheus/configmap.yaml | 51 + .../dist/src/main/k8s/prometheus/deployment.yaml | 46 + .../k8s/prometheus/prometheus-public-serivce.yaml | 28 + .../dist/src/main/k8s/prometheus/role.yaml | 31 + .../dist/src/main/k8s/prometheus/rolebindng.yaml | 27 + .../main/k8s/prometheus/scm-public-service.yaml | 29 + .../src/main/k8s/prometheus/service-account.yaml | 19 + .../dist/src/main/k8s/prometheus/service.yaml | 25 + .../src/main/smoketest/s3/MultipartUpload.robot | 152 + .../hdds/scm/pipeline/TestNode2PipelineMap.java | 7 +- .../org/apache/hadoop/ozone/MiniOzoneCluster.java | 12 +- .../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 16 +- .../org/apache/hadoop/ozone/OzoneTestUtils.java | 3 +- .../hadoop/ozone/TestSecureOzoneCluster.java | 63 +- .../rpc/TestCloseContainerHandlingByClient.java | 73 +- .../rpc/TestContainerStateMachineFailures.java | 6 +- .../client/rpc/TestFailureHandlingByClient.java | 70 +- .../ozone/client/rpc/TestOzoneRpcClient.java | 1267 +-- .../client/rpc/TestOzoneRpcClientAbstract.java | 1822 ++++ .../client/rpc/TestOzoneRpcClientWithRatis.java | 58 + .../TestCloseContainerByPipeline.java | 9 +- .../transport/server/ratis/TestCSMMetrics.java | 2 +- .../container/ozoneimpl/TestOzoneContainer.java | 5 +- .../ozoneimpl/TestOzoneContainerRatis.java | 2 +- .../container/server/TestContainerServer.java | 3 +- .../org/apache/hadoop/ozone/om/TestOmAcls.java | 176 + .../org/apache/hadoop/ozone/om/TestOmMetrics.java | 18 +- .../apache/hadoop/ozone/om/TestOzoneManager.java | 87 +- .../ozone/ozShell/TestOzoneDatanodeShell.java | 250 + .../hadoop/ozone/ozShell/TestOzoneShell.java | 1 + .../hadoop/ozone/scm/TestAllocateContainer.java | 9 +- .../ozone/web/TestOzoneRestWithMiniCluster.java | 3 +- .../hadoop/ozone/web/client/TestBuckets.java | 43 +- .../hadoop/ozone/web/client/TestBucketsRatis.java | 1 + .../apache/hadoop/ozone/web/client/TestVolume.java | 78 +- .../hadoop/ozone/web/client/TestVolumeRatis.java | 1 + .../hdfs/server/datanode/ObjectStoreHandler.java | 4 +- .../apache/hadoop/ozone/web/interfaces/Bucket.java | 1 + .../apache/hadoop/ozone/web/interfaces/Volume.java | 1 + .../web/storage/DistributedStorageHandler.java | 10 +- .../org/apache/hadoop/ozone/om/BucketManager.java | 2 +- .../apache/hadoop/ozone/om/BucketManagerImpl.java | 37 +- .../apache/hadoop/ozone/om/KeyDeletingService.java | 3 +- .../org/apache/hadoop/ozone/om/KeyManager.java | 40 + .../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 583 +- .../java/org/apache/hadoop/ozone/om/OMMetrics.java | 60 + .../java/org/apache/hadoop/ozone/om/OMStorage.java | 3 +- .../hadoop/ozone/om/OmMetadataManagerImpl.java | 215 +- .../org/apache/hadoop/ozone/om/OzoneManager.java | 361 +- .../org/apache/hadoop/ozone/om/VolumeManager.java | 2 +- .../apache/hadoop/ozone/om/VolumeManagerImpl.java | 109 +- .../hadoop/ozone/om/exceptions/OMException.java | 11 +- .../hadoop/ozone/om/ratis/OMRatisHelper.java | 115 + .../ozone/om/ratis/OzoneManagerRatisClient.java | 194 + .../ozone/om/ratis/OzoneManagerRatisServer.java | 319 + .../ozone/om/ratis/OzoneManagerStateMachine.java | 157 + .../apache/hadoop/ozone/om/ratis/package-info.java | 22 + ...OzoneManagerProtocolServerSideTranslatorPB.java | 713 +- .../protocolPB/OzoneManagerRequestHandler.java | 1022 +++ .../web/ozShell/volume/CreateVolumeHandler.java | 2 +- .../web/ozShell/volume/ListVolumeHandler.java | 3 +- .../hadoop/ozone/om/TestBucketManagerImpl.java | 50 +- .../apache/hadoop/ozone/om/TestChunkStreams.java | 96 +- .../hadoop/ozone/om/TestKeyDeletingService.java | 41 +- .../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 187 +- .../om/ratis/TestOzoneManagerRatisServer.java | 140 + .../apache/hadoop/fs/ozone/OzoneFSInputStream.java | 6 +- .../hadoop/fs/ozone/OzoneFSOutputStream.java | 6 +- hadoop-ozone/pom.xml | 2 + .../apache/hadoop/ozone/s3/HeaderPreprocessor.java | 8 + .../hadoop/ozone/s3/SignedChunksInputStream.java | 2 +- .../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 1 + .../endpoint/CompleteMultipartUploadRequest.java | 77 + .../endpoint/CompleteMultipartUploadResponse.java | 78 + .../endpoint/MultipartUploadInitiateResponse.java | 69 + .../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 176 + .../hadoop/ozone/s3/exception/S3ErrorTable.java | 21 + .../ozone/s3/header/AuthorizationHeaderV4.java | 28 +- .../hadoop/ozone/s3/io/S3WrapperInputStream.java | 10 +- .../hadoop/ozone/client/OzoneBucketStub.java | 96 + .../hadoop/ozone/client/OzoneOutputStreamStub.java | 73 + .../s3/endpoint/TestInitiateMultipartUpload.java | 79 + .../s3/endpoint/TestMultipartUploadComplete.java | 222 + .../ozone/s3/endpoint/TestObjectMultiDelete.java | 1 - .../hadoop/ozone/s3/endpoint/TestObjectPut.java | 26 +- .../hadoop/ozone/s3/endpoint/TestPartUpload.java | 126 + .../ozone/s3/header/TestAuthorizationHeaderV4.java | 60 +- .../hadoop/ozone/audit/parser/AuditParser.java | 55 + .../ozone/audit/parser/common/DatabaseHelper.java | 243 + .../ozone/audit/parser/common/ParserConsts.java | 35 + .../ozone/audit/parser/common/package-info.java | 20 + .../audit/parser/handler/LoadCommandHandler.java | 52 + .../audit/parser/handler/QueryCommandHandler.java | 57 + .../parser/handler/TemplateCommandHandler.java | 61 + .../ozone/audit/parser/handler/package-info.java | 20 + .../ozone/audit/parser/model/AuditEntry.java | 188 + .../ozone/audit/parser/model/package-info.java | 20 + .../hadoop/ozone/audit/parser/package-info.java | 20 + .../hadoop/ozone/freon/RandomKeyGenerator.java | 18 +- .../ozone/genesis/BenchMarkMetadataStoreReads.java | 3 + .../genesis/BenchMarkMetadataStoreWrites.java | 4 +- .../ozone/genesis/BenchMarkRocksDbStore.java | 6 +- .../tools/src/main/resources/commands.properties | 22 + .../hadoop/ozone/audit/parser/TestAuditParser.java | 191 + .../hadoop/ozone/audit/parser/package-info.java | 21 + .../tools/src/test/resources/commands.properties | 22 + .../tools/src/test/resources/testaudit.log | 15 + hadoop-project/pom.xml | 31 +- hadoop-project/src/site/site.xml | 2 +- .../fs/aliyun/oss/AliyunOSSCopyFileTask.java | 7 +- .../hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java | 10 +- .../fs/aliyun/oss/AliyunOSSFileSystemStore.java | 31 +- .../aliyun/oss/AssumedRoleCredentialProvider.java | 115 - .../org/apache/hadoop/fs/aliyun/oss/Constants.java | 22 - .../src/site/markdown/tools/hadoop-aliyun/index.md | 58 +- .../fs/aliyun/oss/TestAliyunCredentials.java | 30 +- .../aliyun/oss/TestAliyunOSSBlockOutputStream.java | 1 - .../oss/TestAliyunOSSFileSystemContract.java | 14 +- .../aliyun/oss/TestAliyunOSSFileSystemStore.java | 10 +- .../oss/contract/TestAliyunOSSContractDistCp.java | 1 - hadoop-tools/hadoop-aws/pom.xml | 21 + .../hadoop/fs/s3a/AWSCredentialProviderList.java | 80 +- .../java/org/apache/hadoop/fs/s3a/Constants.java | 51 +- .../hadoop/fs/s3a/DefaultS3ClientFactory.java | 11 +- .../java/org/apache/hadoop/fs/s3a/Invoker.java | 2 +- .../main/java/org/apache/hadoop/fs/s3a/S3A.java | 12 +- .../apache/hadoop/fs/s3a/S3AEncryptionMethods.java | 43 +- .../org/apache/hadoop/fs/s3a/S3AFileSystem.java | 407 +- .../apache/hadoop/fs/s3a/S3AInstrumentation.java | 28 +- .../java/org/apache/hadoop/fs/s3a/S3AUtils.java | 143 +- .../org/apache/hadoop/fs/s3a/S3ClientFactory.java | 6 +- .../fs/s3a/SimpleAWSCredentialsProvider.java | 33 +- .../java/org/apache/hadoop/fs/s3a/Statistic.java | 7 +- .../fs/s3a/TemporaryAWSCredentialsProvider.java | 89 +- .../fs/s3a/auth/AbstractAWSCredentialProvider.java | 70 + .../auth/AbstractSessionCredentialsProvider.java | 170 + .../fs/s3a/auth/AssumedRoleCredentialProvider.java | 41 +- .../s3a/auth/IAMInstanceCredentialsProvider.java | 75 + .../fs/s3a/auth/MarshalledCredentialBinding.java | 205 + .../fs/s3a/auth/MarshalledCredentialProvider.java | 92 + .../hadoop/fs/s3a/auth/MarshalledCredentials.java | 409 + .../hadoop/fs/s3a/auth/NoAuthWithAWSException.java | 8 +- .../fs/s3a/auth/NoAwsCredentialsException.java | 69 + .../org/apache/hadoop/fs/s3a/auth/RoleModel.java | 95 +- .../apache/hadoop/fs/s3a/auth/RolePolicies.java | 190 +- .../hadoop/fs/s3a/auth/STSClientFactory.java | 173 +- .../fs/s3a/auth/delegation/AWSPolicyProvider.java | 59 + .../fs/s3a/auth/delegation/AbstractDTService.java | 154 + .../delegation/AbstractDelegationTokenBinding.java | 305 + .../delegation/AbstractS3ATokenIdentifier.java | 305 + .../s3a/auth/delegation/DelegationConstants.java | 165 + .../delegation/DelegationTokenIOException.java | 50 + .../delegation/EncryptionSecretOperations.java | 73 + .../fs/s3a/auth/delegation/EncryptionSecrets.java | 221 + .../delegation/FullCredentialsTokenBinding.java | 172 + .../delegation/FullCredentialsTokenIdentifier.java | 50 + .../fs/s3a/auth/delegation/RoleTokenBinding.java | 176 + .../s3a/auth/delegation/RoleTokenIdentifier.java | 49 + .../s3a/auth/delegation/S3ADelegationTokens.java | 685 ++ .../fs/s3a/auth/delegation/S3ADtFetcher.java | 80 + .../s3a/auth/delegation/SessionTokenBinding.java | 421 + .../auth/delegation/SessionTokenIdentifier.java | 146 + .../fs/s3a/auth/delegation/package-info.java | 34 + .../apache/hadoop/fs/s3a/auth/package-info.java | 6 +- .../apache/hadoop/fs/s3a/commit/DurationInfo.java | 39 +- .../hadoop/fs/s3a/s3guard/DirListingMetadata.java | 5 +- .../fs/s3a/s3guard/DynamoDBMetadataStore.java | 53 +- .../apache/hadoop/fs/s3a/s3guard/S3GuardTool.java | 121 +- .../apache/hadoop/fs/s3native/S3xLoginHelper.java | 2 - .../org.apache.hadoop.security.token.DtFetcher | 18 + ...rg.apache.hadoop.security.token.TokenIdentifier | 20 + .../markdown/tools/hadoop-aws/assumed_roles.md | 289 +- .../hadoop-aws/delegation_token_architecture.md | 466 + .../markdown/tools/hadoop-aws/delegation_tokens.md | 870 ++ .../src/site/markdown/tools/hadoop-aws/index.md | 87 +- .../src/site/markdown/tools/hadoop-aws/s3guard.md | 16 +- .../src/site/markdown/tools/hadoop-aws/testing.md | 69 +- .../tools/hadoop-aws/troubleshooting_s3a.md | 18 +- .../apache/hadoop/fs/s3a/AbstractS3ATestBase.java | 6 +- .../ITestS3AEncryptionSSECBlockOutputStream.java | 45 - ...ptionSSEKMSUserDefinedKeyBlockOutputStream.java | 50 - .../ITestS3AEncryptionSSES3BlockOutputStream.java | 44 - .../fs/s3a/ITestS3ATemporaryCredentials.java | 364 +- .../apache/hadoop/fs/s3a/MockS3AFileSystem.java | 19 +- .../apache/hadoop/fs/s3a/MockS3ClientFactory.java | 3 +- .../org/apache/hadoop/fs/s3a/S3ATestConstants.java | 20 + .../org/apache/hadoop/fs/s3a/S3ATestUtils.java | 239 + .../fs/s3a/TestS3AAWSCredentialsProvider.java | 218 +- .../apache/hadoop/fs/s3a/TestSSEConfiguration.java | 25 + .../apache/hadoop/fs/s3a/auth/ITestAssumeRole.java | 55 +- .../s3a/auth/ITestAssumedRoleCommitOperations.java | 2 +- .../apache/hadoop/fs/s3a/auth/RoleTestUtils.java | 41 +- .../fs/s3a/auth/TestMarshalledCredentials.java | 138 + .../s3a/auth/delegation/AbstractDelegationIT.java | 207 + .../auth/delegation/CountInvocationsProvider.java | 52 + .../hadoop/fs/s3a/auth/delegation/Csvout.java | 103 + .../auth/delegation/ILoadTestRoleCredentials.java | 38 + .../delegation/ILoadTestSessionCredentials.java | 295 + .../s3a/auth/delegation/ITestDelegatedMRJob.java | 272 + .../delegation/ITestRoleDelegationInFileystem.java | 68 + .../auth/delegation/ITestRoleDelegationTokens.java | 122 + .../ITestSessionDelegationInFileystem.java | 727 ++ .../delegation/ITestSessionDelegationTokens.java | 282 + .../delegation/MiniKerberizedHadoopCluster.java | 378 + .../delegation/TestS3ADelegationTokenSupport.java | 171 + .../fs/s3a/commit/AbstractITCommitMRJob.java | 17 +- .../fs/s3a/commit/AbstractITCommitProtocol.java | 14 - .../fs/s3a/commit/staging/StagingTestBase.java | 28 +- .../s3a/commit/staging/TestStagingCommitter.java | 6 +- .../TestStagingDirectoryOutputCommitter.java | 22 +- .../staging/TestStagingPartitionedFileListing.java | 4 +- .../staging/TestStagingPartitionedJobCommit.java | 4 +- .../staging/TestStagingPartitionedTaskCommit.java | 24 +- .../fileContext/ITestS3AFileContextStatistics.java | 20 +- .../s3a/s3guard/AbstractS3GuardToolTestBase.java | 121 +- .../fs/s3a/s3guard/ITestDynamoDBMetadataStore.java | 75 +- .../fs/s3a/s3guard/ITestS3GuardToolDynamoDB.java | 3 +- .../apache/hadoop/fs/s3a/scale/NanoTimerStats.java | 192 + .../fs/s3a/yarn/ITestS3AMiniYarnCluster.java | 38 +- .../java/org/apache/hadoop/mapreduce/MockJob.java | 115 + hadoop-tools/hadoop-azure-datalake/pom.xml | 2 +- .../fs/azure/AzureNativeFileSystemStore.java | 1 + .../hadoop/fs/azurebfs/AzureBlobFileSystem.java | 31 + .../fs/azurebfs/AzureBlobFileSystemStore.java | 44 +- .../fs/azurebfs/services/AbfsInputStream.java | 12 +- .../fs/azurebfs/services/AbfsOutputStream.java | 16 +- .../fs/azurebfs/services/AbfsRestOperation.java | 6 + .../fs/azurebfs/AbstractAbfsIntegrationTest.java | 4 + .../apache/hadoop/fs/azurebfs/ITestAbfsClient.java | 33 +- .../fs/azurebfs/ITestAzureBlobFileSystemE2E.java | 88 +- .../ITestAzureBlobFileSystemListStatus.java | 67 + .../fs/azurebfs/ITestGetNameSpaceEnabled.java | 96 + .../azurebfs/constants/TestConfigurationKeys.java | 1 + .../apache/hadoop/tools/CopyListingFileStatus.java | 14 +- .../org/apache/hadoop/tools/DistCpConstants.java | 4 + .../apache/hadoop/tools/DistCpOptionSwitch.java | 2 +- .../org/apache/hadoop/tools/TestDistCpOptions.java | 29 + .../namenode/ITestProvidedImplementation.java | 139 +- .../hadoop/yarn/sls/nodemanager/NodeInfo.java | 6 +- .../hadoop-yarn/conf/container-executor.cfg | 1 + .../apache/hadoop/yarn/api/CsiAdaptorProtocol.java | 44 + .../protocolrecords/NodePublishVolumeRequest.java | 94 + .../protocolrecords/NodePublishVolumeResponse.java | 31 + .../NodeUnpublishVolumeRequest.java | 44 + .../NodeUnpublishVolumeResponse.java | 31 + .../apache/hadoop/yarn/api/records/Container.java | 17 +- .../hadoop/yarn/api/records/ContainerReport.java | 20 +- .../hadoop/yarn/api/records/ContainerStatus.java | 18 + .../apache/hadoop/yarn/api/records/Resource.java | 5 - .../yarn/api/records/ResourceInformation.java | 4 +- .../apache/hadoop/yarn/conf/YarnConfiguration.java | 48 +- .../hadoop/yarn/util/csi/CsiConfigUtils.java | 79 + .../apache/hadoop/yarn/util/csi/package-info.java | 21 + .../hadoop/yarn/util/resource/ResourceUtils.java | 26 + .../src/main/proto/YarnCsiAdaptor.proto | 6 + .../src/main/proto/yarn_csi_adaptor.proto | 24 + .../src/main/proto/yarn_protos.proto | 2 + .../distributedshell/ApplicationMaster.java | 58 + .../yarn/applications/distributedshell/Client.java | 77 +- .../distributedshell/TestDSAppMaster.java | 3 +- .../distributedshell/TestDistributedShell.java | 66 +- .../src/test/resources/a.txt | 15 + .../src/test/resources/b.txt | 15 + .../yarn/service/UpgradeComponentsFinder.java | 6 - .../hadoop/yarn/service/api/records/Container.java | 14 + .../service/api/records/ResourceInformation.java | 17 + .../hadoop/yarn/service/component/Component.java | 1 + .../component/instance/ComponentInstance.java | 125 +- .../ServiceTimelineMetricsConstants.java | 2 + .../timelineservice/ServiceTimelinePublisher.java | 2 + .../yarn/service/TestYarnNativeServices.java | 8 +- .../component/instance/TestComponentInstance.java | 36 + .../yarn/service/conf/TestAppJsonResolve.java | 1 + .../yarn/service/conf/examples/external3.json | 1 + .../installation/install.conf | 74 - .../hadoop-yarn-submarine/installation/install.sh | 116 - .../package/calico/calico-node.service | 50 - .../installation/package/calico/calicoctl.cfg | 22 - .../installation/package/docker/daemon.json | 23 - .../installation/package/docker/docker.service | 35 - .../installation/package/etcd/etcd.service | 40 - .../package/hadoop/container-executor.cfg | 41 - .../installation/package/submarine/submarine.sh | 25 - .../installation/scripts/calico.sh | 224 - .../installation/scripts/docker.sh | 166 - .../installation/scripts/download-server.sh | 42 - .../installation/scripts/environment.sh | 213 - .../installation/scripts/etcd.sh | 152 - .../installation/scripts/hadoop.sh | 117 - .../installation/scripts/menu.sh | 444 - .../installation/scripts/nvidia-docker.sh | 99 - .../installation/scripts/nvidia.sh | 120 - .../installation/scripts/submarine.sh | 38 - .../installation/scripts/utils.sh | 123 - .../yarn/submarine/client/cli/CliConstants.java | 1 + .../hadoop/yarn/submarine/client/cli/CliUtils.java | 23 +- .../yarn/submarine/client/cli/RunJobCli.java | 18 + .../submarine/client/cli/param/Localization.java | 133 + .../client/cli/param/RunJobParameters.java | 15 + .../submarine/client/cli/param/package-info.java | 19 + .../yarn/submarine/common/api/JobStatus.java | 4 +- .../common/conf/SubmarineConfiguration.java | 15 + .../common/fs/DefaultRemoteDirectoryManager.java | 70 +- .../common/fs/RemoteDirectoryManager.java | 18 +- .../common/FSBasedSubmarineStorageImpl.java | 8 +- .../yarnservice/YarnServiceJobSubmitter.java | 283 +- .../src/site/markdown/HowToInstall.md | 2 +- .../src/site/markdown/InstallationScriptCN.md | 242 - .../src/site/markdown/InstallationScriptEN.md | 250 - .../src/site/markdown/QuickStart.md | 44 + .../cli/yarnservice/TestYarnServiceRunJobCli.java | 795 +- .../yarn/submarine/common/MockClientContext.java | 8 +- .../common/fs/MockRemoteDirectoryManager.java | 90 +- .../yarn/client/api/ContainerShellWebSocket.java | 7 + .../yarn/client/api/impl/YarnClientImpl.java | 16 +- .../hadoop/yarn/client/cli/ApplicationCLI.java | 8 +- .../apache/hadoop/yarn/client/cli/RMAdminCLI.java | 6 +- .../yarn/client/api/impl/TestYarnClient.java | 50 + .../apache/hadoop/yarn/client/cli/TestYarnCLI.java | 20 +- .../pb/client/CsiAdaptorProtocolPBClientImpl.java | 36 + .../service/CsiAdaptorProtocolPBServiceImpl.java | 36 + .../impl/pb/NodePublishVolumeRequestPBImpl.java | 201 + .../impl/pb/NodePublishVolumeResponsePBImpl.java | 62 + .../impl/pb/NodeUnpublishVolumeRequestPBImpl.java | 89 + .../impl/pb/NodeUnpublishVolumeResponsePBImpl.java | 61 + .../yarn/api/records/impl/pb/ContainerPBImpl.java | 44 +- .../api/records/impl/pb/ContainerReportPBImpl.java | 23 +- .../api/records/impl/pb/ContainerStatusPBImpl.java | 21 + .../yarn/api/records/impl/pb/ProtoUtils.java | 18 + .../hadoop/yarn/nodelabels/NodeLabelUtil.java | 51 +- .../server/metrics/ContainerMetricsConstants.java | 3 + .../util/resource/DefaultResourceCalculator.java | 2 +- .../util/resource/DominantResourceCalculator.java | 36 +- .../yarn/util/resource/ResourceCalculator.java | 18 + .../hadoop/yarn/util/resource/Resources.java | 16 +- .../util/timeline/TimelineEntityV2Converter.java | 14 +- .../src/main/resources/yarn-default.xml | 48 + .../hadoop/yarn/nodelabels/TestNodeLabelUtil.java | 71 + .../CustomResourceTypesConfigurationProvider.java | 186 + .../yarn/util/resource/TestResourceCalculator.java | 26 +- .../yarn/util/resource/TestResourceUtils.java | 70 +- .../resources/resource-types/resource-types-6.xml | 58 + .../hadoop-yarn/hadoop-yarn-csi/pom.xml | 96 +- .../csi/adaptor/CsiAdaptorProtocolService.java | 76 +- .../apache/hadoop/yarn/csi/client/CsiClient.java | 6 + .../hadoop/yarn/csi/client/CsiClientImpl.java | 20 + .../GetPluginInfoResponseProtoTranslator.java | 2 +- .../NodePublishVolumeRequestProtoTranslator.java | 77 + .../NodeUnpublishVolumeRequestProtoTranslator.java | 49 + .../csi/translator/ProtoTranslatorFactory.java | 12 + .../apache/hadoop/yarn/csi/utils/ConfigUtils.java | 61 - .../yarn/csi/adaptor/TestCsiAdaptorService.java | 24 +- .../csi/adaptor/TestNodePublishVolumeRequest.java | 55 + .../hadoop/yarn/csi/client/ICsiClientTest.java | 53 + .../ApplicationHistoryManagerImpl.java | 17 +- .../ApplicationHistoryManagerOnTimelineStore.java | 13 +- .../records/ContainerHistoryData.java | 13 + .../timeline/KeyValueBasedTimelineStore.java | 98 +- .../yarn/server/timeline/MemoryTimelineStore.java | 37 +- .../server/timeline/TimelineStoreMapAdapter.java | 9 +- .../api/protocolrecords/NodeHeartbeatRequest.java | 4 + .../api/protocolrecords/NodeHeartbeatResponse.java | 26 +- .../RegisterNodeManagerRequest.java | 16 + .../RegisterNodeManagerResponse.java | 5 + .../impl/pb/NodeHeartbeatRequestPBImpl.java | 13 + .../impl/pb/NodeHeartbeatResponsePBImpl.java | 106 +- .../impl/pb/RegisterNodeManagerRequestPBImpl.java | 53 + .../impl/pb/RegisterNodeManagerResponsePBImpl.java | 15 + .../yarn/server/utils/YarnServerBuilderUtils.java | 68 + .../yarn/server/webapp/dao/ContainerInfo.java | 7 +- .../proto/yarn_server_common_service_protos.proto | 5 + .../hadoop/yarn/TestYarnServerApiClasses.java | 55 +- .../api/protocolrecords/TestProtocolRecords.java | 21 +- .../metrics/TestAMRMClientRelayerMetrics.java | 6 +- .../hadoop-yarn-server-nodemanager/pom.xml | 1 + .../yarn/server/nodemanager/ContainerExecutor.java | 5 + .../hadoop/yarn/server/nodemanager/Context.java | 5 + .../server/nodemanager/LinuxContainerExecutor.java | 12 +- .../yarn/server/nodemanager/NodeManager.java | 13 + .../server/nodemanager/NodeStatusUpdaterImpl.java | 340 +- .../amrmproxy/FederationInterceptor.java | 5 +- .../api/deviceplugin/DevicePluginScheduler.java | 38 + .../nodemanager/containermanager/AuxServices.java | 861 +- .../AuxiliaryServiceWithCustomClassLoader.java | 8 +- .../containermanager/ContainerManagerImpl.java | 7 +- .../containermanager/container/Container.java | 6 + .../containermanager/container/ContainerImpl.java | 41 +- .../containermanager/launcher/ContainerLaunch.java | 35 + .../launcher/ContainersLauncher.java | 50 +- .../linux/privileged/PrivilegedOperation.java | 2 +- .../resources/CGroupsBlkioResourceHandlerImpl.java | 5 + .../resources/CGroupsCpuResourceHandlerImpl.java | 5 + .../linux/resources/CGroupsHandlerImpl.java | 14 +- .../CGroupsMemoryResourceHandlerImpl.java | 5 + .../resources/NetworkPacketTaggingHandlerImpl.java | 5 + .../linux/resources/ResourceHandlerChain.java | 6 + .../linux/resources/ResourceHandlerModule.java | 37 +- .../TrafficControlBandwidthHandlerImpl.java | 5 + .../resources/fpga/FpgaResourceHandlerImpl.java | 14 +- .../linux/resources/gpu/GpuResourceAllocator.java | 5 + .../resources/gpu/GpuResourceHandlerImpl.java | 15 +- .../resources/numa/NumaResourceHandlerImpl.java | 12 +- .../runtime/DefaultLinuxContainerRuntime.java | 108 +- .../runtime/DelegatingLinuxContainerRuntime.java | 7 + .../linux/runtime/DockerLinuxContainerRuntime.java | 88 +- .../linux/runtime/docker/DockerClient.java | 8 +- .../linux/runtime/docker/DockerInspectCommand.java | 7 + .../linux/runtime/docker/DockerRunCommand.java | 5 + .../localizer/ContainerLocalizer.java | 1 + .../containermanager/monitor/ContainerMetrics.java | 8 +- .../monitor/ContainerStopMonitoringEvent.java | 12 + .../monitor/ContainersMonitorImpl.java | 8 +- .../records/AuxServiceConfiguration.java | 137 + .../containermanager/records/AuxServiceFile.java | 137 + .../containermanager/records/AuxServiceRecord.java | 175 + .../records/AuxServiceRecords.java | 46 + .../containermanager/records/package-info.java | 29 + .../resourceplugin/ResourcePluginManager.java | 47 +- .../deviceframework/AssignedDevice.java | 78 + .../deviceframework/DeviceMappingManager.java | 73 +- .../deviceframework/DevicePluginAdapter.java | 25 +- .../deviceframework/DeviceResourceHandlerImpl.java | 21 +- .../resourceplugin/fpga/FpgaResourcePlugin.java | 5 + .../gpu/GpuDockerCommandPluginFactory.java | 4 + .../resourceplugin/gpu/GpuResourcePlugin.java | 5 + .../gpu/NvidiaDockerV2CommandPlugin.java | 111 + .../containermanager/runtime/ContainerRuntime.java | 12 +- .../scheduler/ContainerScheduler.java | 14 +- .../volume/csi/ContainerVolumePublisher.java | 207 + .../containermanager/volume/csi/package-info.java | 22 + .../nodemanager/executor/ContainerExecContext.java | 28 +- .../executor/ContainerStartContext.java | 12 + .../nodemanager/metrics/NodeManagerMetrics.java | 21 + .../webapp/ContainerShellWebSocket.java | 30 + .../nodemanager/webapp/JAXBContextResolver.java | 3 + .../server/nodemanager/webapp/NMWebServices.java | 34 + .../server/nodemanager/webapp/TerminalServlet.java | 49 + .../yarn/server/nodemanager/webapp/WebServer.java | 9 +- .../webapp/dao/AuxiliaryServiceInfo.java | 60 + .../webapp/dao/AuxiliaryServicesInfo.java | 55 + .../webapp/dao/NMDeviceResourceInfo.java | 56 + .../container-executor/impl/container-executor.c | 248 +- .../container-executor/impl/container-executor.h | 15 +- .../src/main/native/container-executor/impl/main.c | 40 +- .../container-executor/impl/utils/docker-util.c | 28 +- .../container-executor/impl/utils/docker-util.h | 3 +- .../test/test-container-executor.c | 54 +- .../test/utils/test_docker_util.cc | 80 +- .../src/main/resources/TERMINAL/css/style.css | 33 + .../src/main/resources/TERMINAL/terminal.template | 127 + .../TERMINAL/xterm/dist/addons/attach/attach.js | 104 + .../xterm/dist/addons/attach/attach.js.map | 1 + .../TERMINAL/xterm/dist/addons/fit/fit.js | 51 + .../TERMINAL/xterm/dist/addons/fit/fit.js.map | 1 + .../xterm/dist/addons/fullscreen/fullscreen.css | 10 + .../xterm/dist/addons/fullscreen/fullscreen.js | 27 + .../xterm/dist/addons/fullscreen/fullscreen.js.map | 1 + .../TERMINAL/xterm/dist/addons/search/search.js | 166 + .../xterm/dist/addons/search/search.js.map | 1 + .../xterm/dist/addons/terminado/terminado.js | 69 + .../xterm/dist/addons/terminado/terminado.js.map | 1 + .../xterm/dist/addons/webLinks/webLinks.js | 41 + .../xterm/dist/addons/webLinks/webLinks.js.map | 1 + .../xterm/dist/addons/winptyCompat/winptyCompat.js | 31 + .../dist/addons/winptyCompat/winptyCompat.js.map | 1 + .../TERMINAL/xterm/dist/addons/zmodem/zmodem.js | 45 + .../xterm/dist/addons/zmodem/zmodem.js.map | 1 + .../main/resources/TERMINAL/xterm/dist/xterm.css | 164 + .../main/resources/TERMINAL/xterm/dist/xterm.js | 8907 ++++++++++++++++++++ .../resources/TERMINAL/xterm/dist/xterm.js.map | 1 + .../server/nodemanager/TestNodeStatusUpdater.java | 32 +- .../TestNodeStatusUpdaterForAttributes.java | 439 + .../nodemanager/amrmproxy/BaseAMRMProxyTest.java | 10 + .../amrmproxy/TestFederationInterceptor.java | 75 + .../containermanager/TestAuxServices.java | 394 +- .../launcher/TestContainersLauncher.java | 14 +- .../resources/fpga/TestFpgaResourceHandler.java | 5 +- .../resources/gpu/TestGpuResourceHandler.java | 5 +- .../linux/runtime/MockLinuxContainerRuntime.java | 5 + .../linux/runtime/docker/TestDockerClient.java | 46 + .../linux/runtime/docker/TestDockerRunCommand.java | 5 +- .../monitor/TestContainerMetrics.java | 6 +- .../resourceplugin/TestResourcePluginManager.java | 28 + .../deviceframework/FakeTestDevicePlugin1.java | 18 +- .../deviceframework/FakeTestDevicePlugin5.java | 56 + .../deviceframework/TestDevicePluginAdapter.java | 137 +- .../gpu/TestNvidiaDockerV2CommandPlugin.java | 130 + .../scheduler/TestContainerSchedulerQueuing.java | 30 + .../server/nodemanager/webapp/MockContainer.java | 14 + .../webapp/TestNMWebServicesAuxServices.java | 296 + .../nodemanager/webapp/TestNMWebTerminal.java | 124 + .../yarn/server/resourcemanager/AdminService.java | 7 + .../OpportunisticContainerAllocatorAMService.java | 8 +- .../resourcemanager/RMActiveServiceContext.java | 29 +- .../yarn/server/resourcemanager/RMContext.java | 11 +- .../yarn/server/resourcemanager/RMContextImpl.java | 15 +- .../resourcemanager/ResourceTrackerService.java | 136 +- .../metrics/TimelineServiceV2Publisher.java | 3 + .../AbstractPreemptableResourceCalculator.java | 2 +- .../monitor/capacity/TempQueuePerPartition.java | 2 +- .../nodelabels/NodeAttributesManagerImpl.java | 13 +- .../resourcemanager/reservation/InMemoryPlan.java | 4 +- .../resourcemanager/rmcontainer/RMContainer.java | 5 + .../rmcontainer/RMContainerImpl.java | 18 +- .../server/resourcemanager/rmnode/RMNodeImpl.java | 36 +- .../rmnode/UpdatedContainerInfo.java | 14 +- .../scheduler/AbstractYarnScheduler.java | 49 +- .../scheduler/ClusterNodeTracker.java | 2 +- .../scheduler/ContainerUpdateContext.java | 2 + .../resourcemanager/scheduler/SchedulerUtils.java | 14 +- .../scheduler/capacity/CSQueueMetrics.java | 39 + .../scheduler/capacity/CSQueueUtils.java | 17 + .../scheduler/capacity/LeafQueue.java | 4 + .../scheduler/capacity/ManagedParentQueue.java | 20 +- .../scheduler/capacity/ParentQueue.java | 8 +- .../scheduler/fair/ConfigurableResource.java | 2 +- .../scheduler/fair/FSParentQueue.java | 7 +- .../scheduler/fair/FSPreemptionThread.java | 55 +- .../scheduler/fair/FairScheduler.java | 52 +- .../scheduler/fair/FairSchedulerConfiguration.java | 97 +- .../scheduler/fair/policies/ComputeFairShares.java | 62 +- .../policies/DominantResourceFairnessPolicy.java | 2 +- .../security/DelegationTokenRenewer.java | 17 +- .../volume/csi/lifecycle/VolumeImpl.java | 2 + .../resourcemanager/webapp/dao/AppAttemptInfo.java | 6 + .../hadoop/yarn/server/resourcemanager/MockNM.java | 3 + .../yarn/server/resourcemanager/MockNodes.java | 47 +- .../resourcemanager/TestClientRMService.java | 3 +- ...stOpportunisticContainerAllocatorAMService.java | 78 + .../TestResourceTrackerService.java | 611 +- .../scheduler/TestSchedulerUtils.java | 51 +- .../scheduler/capacity/TestCapacityScheduler.java | 44 + .../capacity/TestContainerAllocation.java | 5 +- .../scheduler/fair/FakeSchedulable.java | 17 +- .../fair/TestAllocationFileLoaderService.java | 7 +- .../scheduler/fair/TestComputeFairShares.java | 104 +- .../scheduler/fair/TestFairScheduler.java | 102 +- .../fair/TestFairSchedulerConfiguration.java | 320 +- .../fair/TestFairSchedulerPreemption.java | 155 +- .../security/TestDelegationTokenRenewer.java | 110 +- .../volume/csi/TestVolumeProcessor.java | 61 + .../webapp/TestRMWebServicesAppAttempts.java | 10 +- .../TestRMWebServicesAppCustomResourceTypes.java | 2 +- .../TestRMWebServicesAppsCustomResourceTypes.java | 2 +- ...estRMWebServicesCustomResourceTypesCommons.java | 3 +- .../CustomResourceTypesConfigurationProvider.java | 140 - ...ebServicesFairSchedulerCustomResourceTypes.java | 5 +- .../server/timeline/LevelDBCacheTimelineStore.java | 14 +- .../timeline/TestEntityGroupFSTimelineStore.java | 6 +- .../collector/NodeTimelineCollectorManager.java | 15 +- .../collector/TestNMTimelineCollectorManager.java | 6 +- .../hadoop/yarn/server/webproxy/WebAppProxy.java | 14 +- .../yarn/server/webproxy/WebAppProxyServer.java | 10 +- .../server/webproxy/TestWebAppProxyServer.java | 31 +- .../src/site/markdown/DockerContainers.md | 1 + .../src/site/markdown/NodeManager.md | 46 +- .../src/site/markdown/NodeManagerRest.md | 93 + .../src/site/markdown/SecureContainer.md | 3 + .../src/site/markdown/TimelineServiceV2.md | 16 + .../src/site/markdown/UsingGpus.md | 9 +- .../src/site/markdown/yarn-service/QuickStart.md | 1 + .../site/markdown/yarn-service/YarnServiceAPI.md | 57 +- .../hadoop-yarn/hadoop-yarn-ui/pom.xml | 41 + .../main/webapp/app/components/timeline-view.js | 43 +- .../src/main/webapp/app/initializers/env.js | 1 + .../src/main/webapp/app/initializers/loader.js | 26 + .../src/main/webapp/app/models/yarn-app-attempt.js | 1 + .../webapp/app/models/yarn-component-instance.js | 11 + .../src/main/webapp/app/models/yarn-container.js | 3 +- .../webapp/app/models/yarn-timeline-appattempt.js | 1 + .../webapp/app/models/yarn-timeline-container.js | 3 +- .../hadoop-yarn-ui/src/main/webapp/app/router.js | 2 +- .../src/main/webapp/app/routes/yarn-app-attempt.js | 5 + .../app/routes/yarn-component-instance/info.js | 5 + .../webapp/app/serializers/yarn-app-attempt.js | 1 + .../app/serializers/yarn-component-instance.js | 1 + .../main/webapp/app/serializers/yarn-container.js | 3 +- .../app/serializers/yarn-timeline-appattempt.js | 1 + .../app/serializers/yarn-timeline-container.js | 1 + .../app/templates/components/app-attempt-table.hbs | 6 + .../app/templates/components/container-table.hbs | 12 + .../app/templates/components/timeline-view.hbs | 2 +- .../main/webapp/app/templates/yarn-app-attempt.hbs | 1 + .../app/templates/yarn-component-instance/info.hbs | 8 + 1043 files changed, 62621 insertions(+), 13210 deletions(-) create mode 100644 hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/AlignmentContext.java create mode 100644 hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ObserverRetryOnActiveException.java create mode 100644 hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java create mode 100644 hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java delete mode 100644 hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ChunkInputStream.java delete mode 100644 hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ChunkOutputStream.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/UnknownPipelineStateException.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/SCMAction.java create mode 100644 hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/DeleteContainerCommandHandler.java create mode 100644 hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/commands/DeleteContainerCommand.java create mode 100644 hadoop-hdds/docs/content/AuditParser.md create mode 100644 hadoop-hdds/docs/content/genconf.md create mode 100644 hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/DeleteContainerCommandWatcher.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ObserverReadProxyProvider.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ObserverReadProxyProviderWithIPFailover.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ReadOnly.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/protocol/TestReadOnly.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournaledEditsCache.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/TokenVerifier.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GlobalStateIdContext.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNameNode.md create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingAddConfig.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestStateAlignmentContextWithHA.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/JournalTestUtil.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournaledEditsCache.java delete mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/aliasmap/TestInMemoryAliasMap.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestBootstrapAliasmap.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestConsistentReadsObserver.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestMultiObserverNode.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverNode.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverReadProxyProvider.java create mode 100644 hadoop-ozone/Jenkinsfile create mode 100644 hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockOutputStreamEntry.java delete mode 100644 hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ChunkGroupInputStream.java delete mode 100644 hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ChunkGroupOutputStream.java delete mode 100644 hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ChunkOutputStreamEntry.java create mode 100644 hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyInputStream.java create mode 100644 hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/hdds/protocol/package-info.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/codec/OmBucketInfoCodec.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/codec/OmKeyInfoCodec.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/codec/OmMultipartKeyInfoCodec.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/codec/OmVolumeArgsCodec.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/codec/VolumeListCodec.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/codec/package-info.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartCommitUploadPartInfo.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartInfo.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartKeyInfo.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartUploadCompleteInfo.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartUploadList.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/IAccessAuthorizer.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/IOzoneObj.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/OzoneAccessAuthorizer.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/OzoneAclException.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/OzoneObj.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/OzoneObjInfo.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/RequestContext.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/package-info.java create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/package-info.java create mode 100644 hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/codec/TestOmMultipartKeyInfoCodec.java create mode 100644 hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/codec/package-info.java create mode 100644 hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/security/acl/TestOzoneObjInfo.java create mode 100755 hadoop-ozone/dev-support/checks/acceptance.sh create mode 100755 hadoop-ozone/dev-support/checks/author.sh create mode 100755 hadoop-ozone/dev-support/checks/build.sh create mode 100755 hadoop-ozone/dev-support/checks/checkstyle.sh create mode 100755 hadoop-ozone/dev-support/checks/findbugs.sh create mode 100755 hadoop-ozone/dev-support/checks/isolation.sh create mode 100755 hadoop-ozone/dev-support/checks/rat.sh create mode 100755 hadoop-ozone/dev-support/checks/unit.sh create mode 100644 hadoop-ozone/dev-support/docker/Dockerfile create mode 100644 hadoop-ozone/dist/Dockerfile create mode 100644 hadoop-ozone/dist/README.md create mode 100644 hadoop-ozone/dist/skaffold.yaml create mode 100644 hadoop-ozone/dist/src/main/blockade/README.md create mode 100644 hadoop-ozone/dist/src/main/blockade/blockadeUtils/__init__.py create mode 100644 hadoop-ozone/dist/src/main/blockade/blockadeUtils/blockade.py copy hadoop-tools/hadoop-resourceestimator/src/main/resources/webapps/ResourceEstimatorServer/.gitignore => hadoop-ozone/dist/src/main/blockade/clusterUtils/__init__.py (100%) create mode 100644 hadoop-ozone/dist/src/main/blockade/clusterUtils/cluster_utils.py create mode 100644 hadoop-ozone/dist/src/main/blockade/conftest.py create mode 100644 hadoop-ozone/dist/src/main/blockade/test_blockade.py create mode 100644 hadoop-ozone/dist/src/main/compose/common/grafana/conf/grafana.ini create mode 100644 hadoop-ozone/dist/src/main/compose/common/grafana/dashboards/Ozone - Object Metrics.json create mode 100644 hadoop-ozone/dist/src/main/compose/common/grafana/dashboards/Ozone - RPC Metrics.json create mode 100755 hadoop-ozone/dist/src/main/compose/common/grafana/provisioning/dashboards/dashboards.yml create mode 100755 hadoop-ozone/dist/src/main/compose/common/grafana/provisioning/datasources/datasources.yml rename hadoop-ozone/dist/src/main/compose/{ozoneperf => common/prometheus}/prometheus.yml (100%) create mode 100644 hadoop-ozone/dist/src/main/conf/scm-audit-log4j2.properties create mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/config-configmap.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/datanode-public-service.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/datanode-service.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/datanode-statefulset.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/om-public-service.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/om-service.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/om-statefulset.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/s3g-public-service.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/s3g-service.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/s3g-statefulset.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/scm-public-service.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/scm-service.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/scm-statefulset.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/prometheus/configmap.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/prometheus/deployment.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/prometheus/prometheus-public-serivce.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/prometheus/role.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/prometheus/rolebindng.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/prometheus/scm-public-service.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/prometheus/service-account.yaml create mode 100644 hadoop-ozone/dist/src/main/k8s/prometheus/service.yaml create mode 100644 hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientWithRatis.java create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmAcls.java create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestOzoneDatanodeShell.java create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OMRatisHelper.java create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisClient.java create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServer.java create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/package-info.java create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerRatisServer.java create mode 100644 hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/CompleteMultipartUploadRequest.java create mode 100644 hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/CompleteMultipartUploadResponse.java create mode 100644 hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/MultipartUploadInitiateResponse.java create mode 100644 hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/OzoneOutputStreamStub.java create mode 100644 hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestInitiateMultipartUpload.java create mode 100644 hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestMultipartUploadComplete.java create mode 100644 hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPartUpload.java create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/audit/parser/AuditParser.java create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/audit/parser/common/DatabaseHelper.java create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/audit/parser/common/ParserConsts.java create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/audit/parser/common/package-info.java create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/audit/parser/handler/LoadCommandHandler.java create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/audit/parser/handler/QueryCommandHandler.java create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/audit/parser/handler/TemplateCommandHandler.java create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/audit/parser/handler/package-info.java create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/audit/parser/model/AuditEntry.java create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/audit/parser/model/package-info.java create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/audit/parser/package-info.java create mode 100644 hadoop-ozone/tools/src/main/resources/commands.properties create mode 100644 hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/audit/parser/TestAuditParser.java create mode 100644 hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/audit/parser/package-info.java create mode 100644 hadoop-ozone/tools/src/test/resources/commands.properties create mode 100644 hadoop-ozone/tools/src/test/resources/testaudit.log delete mode 100644 hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AssumedRoleCredentialProvider.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/AbstractAWSCredentialProvider.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/AbstractSessionCredentialsProvider.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/IAMInstanceCredentialsProvider.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/MarshalledCredentialBinding.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/MarshalledCredentialProvider.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/MarshalledCredentials.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/NoAwsCredentialsException.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/AWSPolicyProvider.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/AbstractDTService.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/AbstractDelegationTokenBinding.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/AbstractS3ATokenIdentifier.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/DelegationConstants.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/DelegationTokenIOException.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/EncryptionSecretOperations.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/EncryptionSecrets.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/FullCredentialsTokenBinding.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/FullCredentialsTokenIdentifier.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/RoleTokenBinding.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/RoleTokenIdentifier.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/S3ADelegationTokens.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/S3ADtFetcher.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/SessionTokenBinding.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/SessionTokenIdentifier.java create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/package-info.java create mode 100644 hadoop-tools/hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.security.token.DtFetcher create mode 100644 hadoop-tools/hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier create mode 100644 hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/delegation_token_architecture.md create mode 100644 hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/delegation_tokens.md delete mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSECBlockOutputStream.java delete mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEKMSUserDefinedKeyBlockOutputStream.java delete mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSES3BlockOutputStream.java create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/TestMarshalledCredentials.java create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/AbstractDelegationIT.java create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/CountInvocationsProvider.java create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/Csvout.java create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ILoadTestRoleCredentials.java create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ILoadTestSessionCredentials.java create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ITestDelegatedMRJob.java create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ITestRoleDelegationInFileystem.java create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ITestRoleDelegationTokens.java create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ITestSessionDelegationInFileystem.java create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ITestSessionDelegationTokens.java create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/MiniKerberizedHadoopCluster.java create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/TestS3ADelegationTokenSupport.java create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/NanoTimerStats.java create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/mapreduce/MockJob.java create mode 100644 hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestGetNameSpaceEnabled.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/NodePublishVolumeRequest.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/NodePublishVolumeResponse.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/NodeUnpublishVolumeRequest.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/NodeUnpublishVolumeResponse.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/csi/CsiConfigUtils.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/csi/package-info.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/resources/a.txt create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/resources/b.txt delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/install.conf delete mode 100755 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/install.sh delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/package/calico/calico-node.service delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/package/calico/calicoctl.cfg delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/package/docker/daemon.json delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/package/docker/docker.service delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/package/etcd/etcd.service delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/package/hadoop/container-executor.cfg delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/package/submarine/submarine.sh delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/scripts/calico.sh delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/scripts/docker.sh delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/scripts/download-server.sh delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/scripts/environment.sh delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/scripts/etcd.sh delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/scripts/hadoop.sh delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/scripts/menu.sh delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/scripts/nvidia-docker.sh delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/scripts/nvidia.sh delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/scripts/submarine.sh delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation/scripts/utils.sh create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/Localization.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/package-info.java delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/site/markdown/InstallationScriptCN.md delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/site/markdown/InstallationScriptEN.md create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/NodePublishVolumeRequestPBImpl.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/NodePublishVolumeResponsePBImpl.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/NodeUnpublishVolumeRequestPBImpl.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/NodeUnpublishVolumeResponsePBImpl.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/resource/CustomResourceTypesConfigurationProvider.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/resources/resource-types/resource-types-6.xml create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/main/java/org/apache/hadoop/yarn/csi/translator/NodePublishVolumeRequestProtoTranslator.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/main/java/org/apache/hadoop/yarn/csi/translator/NodeUnpublishVolumeRequestProtoTranslator.java delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/main/java/org/apache/hadoop/yarn/csi/utils/ConfigUtils.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/test/java/org/apache/hadoop/yarn/csi/adaptor/TestNodePublishVolumeRequest.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/test/java/org/apache/hadoop/yarn/csi/client/ICsiClientTest.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DevicePluginScheduler.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/records/AuxServiceConfiguration.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/records/AuxServiceFile.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/records/AuxServiceRecord.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/records/AuxServiceRecords.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/records/package-info.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/deviceframework/AssignedDevice.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/NvidiaDockerV2CommandPlugin.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/volume/csi/ContainerVolumePublisher.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/volume/csi/package-info.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TerminalServlet.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/dao/AuxiliaryServiceInfo.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/dao/AuxiliaryServicesInfo.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/dao/NMDeviceResourceInfo.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/css/style.css create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/terminal.template create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/addons/attach/attach.js create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/addons/attach/attach.js.map create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/addons/fit/fit.js create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/addons/fit/fit.js.map create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/addons/fullscreen/fullscreen.css create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/addons/fullscreen/fullscreen.js create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/addons/fullscreen/fullscreen.js.map create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/addons/search/search.js create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/addons/search/search.js.map create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/addons/terminado/terminado.js create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/addons/terminado/terminado.js.map create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/addons/webLinks/webLinks.js create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/addons/webLinks/webLinks.js.map create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/addons/winptyCompat/winptyCompat.js create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/addons/winptyCompat/winptyCompat.js.map create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/addons/zmodem/zmodem.js create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/addons/zmodem/zmodem.js.map create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/xterm.css create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/xterm.js create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/TERMINAL/xterm/dist/xterm.js.map create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForAttributes.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/deviceframework/FakeTestDevicePlugin5.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/TestNvidiaDockerV2CommandPlugin.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesAuxServices.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebTerminal.java delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/fairscheduler/CustomResourceTypesConfigurationProvider.java --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org