This is an automated email from the ASF dual-hosted git repository.
ayushsaxena pushed a change to branch HDFS-13891
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
discard aeb3b61 HDFS-14457. RBF: Add order text SPACE in CLI command 'hdfs
dfsrouteradmin'. Contributed by luhuachao.
discard 55f2f7a HDFS-13972. RBF: Support for Delegation Token (WebHDFS).
Contributed by CR Hota.
discard bd3161e HDFS-14422. RBF: Router shouldn't allow READ operations in
safe mode. Contributed by Inigo Goiri.
discard e508ab9 HDFS-14369. RBF: Fix trailing / for webhdfs. Contributed by
Akira Ajisaka.
discard 007b8ea HDFS-13853. RBF: RouterAdmin update cmd is overwriting the
entry not updating the existing. Contributed by Ayush Saxena.
discard 2577e3e HDFS-14316. RBF: Support unavailable subclusters for mount
points with multiple destinations. Contributed by Inigo Goiri.
discard d10765d HDFS-14388. RBF: Prevent loading metric system when disabled.
Contributed by Inigo Goiri.
discard 0b0c334 HDFS-14351. RBF: Optimize configuration item resolving for
monitor namenode. Contributed by He Xiaoqiao and Inigo Goiri.
discard ba429bc2 HDFS-14343. RBF: Fix renaming folders spread across multiple
subclusters. Contributed by Ayush Saxena.
discard 64ad3d6 HDFS-14334. RBF: Use human readable format for long numbers
in the Router UI. Contributed by Inigo Goiri.
discard 4a21db8 HDFS-14335. RBF: Fix heartbeat typos in the Router.
Contributed by CR Hota.
discard e6eacbd HDFS-14331. RBF: IOE While Removing Mount Entry. Contributed
by Ayush Saxena.
discard 5634f14 HDFS-14329. RBF: Add maintenance nodes to federation metrics.
Contributed by Ayush Saxena.
discard 3e01881 HDFS-14259. RBF: Fix safemode message for Router. Contributed
by Ranith Sadar.
discard 58c5457 HDFS-14322. RBF: Security manager should not load if security
is disabled. Contributed by CR Hota.
discard 9d809a2 HDFS-14052. RBF: Use Router keytab for WebHDFS. Contributed
by CR Hota.
discard d897cee HDFS-14307. RBF: Update tests to use internal Whitebox
instead of Mockito. Contributed by CR Hota.
discard f330e6f HDFS-14249. RBF: Tooling to identify the subcluster location
of a file. Contributed by Inigo Goiri.
discard 1761b90 HDFS-14268. RBF: Fix the location of the DNs in
getDatanodeReport(). Contributed by Inigo Goiri.
discard 8478112 HDFS-14226. RBF: Setting attributes should set on all
subclusters' directories. Contributed by Ayush Saxena.
discard 49d489c HDFS-13358. RBF: Support for Delegation Token (RPC).
Contributed by CR Hota.
discard 7bbe35e HDFS-14230. RBF: Throw RetriableException instead of
IOException when no namenodes available. Contributed by Fei Hui.
discard 4585b97 HDFS-14252. RBF : Exceptions are exposing the actual sub
cluster path. Contributed by Ayush Saxena.
discard 08a4e69 HDFS-14225. RBF : MiniRouterDFSCluster should configure the
failover proxy provider for namespace. Contributed by Ranith Sardar.
discard e43400c HDFS-13404. RBF:
TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails.
discard 4feb3ae HDFS-14215. RBF: Remove dependency on availability of default
namespace. Contributed by Ayush Saxena.
discard 8dc059d HDFS-14224. RBF: NPE in getContentSummary() for getEcPolicy()
in case of multiple destinations. Contributed by Ayush Saxena.
discard 29fa6ae HDFS-14223. RBF: Add configuration documents for using
multiple sub-clusters. Contributed by Takanobu Asanuma.
discard 3864de4 HDFS-14209. RBF: setQuota() through router is working for
only the mount Points under the Source column in MountTable. Contributed by
Shubham Dewan.
discard 4257376 HDFS-14156. RBF: rollEdit() command fails with Router.
Contributed by Shubham Dewan.
discard 9c22816 HDFS-14193. RBF: Inconsistency with the Default Namespace.
Contributed by Ayush Saxena.
discard 7c01d25 HDFS-14129. addendum to HDFS-14129. Contributed by Ranith
Sardar.
discard e86634a HDFS-14129. RBF: Create new policy provider for router.
Contributed by Ranith Sardar.
discard 692c581 HDFS-14206. RBF: Cleanup quota modules. Contributed by Inigo
Goiri.
discard 03d98f5 HDFS-13856. RBF: RouterAdmin should support dfsrouteradmin
-refreshRouterArgs command. Contributed by yanghuafeng.
discard 64689a2 HDFS-14191. RBF: Remove hard coded router status from
FederationMetrics. Contributed by Ranith Sardar.
discard 2a7998b HDFS-14150. RBF: Quotas of the sub-cluster should be removed
when removing the mount point. Contributed by Takanobu Asanuma.
discard 1372cc8 HDFS-14161. RBF: Throw StandbyException instead of
IOException so that client can retry when can not get connection. Contributed
by Fei Hui.
discard 2b48aa0 HDFS-14167. RBF: Add stale nodes to federation metrics.
Contributed by Inigo Goiri.
discard e3ab7a4 HDFS-13443. RBF: Update mount table cache immediately after
changing (add/update/remove) mount table entries. Contributed by Mohammad
Arshad.
discard aa6cff0 HDFS-14151. RBF: Make the read-only column of Mount Table
clearly understandable.
discard c6eb9ee HDFS-13869. RBF: Handle NPE for
NamenodeBeanMetrics#getFederationMetrics. Contributed by Ranith Sardar.
discard 1555e5c HDFS-14152. RBF: Fix a typo in RouterAdmin usage. Contributed
by Ayush Saxena.
discard 9c16a45 HDFS-14114. RBF: MIN_ACTIVE_RATIO should be configurable.
Contributed by Fei Hui.
discard 83a1beb Revert "HDFS-14114. RBF: MIN_ACTIVE_RATIO should be
configurable. Contributed by Fei Hui."
discard cf474b5 HDFS-14114. RBF: MIN_ACTIVE_RATIO should be configurable.
Contributed by Fei Hui.
discard 6d28616 HDFS-14085. RBF: LS command for root shows wrong owner and
permission information. Contributed by Ayush Saxena.
discard 03baa6d HDFS-14089. RBF: Failed to specify server's Kerberos pricipal
name in NamenodeHeartbeatService. Contributed by Ranith Sardar.
discard 30381b7 HDFS-13776. RBF: Add Storage policies related ClientProtocol
APIs. Contributed by Dibyendu Karmakar.
discard 0273420 HDFS-14082. RBF: Add option to fail operations when a
subcluster is unavailable. Contributed by Inigo Goiri.
discard 5870f70 HDFS-13834. RBF: Connection creator thread should catch
Throwable. Contributed by CR Hota.
discard 79f6fda HDFS-13852. RBF: The DN_REPORT_TIME_OUT and
DN_REPORT_CACHE_EXPIRE should be configured in RBFConfigKeys. Contributed by
yanghuafeng.
discard fb65537 HDFS-12284. addendum to HDFS-12284. Contributed by Inigo
Goiri.
discard fbe3201 HDFS-12284. RBF: Support for Kerberos authentication.
Contributed by Sherwood Zheng and Inigo Goiri.
discard 9fb5bc4 HDFS-14024. RBF: ProvidedCapacityTotal json exception in
NamenodeHeartbeatService. Contributed by CR Hota.
discard e81ba43 HDFS-13845. RBF: The default MountTableResolver should fail
resolving multi-destination paths. Contributed by yanghuafeng.
discard 33762ec HDFS-14011. RBF: Add more information to HdfsFileStatus for a
mount point. Contributed by Akira Ajisaka.
discard ee6c231 HDFS-13906. RBF: Add multiple paths for dfsrouteradmin 'rm'
and 'clrquota' commands. Contributed by Ayush Saxena.
add f09a78f HDDS-1339. Implement ratis snapshots on OM (#651)
add eb03f7c YARN-9303 Username splits won't help
timelineservice.app_flow table. Contributed by Prabhu Joseph.
add a92806d HDDS-1189. Recon Aggregate DB schema and ORM. Contributed by
Siddharth Wagle.
add 945e8c6 YARN-9394. Use new API of RackResolver to get better
performance. Contributed by Lantao Jin.
add 48a58bc HDDS-1207. Refactor Container Report Processing logic and
plugin new Replication Manager. (#662)
add 33e159b HDDS-1349. Remove watchClient from XceiverClientRatis.
Contributed by Shashikant Banerjee.
add d2637cb HDFS-14389. getAclStatus returns incorrect permissions and
owner when an iNodeAttributeProvider is configured. Contributed by Stephen
O'Donnell.
add 993f36e HDDS-1353 : Metrics
scm_pipeline_metrics_num_pipeline_creation_failed keeps increasing because of
BackgroundPipelineCreator. (#681)
add c907363 HADOOP-16208. Do Not Log InterruptedException in Client.
add 8d15006 YARN-9396. Fixed duplicated RM Container created event to
ATS. Contributed by Prabhu Joseph
add 2bf82ae YARN-9441. Updated YARN app catalog name for consistency.
Contributed by Weiwei Yang
add 215ffc7 HADOOP-16197 S3AUtils.translateException to map
CredentialInitializationException to AccessDeniedException
add d2141ef HDDS-1189. Build failing due to rat check failure introduced
by HDDS-1189. Contributed by Siddharth Wagle.
add 67020f0 HDFS-14407. Fix misuse of SLF4j logging API in
DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji.
add 77fe51e HDDS-1379. Convert all OM Volume related operations to HA
model. (#689)
add 260d843 HDFS-14416. Fix TestHdfsConfigFields for field
dfs.client.failover.resolver.useFQDN. Contributed by Fengnan Li.
add 0356b36 HDDS-1333. OzoneFileSystem can't work with spark/hadoop2.7
because incompatible security classes. Contributed by Elek, Marton. (#653)
add 89f9c86 HDDS-1385. Make the ozonesecure-mr environment definition
version independent. Contributed by Elek, Marton. (#695)
add 5750bb9 HDDS-1389. Fix testSCMChillModeRestrictedOp. (#696)
add 27039a2 YARN-9382 Publish container killed, paused and resumed events
to ATSv2. Contributed by Abhishesk Modi.
add 22362c8 YARN-9335 [atsv2] Restrict the number of elements held in
timeline collector when backend is unreachable for async calls. Contributed by
Abhishesk Modi.
add 26dc940 HDDS-1390. Fix OzoneS3 Gateway server due to exclusion of
hk2-api. (#698)
add 274ab4e HDDS-1332. Attempt to fix flaky test
testStartStopDatanodeStateMachine (#697)
add dc013f7 HDDS-1329. Update documentation for Ozone-0.4.0 release.
Contributed By Ajay Kumar.
add e9b859f HDFS-14371. Improve Logging in FSNamesystem by adding
parameterized logging. Contributed by Shweta.
add 1b9ba0e SUBMARINE-47. Provide an implementation to parse
configuration values from a YAML file for submarine run CLI. Contributed by
Szilard Nemeth.
add ec143cb YARN-9413. Queue resource leak after app fail for
CapacityScheduler. Contributed by Tao Yang.
add 0d47d28 HADOOP-10848. Cleanup calling of sun.security.krb5.Config.
add 72f4b9c HADOOP-15242. Fix typos in hadoop-functions.sh. Contributed
by Ray Chiang.
add fc05b0e YARN-9313. Support asynchronized scheduling mode and
multi-node lookup mechanism for scheduler activities. Contributed by Tao Yang.
add ab645b3 HADOOP-14635. Javadoc correction for
AccessControlList#buildACL. Contributed by Yeliang Cang.
add bb8dda2 HADOOP-12890. Fix typo in AbstractService. Contributed by
Gabor Liptak.
add 69e3745 HDDS-1340. Add List Containers API for Recon (#648)
add 2d4f6b6 HDDS-372. There are three buffer copies in BlockOutputStream.
Contributed by Shashikant Banerjee.
add 73f43ac HDDS-1372. getContainerWithPipeline for a standalone pipeline
fails with ConcurrentModificationException. (#682)
add 8ef3bc8 HDDS-1383. Create project skeleton with CLI interface for
In-place upgrade.
add a78bdd1 HDDS-1405. ITestOzoneContractCreate is failing. (#712)
add cfec455 YARN-999. In case of long running tasks, reduce node resource
should balloon out resource quickly by calling preemption API and suspending
running task. Contributed by Inigo Goiri.
add 312d4d4 HDDS-1388. Add a shell script to run MiniOzoneChaosCluster
using mvn exec. Contributed by Mukul Kumar Singh. (#709)
add 358e928 YARN-999. In case of long running tasks, reduce node resource
should balloon out resource quickly by calling preemption API and suspending
running task. Addendum. Contributed by Inigo Goiri.
add 25c421b HDDS-1393. Convert all OM Bucket related operations to HA
model. (#704)
add 32722d2 YARN-9433. Remove unused constants in YARN resource manager
add df01469 HDDS-1401. Static ContainerCache in Datanodes can result in
overwrite of container db. Contributed by Mukul Kumar Singh. (#708)
add 8c1bba3 YARN-9463. Add queueName info when failing with queue
capacity sanity check. Contributed by Aihua Xu.
add d8239f3 HADOOP-16240. start-build-env.sh can consume all disk space
during image creation. Contributed by Craig Condit.
add 65deb1a HADOOP-16179. hadoop-common pom should not depend on
kerb-simplekdc
add dfb518b HDDS-1397. Avoid the usage of signal handlers in datanodes of
the MiniOzoneClusters.
add 0e770a6 HDDS-1370. Command Execution in Datanode fails because of NPE
(#715)
add 813cee1 HDFS-14420. Fix typo in KeyShell console. Contributed by Hu
Xiaodong.
add feaab24 HDDS-1418. Move bang line to the start of the start-chaos.sh
script. Contributed by Arpit Agarwal. (#720)
add 8740755 HADOOP-16243. Change Log Level to trace in NetUtils.java.
Contributed by chencan.
add e9c4109 HDDS-1417. After successfully importing a container, datanode
should delete the container tar.gz file from working directory. (#721)
add 586826f HDFS-12245. Fix INodeId javadoc
add bdbca0e HADOOP-16052. Remove Subversion and Forrest from Dockerfile
add a0468c5 HDDS-1348. Refactor BlockOutpuStream Class. Contributed by
Shashikant Banerjee.
add ed3747c YARN-9435. Add Opportunistic Scheduler metrics in
ResourceManager. Contributed by Abhishek Modi.
add fcc7f9b HDDS-1410. TestSCMNodeMetrics is flaky. Contributed by
Siddharth Wagle.
add cf4efca HADOOP-16118. S3Guard to support on-demand DDB tables.
add bbdbc7a HADOOP-14544. DistCp documentation for command line options
is misaligned. Contributed by Masatake Iwasaki.
add c298415 HDDS-1404. Fix typos in HDDS. Contributed by bianqi.
add 4003849 HDDS-1416. MiniOzoneCluster should set custom value for
hdds.datanode.replication.work.dir. Contributed by chencan.
add 3b08ac4 HDDS-1363. ozone.metadata.dirs doesn't pick multiple dirs
(#691)
add fd676e1 HDDS-1426. Minor logging improvements for
MiniOzoneChaosCluster
add a9b46c5 HDDS-1421. Avoid unnecessary object allocations in TracingUtil
add 75ba719 HDDS-1419. Fix shellcheck errors in start-chaos.sh
add abace70 HDDS-1420. Tracing exception in DataNode HddsDispatcher
add a458c5b HDDS-1285. Implement actions need to be taken after chill
mode exit wait time. (#612)
add 0c1fec3 HDDS-1425. Ozone compose files are not compatible with the
latest docker-compose. (#727)
add 62f4808 HDDS-1428. Remove benign warning in handleCreateContainer.
Contributed by Siddharth Wagle.
add 5379d85 HDDS-1387. ConcurrentModificationException in
TestMiniChaosOzoneCluster. Contributed by Marton Elek. (#732)
add 626fec6 HDFS-13699. Add DFSClient sending handshake token to
DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen
Liang.
add 2382f63 HADOOP-14747. S3AInputStream to implement CanUnbuffer.
add 1943db5 HADOOP-16237. Fix new findbugs issues after updating guava to
27.0-jre.
add ebbda18 HDDS-1198. Rename chill mode to safe mode. Contributed by
Siddharth Wagle. (#737)
add b2cdf80 YARN-9281. Add express upgrade button to Appcatalog UI.
Contributed by Eric Yang
add 4b4fef2 HADOOP-16227. Upgrade checkstyle to 8.19
add 7a68e7a YARN-9474. Remove hard coded sleep from Opportunistic
Scheduler tests. Contributed by Abhishek Modi.
add c4c16ca YARN-9481. [JDK 11] Build fails due to hard-coded target
version in hadoop-yarn-applications-catalog-webapp. Contributed by Kei Kori.
add 7fa73fa YARN-9439. Support asynchronized scheduling mode and
multi-node lookup mechanism for app activities. Contributed by Tao Yang.
add 254efc9 YARN-9379. Can't specify docker runtime through environment.
Contributed by caozhiqiang
add 62e38ea HDDS-1396 : Recon start fails due to changes in Aggregate
Schema definition. (#700)
add 5583e1b YARN-7848 Force removal of docker containers that do not get
removed on first try. Contributed by Eric Yang
add b1120d2 HADOOP-16249. Make CallerContext LimitedPrivate scope to
Public.
add c65218e YARN-8943. Upgrade JUnit from 4 to 5 in hadoop-yarn-api.
add a5ceed2 HADOOP-16253. Update AssertJ to 3.12.2.
add 008766c HADOOP-15014. KMS should log the IP address of the clients.
Contributed by Zsombor Gegesy.
add bfcb653 HDDS-1380. Add functonality to write from multiple clients in
MiniOzoneChaosCluster. Contributed by Shashikant Banerjee.
add ad86588 YARN-8530. Add SPNEGO filter to application catalog.
Contributed by Eric Yang
add 2364c7d YARN-9466. Fixed application catalog navigation bar height in
Safari. Contributed by Eric Yang
add be6c801 HDFS-14418. Remove redundant super user priveledge checks
from namenode. Contributed by Ayush Saxena.
add b8086ae YARN-9123. Clean up and split testcases in TestNMWebServices
for GPU support. Contributed by Szilard Nemeth.
add f2ab279 HDDS-1432. Ozone client list command truncates response
without any indication. Contributed by Siddharth Wagle.
add 3ac3e50 HDDS-1374. ContainerStateMap cannot find container while
allocating blocks. (#735)
add e543c3b HDDS-1376. Datanode exits while executing client command when
scmId is null (#724)
add 9cf7401 YARN-9349. Improved log level practices for
InvalidStateTransitionException. Contributed by Anuhan Torgonshar
add 0a87fd1 HDDS-1434. TestDatanodeStateMachine is flaky (#740)
add 04c0437 HDDS-1402. Remove unused ScmBlockLocationProtocol from
ObjectStoreHandler (#707)
add 22d0468 HDDS-1373. KeyOutputStream, close after write request fails
after retries, runs into IllegalArgumentException. Contributed by Shashikant
Banerjee
add 4ff4314 Merge pull request #729 from bshashikant/HDDS-1373
add e5a918c Revert "Merge pull request #729 from bshashikant/HDDS-1373"
add df2ae27 HDDS-1373. KeyOutputStream, close after write request fails
after retries, runs into IllegalArgumentException..(#729)
add 082f1e0 Revert "HDDS-1373. KeyOutputStream, close after write request
fails after retries, runs into IllegalArgumentException..(#729)"
add d608be6 HDDS-1373. KeyOutputStream, close after write request fails
after retries, runs into IllegalArgumentException.(#729)
add 75e94b5 HADOOP-15881. Remove JUnit from LICENSE.txt
add 13907d8 HDFS-14432. dfs.datanode.shared.file.descriptor.paths
duplicated in hdfs-default.xml
add 685cb83 HDFS-14433. Remove the extra empty space in the DataStreamer
logging. Contributed by Yishuang Lu. (#747)
add 6e4399e YARN-9487. NodeManager native build shouldn't link against
librt on macOS. Contributed by Siyao Meng.
add aa4c744 YARN-9470. Fix order of actual and expected expression in
assert statements
add 8aea5e4 HDDS-1433. Rename GetScmInfoRespsonseProto to
GetScmInfoResponseProto due to typos. Contributed by Wanqiang Ji.
add b979fdd HDDS-1447. Fix CheckStyle warnings. Contributed by Wanqiang
Ji.
add df76cdc YARN-6695. Fixed NPE in publishing appFinished events to
ATSv2. Contributed by Prabhu Joseph
add ef97a20 YARN-8622. Fixed container-executor compilation on MacOSX.
Contributed by Siyao Meng
add 518f47b HDDS-976: Parse network topology from yaml file. Contributed
by Junjie Chen. (#661)
add aeadb94 YARN-9448. Fix Opportunistic Scheduling for node local
allocations. Contributed by Abhishek Modi.
add 317fcba YARN-9254. Add support for storing application catalog data
to HDFS. Contributed by Eric Yang
add adefd37 YARN-9495. Fix findbugs warnings in
hadoop-yarn-server-resourcemanager module. Contributed by Tao Yang.
add 1ddb488 HADOOP-16265. Fix bug causing Configuration#getTimeDuration
to use incorrect units when the default value is used. Contributed by starphin.
add 96e3027 YARN-2889. Limit the number of opportunistic container
allocated per AM heartbeat. Contributed by Abhishek Modi.
add 5321235 HDFS-14445. TestTrySendErrorReportWhenNNThrowsIOException
fails in trunk. Contributed by Ayush Saxena.
add 174b7d3 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to
gracefully handle StandbyException when fetching HAServiceState. Contributed by
Erik Krogen.
add fb1c549 HDFS-14374. Expose total number of delegation tokens in
AbstractDelegationTokenSecretManager. Contributed by CR Hota.
add a54c1e3 YARN-8551. Project setup for MaWo application.
Contributed by Yesha Vora
add f4ab937 HADOOP-16026:Replace incorrect use of system property
user.name. Contributed by Dinesh Chitlangia.
add 1c8046d YARN-9325. TestQueueManagementDynamicEditPolicy fails
intermittent. Contributed by Prabhu Joseph.
add e79a9c1 SUBMARINE-40. Add TonY runtime to Submarine. Contributed by
Keqiu Hu.
add 8a95ea6 YARN-9475. [YARN-9473] Create basic VE plugin. Contributed by
Peter Bacsko.
add 7e1f8d3 HDDS-1368. Cleanup old ReplicationManager code from SCM.
add 59ded76 HDDS-1411. Add unit test to check if SCM correctly sends
close commands for containers in closing state after a restart. (#755)
add c504eee YARN-9339. Apps pending metric incorrect after moving app to
a new queue. Contributed by Abhishek Modi.
add 4a0ba24 YARN-9491.
TestApplicationMasterServiceFair#ApplicationMasterServiceTestBase.testUpdateTrackingUrl
fails intermittent. Contributed by Prabhu Joseph.
add fec9bf4 YARN-9501.
TestCapacitySchedulerOvercommit#testReducePreemptAndCancel fails intermittent.
Contributed by Prabhu Joseph.
add 9d40062 YARN-9081. Update jackson from 1.9.13 to 2.x in
hadoop-yarn-services-core.
add 3f2f418 YARN-9424. Change getDeclaredMethods to getMethods in
FederationClientInterceptor#invokeConcurrent. Contributed by Shen Yinjie.
add 64f30da HDDS-1450. Fix nightly run failures after HDDS-976.
Contributed by Xiaoyu Yao. (#757)
add 3f787cd HDDS-1192. Support -conf command line argument in GenericCli
add e95c055 HDDS-1457. Sequence of configuring Ozone File System
instruction needs some work. Contributed by Eric Yang.
add e1c5ddf HADOOP-16252. Add prefix to dynamo tables in tests.
add a703dae HADOOP-16222. Fix new deprecations after guava 27.0 update in
trunk. Contributed by Gabor Bota.
add afe6613 MAPREDUCE-7200. Remove stale eclipse templates. Contributed
by Wanqiang Ji.
add 0b3d41b SUBMARINE-54. Add test coverage for YarnServiceJobSubmitter
and make it ready for extension for PyTorch. Contributed by Szilard Nemeth.
add b5dcf64 SUBMARINE-44. Upgrade Docker image's tensorflow version to
1.13.1. Contributed by Zac Zhou.
add 79d3d35 YARN-9486. Docker container exited with failure does not get
clean up correctly. Contributed by Eric Yang
add c35abcd HDDS-999. Make the DNS resolution in OzoneManager more
resilient
add 556eafd HDDS-1460: Add the optmizations of HDDS-1300 to
BasicOzoneFileSystem (#765)
add 3758270 HDDS-1403. KeyOutputStream writes fails after max retries
while writing to a closed container (#753)
add 43b2a4b HDDS-1456. Stop the datanode, when any datanode statemachine
state is… (#769)
add b434f55 HDDS-1471. Update ratis dependency to 0.3.0. Contributed by
Ajay Kumar. (#777)
add 1cef194 HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream.
add dbce729 HDDS-1472. Add retry to kinit command in smoketests.
Contributed by Ajay Kumar. (#778)
add a96d16a HDDS-1455. Inconsistent naming convention with Ozone Kerberos
configuration. Contributed by Xiaoyu Yao. (#762)
add 5bca062 HDDS-1476. Fix logIfNeeded logic in EndPointStateMachine.
(#779)
add 59816df HDDS-1462. Fix content and format of Ozone documentation.
Contributed by Elek, Marton. (#767)
add 95790bb HDDS-1430. NPE if secure ozone if KMS uri is not defined.
Contributed by Ajay Kumar. (#752)
add 4b4200f HDFS-13677. Dynamic refresh Disk configuration results in
overwriting VolumeMap. Contributed by xuzq.
add 382a962 Revert "HDDS-973. HDDS/Ozone fail to build on Windows."
add 24f218a SUBMARINE-64. Improve TonY runtime's document. Contributed by
Keqiu Hu.
add 7fbaa7d YARN-9476. [YARN-9473] Create unit tests for VE plugin.
Contributed by Peter Bacsko.
add 0af4011 HADOOP-16221. S3Guard: add option to fail operation on
metadata write failure.
add dead9b4 HDDS-1384. TestBlockOutputStreamWithFailures is failing
add accb811 YARN-6929. Improved partition algorithm for yarn
remote-app-log-dir. Contributed by Prabhu Joseph
add 4877f0a HDFS-3246: pRead equivalent for direct read path (#597)
add 7cb46f0 HDFS-14463. Add Log Level link under NameNode and DataNode
Web UI Utilities dropdown. Contributed by Siyao Meng.
add f682a17 HDDS-1483. Fix getMultipartKey javadoc. (#790)
add 2ab318b HDDS-1412. Provide example k8s deployment files as part of
the release package
add f1673b0 HDDS-1478. Provide k8s resources files for prometheus and
performance tests
add a2887f5 HDDS-1468. Inject configuration values to Java objects
add 3cb1d09 HDDS-1479. Update S3.md documentation
add e2f0f72 HDDS-1469. Generate default configuration fragments based on
annotations
add 4605db3 HDDS-1482. Use strongly typed codec implementations for the
S3Table
add 6a42745 HADOOP-16183. Use latest Yetus to support ozone specific
build process
new 55b0442 HDFS-13906. RBF: Add multiple paths for dfsrouteradmin 'rm'
and 'clrquota' commands. Contributed by Ayush Saxena.
new a99dc9a HDFS-14011. RBF: Add more information to HdfsFileStatus for a
mount point. Contributed by Akira Ajisaka.
new 8e1e520 HDFS-13845. RBF: The default MountTableResolver should fail
resolving multi-destination paths. Contributed by yanghuafeng.
new e450f07 HDFS-14024. RBF: ProvidedCapacityTotal json exception in
NamenodeHeartbeatService. Contributed by CR Hota.
new 68e5200 HDFS-12284. RBF: Support for Kerberos authentication.
Contributed by Sherwood Zheng and Inigo Goiri.
new a5cee7c HDFS-12284. addendum to HDFS-12284. Contributed by Inigo
Goiri.
new 7241668 HDFS-13852. RBF: The DN_REPORT_TIME_OUT and
DN_REPORT_CACHE_EXPIRE should be configured in RBFConfigKeys. Contributed by
yanghuafeng.
new bf1111e HDFS-13834. RBF: Connection creator thread should catch
Throwable. Contributed by CR Hota.
new 628fc5a HDFS-14082. RBF: Add option to fail operations when a
subcluster is unavailable. Contributed by Inigo Goiri.
new 5ee7fb9 HDFS-13776. RBF: Add Storage policies related ClientProtocol
APIs. Contributed by Dibyendu Karmakar.
new f04896d HDFS-14089. RBF: Failed to specify server's Kerberos pricipal
name in NamenodeHeartbeatService. Contributed by Ranith Sardar.
new aa14e12 HDFS-14085. RBF: LS command for root shows wrong owner and
permission information. Contributed by Ayush Saxena.
new 27fe4c7 HDFS-14114. RBF: MIN_ACTIVE_RATIO should be configurable.
Contributed by Fei Hui.
new d17a51c Revert "HDFS-14114. RBF: MIN_ACTIVE_RATIO should be
configurable. Contributed by Fei Hui."
new 40945c3 HDFS-14114. RBF: MIN_ACTIVE_RATIO should be configurable.
Contributed by Fei Hui.
new bdb1ff2 HDFS-14152. RBF: Fix a typo in RouterAdmin usage. Contributed
by Ayush Saxena.
new 8360890 HDFS-13869. RBF: Handle NPE for
NamenodeBeanMetrics#getFederationMetrics. Contributed by Ranith Sardar.
new cce6ae5 HDFS-14151. RBF: Make the read-only column of Mount Table
clearly understandable.
new 00f4bb2 HDFS-13443. RBF: Update mount table cache immediately after
changing (add/update/remove) mount table entries. Contributed by Mohammad
Arshad.
new 2828b09 HDFS-14167. RBF: Add stale nodes to federation metrics.
Contributed by Inigo Goiri.
new 45895e6 HDFS-14161. RBF: Throw StandbyException instead of
IOException so that client can retry when can not get connection. Contributed
by Fei Hui.
new d3370f3 HDFS-14150. RBF: Quotas of the sub-cluster should be removed
when removing the mount point. Contributed by Takanobu Asanuma.
new d90d20f HDFS-14191. RBF: Remove hard coded router status from
FederationMetrics. Contributed by Ranith Sardar.
new b1d250f HDFS-13856. RBF: RouterAdmin should support dfsrouteradmin
-refreshRouterArgs command. Contributed by yanghuafeng.
new a48c057 HDFS-14206. RBF: Cleanup quota modules. Contributed by Inigo
Goiri.
new 50cb4a2 HDFS-14129. RBF: Create new policy provider for router.
Contributed by Ranith Sardar.
new 2d8ebf5 HDFS-14129. addendum to HDFS-14129. Contributed by Ranith
Sardar.
new 17b77c9 HDFS-14193. RBF: Inconsistency with the Default Namespace.
Contributed by Ayush Saxena.
new 2e50862 HDFS-14156. RBF: rollEdit() command fails with Router.
Contributed by Shubham Dewan.
new a9aadbb HDFS-14209. RBF: setQuota() through router is working for
only the mount Points under the Source column in MountTable. Contributed by
Shubham Dewan.
new 9335ec5 HDFS-14223. RBF: Add configuration documents for using
multiple sub-clusters. Contributed by Takanobu Asanuma.
new 6fe8017 HDFS-14224. RBF: NPE in getContentSummary() for getEcPolicy()
in case of multiple destinations. Contributed by Ayush Saxena.
new 4d7cf87 HDFS-14215. RBF: Remove dependency on availability of default
namespace. Contributed by Ayush Saxena.
new c14bd63 HDFS-13404. RBF:
TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails.
new d6aeb7c HDFS-14225. RBF : MiniRouterDFSCluster should configure the
failover proxy provider for namespace. Contributed by Ranith Sardar.
new b3b157e HDFS-14252. RBF : Exceptions are exposing the actual sub
cluster path. Contributed by Ayush Saxena.
new 92cd6e2 HDFS-14230. RBF: Throw RetriableException instead of
IOException when no namenodes available. Contributed by Fei Hui.
new a0138b8 HDFS-13358. RBF: Support for Delegation Token (RPC).
Contributed by CR Hota.
new f63b6a2 HDFS-14226. RBF: Setting attributes should set on all
subclusters' directories. Contributed by Ayush Saxena.
new 35a7e46 HDFS-14268. RBF: Fix the location of the DNs in
getDatanodeReport(). Contributed by Inigo Goiri.
new 42886a0 HDFS-14249. RBF: Tooling to identify the subcluster location
of a file. Contributed by Inigo Goiri.
new 8223879 HDFS-14307. RBF: Update tests to use internal Whitebox
instead of Mockito. Contributed by CR Hota.
new 12924f2 HDFS-14052. RBF: Use Router keytab for WebHDFS. Contributed
by CR Hota.
new 9204211 HDFS-14322. RBF: Security manager should not load if security
is disabled. Contributed by CR Hota.
new 0ed0226 HDFS-14259. RBF: Fix safemode message for Router. Contributed
by Ranith Sadar.
new baa2cb1 HDFS-14329. RBF: Add maintenance nodes to federation metrics.
Contributed by Ayush Saxena.
new 64ca9c0 HDFS-14331. RBF: IOE While Removing Mount Entry. Contributed
by Ayush Saxena.
new 9e39186 HDFS-14335. RBF: Fix heartbeat typos in the Router.
Contributed by CR Hota.
new 364811a HDFS-14334. RBF: Use human readable format for long numbers
in the Router UI. Contributed by Inigo Goiri.
new de22e9b HDFS-14343. RBF: Fix renaming folders spread across multiple
subclusters. Contributed by Ayush Saxena.
new 09f20d0 HDFS-14351. RBF: Optimize configuration item resolving for
monitor namenode. Contributed by He Xiaoqiao and Inigo Goiri.
new b0bc109 HDFS-14388. RBF: Prevent loading metric system when disabled.
Contributed by Inigo Goiri.
new de204c3 HDFS-14316. RBF: Support unavailable subclusters for mount
points with multiple destinations. Contributed by Inigo Goiri.
new 9b2e8d4 HDFS-13853. RBF: RouterAdmin update cmd is overwriting the
entry not updating the existing. Contributed by Ayush Saxena.
new 940c0c7 HDFS-14369. RBF: Fix trailing / for webhdfs. Contributed by
Akira Ajisaka.
new e8c6c20 HDFS-14422. RBF: Router shouldn't allow READ operations in
safe mode. Contributed by Inigo Goiri.
new 91fea6f HDFS-13972. RBF: Support for Delegation Token (WebHDFS).
Contributed by CR Hota.
new 893c708 HDFS-14457. RBF: Add order text SPACE in CLI command 'hdfs
dfsrouteradmin'. Contributed by luhuachao.
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (aeb3b61)
\
N -- N -- N refs/heads/HDFS-13891 (893c708)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 58 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
Jenkinsfile | 4 +-
LICENSE.txt | 1 -
dev-support/docker/Dockerfile | 17 -
.../security/authentication/util/KerberosUtil.java | 49 +-
hadoop-common-project/hadoop-common/pom.xml | 2 +-
.../hadoop-common/src/main/bin/hadoop-functions.sh | 4 +-
.../java/org/apache/hadoop/conf/Configuration.java | 2 +-
.../apache/hadoop/crypto/CryptoInputStream.java | 292 +++--
.../org/apache/hadoop/crypto/key/KeyShell.java | 2 +-
.../org/apache/hadoop/fs/AbstractFileSystem.java | 13 +-
.../hadoop/fs/ByteBufferPositionedReadable.java | 66 ++
.../org/apache/hadoop/fs/ByteBufferReadable.java | 17 +-
.../org/apache/hadoop/fs/FSDataInputStream.java | 15 +-
.../main/java/org/apache/hadoop/fs/FileSystem.java | 10 +-
.../org/apache/hadoop/fs/StreamCapabilities.java | 6 +
.../java/org/apache/hadoop/io/ArrayWritable.java | 2 +-
.../java/org/apache/hadoop/ipc/CallerContext.java | 3 +-
.../main/java/org/apache/hadoop/ipc/Client.java | 8 +-
.../main/java/org/apache/hadoop/net/NetUtils.java | 2 +-
.../security/authorize/AccessControlList.java | 6 +-
.../hadoop/security/token/SecretManager.java | 2 +-
.../AbstractDelegationTokenSecretManager.java | 9 +-
.../org/apache/hadoop/service/AbstractService.java | 2 +-
.../hadoop/util/SemaphoredDelegatingExecutor.java | 6 +-
.../main/java/org/apache/hadoop/util/ZKUtil.java | 2 +-
.../src/main/resources/core-default.xml | 13 +
.../site/markdown/filesystem/fsdatainputstream.md | 37 +
.../org/apache/hadoop/conf/TestConfiguration.java | 4 +
.../hadoop/crypto/CryptoStreamsTestBase.java | 185 ++-
.../apache/hadoop/crypto/TestCryptoStreams.java | 35 +-
.../hadoop/crypto/TestCryptoStreamsForLocalFS.java | 10 +
.../hadoop/crypto/TestCryptoStreamsNormal.java | 10 +
.../fs/contract/AbstractContractUnbufferTest.java | 125 ++
.../apache/hadoop/fs/contract/ContractOptions.java | 5 +
.../org/apache/hadoop/io/TestArrayWritable.java | 2 +-
.../org/apache/hadoop/net/TestTableMapping.java | 16 +-
.../apache/hadoop/security/TestSecurityUtil.java | 3 +-
.../token/delegation/TestDelegationToken.java | 23 +
.../java/org/apache/hadoop/util/TestZKUtil.java | 2 +-
.../hadoop-kms/dev-support/findbugsExcludeFile.xml | 8 +
.../key/kms/server/KMSExceptionsProvider.java | 5 +-
.../hadoop/crypto/key/kms/server/KMSMDCFilter.java | 58 +-
.../registry/client/impl/zk/RegistrySecurity.java | 17 +-
.../apache/hadoop/hdds/scm/XceiverClientRatis.java | 28 +-
.../hdds/scm/client/ContainerOperationClient.java | 14 +-
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 142 +--
.../hadoop/hdds/scm/storage/CommitWatcher.java | 237 ++++
hadoop-hdds/common/pom.xml | 10 +
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 44 +-
.../java/org/apache/hadoop/hdds/HddsUtils.java | 24 -
.../org/apache/hadoop/hdds/cli/GenericCli.java | 12 +-
.../hdds/cli/MissingSubcommandException.java | 14 +-
.../hadoop/hdds/conf/OzoneConfiguration.java | 138 ++-
.../apache/hadoop/hdds/scm/ByteStringHelper.java | 69 ++
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 14 +-
.../apache/hadoop/hdds/scm/client/ScmClient.java | 10 +-
.../hadoop/hdds/scm/container/ContainerInfo.java | 7 -
.../scm/container/common/helpers/ExcludeList.java | 6 +
.../org/apache/hadoop/hdds/scm/net/NodeSchema.java | 43 +-
.../hadoop/hdds/scm/net/NodeSchemaLoader.java | 86 +-
.../hadoop/hdds/scm/net/NodeSchemaManager.java | 8 +-
.../protocol/StorageContainerLocationProtocol.java | 10 +-
...lockLocationProtocolClientSideTranslatorPB.java | 2 +-
...inerLocationProtocolClientSideTranslatorPB.java | 38 +-
.../hdds/scm/storage/ContainerProtocolCalls.java | 2 +-
.../apache/hadoop/hdds/tracing/TracingUtil.java | 12 +-
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 16 +-
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 3 +
.../org/apache/hadoop/ozone/audit/SCMAction.java | 4 +-
.../org/apache/hadoop/ozone/common/Checksum.java | 38 +-
.../org/apache/hadoop/ozone/common/Storage.java | 2 +-
...lockLocationProtocolServerSideTranslatorPB.java | 4 +-
...inerLocationProtocolServerSideTranslatorPB.java | 32 +-
.../java/org/apache/hadoop/utils/db/DBStore.java | 6 +
.../java/org/apache/hadoop/utils/db/RDBStore.java | 11 +
.../src/main/proto/DatanodeContainerProtocol.proto | 4 +-
.../src/main/proto/ScmBlockLocationProtocol.proto | 4 +-
.../proto/StorageContainerLocationProtocol.proto | 26 +-
hadoop-hdds/common/src/main/proto/hdds.proto | 6 +-
.../main/resources/network-topology-default.yaml | 61 +
.../common/src/main/resources/ozone-default.xml | 171 ++-
.../hadoop/hdds/conf/SimpleConfiguration.java | 88 ++
.../hadoop/hdds/conf/TestOzoneConfiguration.java | 32 +
.../hadoop/hdds/scm/net/TestNodeSchemaLoader.java | 6 +-
.../hadoop/hdds/scm/net/TestYamlSchemaLoader.java | 90 ++
.../resources/networkTopologyTestFiles/good.yaml | 59 +
.../networkTopologyTestFiles/middle-leaf.yaml | 59 +
.../networkTopologyTestFiles/multiple-root.yaml | 59 +
hadoop-hdds/config/pom.xml | 66 ++
.../java/org/apache/hadoop/hdds/conf/Config.java | 59 +
.../hadoop/hdds/conf/ConfigFileAppender.java | 127 ++
.../hadoop/hdds/conf/ConfigFileGenerator.java | 113 ++
.../org/apache/hadoop/hdds/conf/ConfigGroup.java | 32 +
.../org/apache/hadoop/hdds/conf/ConfigTag.java | 44 +
.../org/apache/hadoop/hdds/conf/ConfigType.java | 34 +
.../hadoop/hdds/conf/ConfigurationException.java | 34 +
.../org/apache/hadoop/hdds/conf/package-info.java | 22 +
.../services/javax.annotation.processing.Processor | 16 +
.../hadoop/hdds/conf/ConfigurationExample.java | 89 ++
.../hadoop/hdds/conf/TestConfigFileAppender.java | 48 +
.../org/apache/hadoop/hdds/conf/package-info.java | 24 +
hadoop-hdds/container-service/pom.xml | 2 +-
.../org/apache/hadoop/hdds/scm/HddsServerUtil.java | 22 +
.../apache/hadoop/ozone/HddsDatanodeService.java | 148 ++-
.../hadoop/ozone/HddsDatanodeStopService.java | 27 +
.../common/statemachine/DatanodeStateMachine.java | 15 +-
.../common/statemachine/EndpointStateMachine.java | 7 +-
.../common/statemachine/SCMConnectionManager.java | 13 +-
.../common/statemachine/StateContext.java | 87 +-
.../CloseContainerCommandHandler.java | 2 -
.../DeleteContainerCommandHandler.java | 4 -
.../ReplicateContainerCommandHandler.java | 28 +-
.../common/states/datanode/InitDatanodeState.java | 4 +-
.../states/datanode/RunningDatanodeState.java | 11 +-
.../states/endpoint/VersionEndpointTask.java | 3 +-
.../container/common/utils/ContainerCache.java | 36 +-
.../ozone/container/common/volume/VolumeSet.java | 10 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 7 +-
.../container/keyvalue/helpers/BlockUtils.java | 2 +-
.../container/keyvalue/helpers/ChunkUtils.java | 5 +-
.../container/keyvalue/impl/BlockManagerImpl.java | 2 +-
.../ozone/container/ozoneimpl/OzoneContainer.java | 3 +-
.../replication/DownloadAndImportReplicator.java | 7 +-
.../hadoop/ozone/TestHddsDatanodeService.java | 7 +-
.../hadoop/ozone/TestHddsSecureDatanodeInit.java | 4 +-
.../container/common/TestDatanodeStateMachine.java | 32 +-
.../container/common/impl/TestHddsDispatcher.java | 1 -
.../TestCloseContainerCommandHandler.java | 15 +-
.../common/volume/TestVolumeSetDiskChecks.java | 9 +-
.../container/keyvalue/TestKeyValueContainer.java | 2 +-
hadoop-hdds/docs/content/BucketCommands.md | 4 +-
hadoop-hdds/docs/content/BuildingSources.md | 30 +-
hadoop-hdds/docs/content/CommandShell.md | 2 +
hadoop-hdds/docs/content/KeyCommands.md | 2 +-
hadoop-hdds/docs/content/OzoneFS.md | 39 +-
.../docs/content/OzoneSecurityArchitecture.md | 110 ++
hadoop-hdds/docs/content/Prometheus.md | 6 +-
hadoop-hdds/docs/content/RunningViaDocker.md | 8 +-
hadoop-hdds/docs/content/RunningWithHDFS.md | 4 +-
hadoop-hdds/docs/content/S3.md | 29 +-
hadoop-hdds/docs/content/S3Commands.md | 41 -
hadoop-hdds/docs/content/Settings.md | 4 +-
hadoop-hdds/docs/content/SetupSecureOzone.md | 98 ++
hadoop-hdds/docs/content/SparkOzoneFSK8S.md | 6 +-
hadoop-hdds/docs/content/VolumeCommands.md | 4 +-
hadoop-hdds/docs/pom.xml | 11 +-
.../docs/static/prometheus-key-allocate.png | Bin 0 -> 51155 bytes
hadoop-hdds/docs/static/prometheus.png | Bin 0 -> 38962 bytes
.../themes/ozonedoc/layouts/partials/sidebar.html | 16 +-
.../docs/themes/ozonedoc/static/css/ozonedoc.css | 11 +
.../org/apache/hadoop/hdds/server/ServerUtils.java | 25 +-
.../apache/hadoop/hdds/server/TestServerUtils.java | 123 ++
hadoop-hdds/pom.xml | 9 +-
.../java/org/apache/hadoop/hdds/scm/ScmUtils.java | 22 +-
.../apache/hadoop/hdds/scm/block/BlockManager.java | 6 +-
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 22 +-
.../hdds/scm/chillmode/ChillModeExitRule.java | 110 --
.../hdds/scm/chillmode/ChillModeHandler.java | 106 --
.../hdds/scm/chillmode/ChillModePrecheck.java | 69 --
.../hdds/scm/chillmode/ChillModeRestrictedOps.java | 41 -
.../hdds/scm/chillmode/ContainerChillModeRule.java | 124 --
.../hdds/scm/chillmode/DataNodeChillModeRule.java | 82 --
.../chillmode/HealthyPipelineChillModeRule.java | 167 ---
.../chillmode/OneReplicaPipelineChillModeRule.java | 153 ---
.../apache/hadoop/hdds/scm/chillmode/Precheck.java | 29 -
.../hdds/scm/chillmode/SCMChillModeManager.java | 242 ----
.../hadoop/hdds/scm/chillmode/package-info.java | 18 -
.../scm/command/CommandStatusReportHandler.java | 43 +-
.../container/AbstractContainerReportHandler.java | 236 ++++
.../hdds/scm/container/ContainerReportHandler.java | 234 ++--
.../hdds/scm/container/ContainerStateManager.java | 19 +-
.../container/DeleteContainerCommandWatcher.java | 56 -
.../IncrementalContainerReportHandler.java | 47 +-
.../hdds/scm/container/ReplicationManager.java | 108 +-
.../hdds/scm/container/ReportHandlerHelper.java | 365 ------
.../hdds/scm/container/SCMContainerManager.java | 37 +-
.../replication/ReplicationActivityStatus.java | 6 +-
.../replication/ReplicationCommandWatcher.java | 56 -
.../container/replication/ReplicationManager.java | 384 ------
.../container/replication/ReplicationQueue.java | 73 --
.../container/replication/ReplicationRequest.java | 123 --
.../scm/container/states/ContainerStateMap.java | 11 +-
.../apache/hadoop/hdds/scm/events/SCMEvents.java | 73 +-
.../hadoop/hdds/scm/exceptions/SCMException.java | 2 +-
.../hadoop/hdds/scm/node/DeadNodeHandler.java | 186 +--
.../hadoop/hdds/scm/node/SCMNodeManager.java | 6 +-
.../pipeline/InsufficientDatanodesException.java | 36 +
.../hdds/scm/pipeline/PipelineReportHandler.java | 16 +-
.../hdds/scm/pipeline/RatisPipelineProvider.java | 2 +-
.../hdds/scm/pipeline/SCMPipelineManager.java | 2 +
.../hdds/scm/safemode/ContainerSafeModeRule.java | 124 ++
.../hdds/scm/safemode/DataNodeSafeModeRule.java | 82 ++
.../scm/safemode/HealthyPipelineSafeModeRule.java | 167 +++
.../safemode/OneReplicaPipelineSafeModeRule.java | 153 +++
.../apache/hadoop/hdds/scm/safemode/Precheck.java | 29 +
.../hdds/scm/safemode/SCMSafeModeManager.java | 242 ++++
.../hadoop/hdds/scm/safemode/SafeModeExitRule.java | 110 ++
.../hadoop/hdds/scm/safemode/SafeModeHandler.java | 147 +++
.../hadoop/hdds/scm/safemode/SafeModePrecheck.java | 69 ++
.../hdds/scm/safemode/SafeModeRestrictedOps.java | 41 +
.../hadoop/hdds/scm/safemode/package-info.java | 18 +
.../hdds/scm/server/SCMBlockProtocolServer.java | 4 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 44 +-
.../hadoop/hdds/scm/server/SCMConfigurator.java | 24 +-
.../apache/hadoop/hdds/scm/server/SCMMXBean.java | 8 +-
.../hdds/scm/server/StorageContainerManager.java | 104 +-
.../main/resources/webapps/scm/scm-overview.html | 4 +-
.../hadoop/hdds/scm/TestHddsServerUtils.java | 49 -
.../java/org/apache/hadoop/hdds/scm/TestUtils.java | 1 +
.../hadoop/hdds/scm/block/TestBlockManager.java | 59 +-
.../hdds/scm/chillmode/TestChillModeHandler.java | 104 --
.../TestHealthyPipelineChillModeRule.java | 264 -----
.../TestOneReplicaPipelineChillModeRule.java | 209 ----
.../scm/chillmode/TestSCMChillModeManager.java | 469 --------
.../hadoop/hdds/scm/chillmode/package-info.java | 21 -
.../command/TestCommandStatusReportHandler.java | 5 -
.../hadoop/hdds/scm/container/MockNodeManager.java | 12 +-
.../scm/container/TestContainerReportHandler.java | 585 ++++-----
.../scm/container/TestContainerReportHelper.java | 73 --
.../TestIncrementalContainerReportHandler.java | 158 +--
.../hdds/scm/container/TestReplicationManager.java | 22 +-
.../scm/container/TestSCMContainerManager.java | 50 +-
.../replication/TestReplicationActivityStatus.java | 85 --
.../replication/TestReplicationManager.java | 290 -----
.../replication/TestReplicationQueue.java | 134 ---
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 63 +-
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 4 +-
.../hadoop/hdds/scm/node/TestStatisticsUpdate.java | 4 +-
.../safemode/TestHealthyPipelineSafeModeRule.java | 264 +++++
.../TestOneReplicaPipelineSafeModeRule.java | 209 ++++
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 469 ++++++++
.../hdds/scm/safemode/TestSafeModeHandler.java | 119 ++
.../hadoop/hdds/scm/safemode/package-info.java | 21 +
.../scm/server/TestSCMClientProtocolServer.java | 22 +-
.../ozone/container/common/TestEndPoint.java | 19 +-
hadoop-hdds/tools/pom.xml | 3 +-
.../hdds/scm/cli/ChillModeCheckSubcommand.java | 61 -
.../hadoop/hdds/scm/cli/ChillModeCommands.java | 60 -
.../hdds/scm/cli/ChillModeExitSubcommand.java | 57 -
.../org/apache/hadoop/hdds/scm/cli/SCMCLI.java | 2 +-
.../hdds/scm/cli/SafeModeCheckSubcommand.java | 61 +
.../hadoop/hdds/scm/cli/SafeModeCommands.java | 59 +
.../hdds/scm/cli/SafeModeExitSubcommand.java | 57 +
.../org/apache/hadoop/hdfs/DFSInputStream.java | 14 +-
.../java/org/apache/hadoop/hdfs/DataStreamer.java | 4 +-
.../hadoop/hdfs/client/HdfsClientConfigKeys.java | 5 +-
.../datatransfer/sasl/DataTransferSaslUtil.java | 72 ++
.../datatransfer/sasl/SaslDataTransferClient.java | 117 +-
.../namenode/ha/ObserverReadProxyProvider.java | 38 +-
.../src/main/proto/datatransfer.proto | 6 +
.../src/main/native/libhdfs-tests/hdfs_test.h | 18 +
.../main/native/libhdfs-tests/test_libhdfs_ops.c | 142 ++-
.../src/main/native/libhdfs/hdfs.c | 210 +++-
.../java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 4 +
.../datatransfer/sasl/SaslDataTransferServer.java | 53 +-
.../apache/hadoop/hdfs/server/datanode/DNConf.java | 6 +
.../hadoop/hdfs/server/datanode/DataNode.java | 7 +-
.../hadoop/hdfs/server/datanode/DataXceiver.java | 14 +-
.../datanode/checker/DatasetVolumeChecker.java | 2 +-
.../datanode/fsdataset/impl/FsDatasetImpl.java | 2 +-
.../server/datanode/fsdataset/impl/ReplicaMap.java | 14 +
.../hadoop/hdfs/server/namenode/FSDirAclOp.java | 9 +-
.../hadoop/hdfs/server/namenode/FSNamesystem.java | 26 +-
.../hadoop/hdfs/server/namenode/INodeId.java | 8 +-
.../hdfs/server/namenode/NameNodeRpcServer.java | 1 -
.../src/main/resources/hdfs-default.xml | 36 +-
.../src/main/webapps/datanode/datanode.html | 1 +
.../src/main/webapps/hdfs/dfshealth.html | 1 +
.../src/main/webapps/hdfs/explorer.html | 1 +
.../fs/contract/hdfs/TestHDFSContractUnbuffer.java | 46 +
.../apache/hadoop/hdfs/TestByteBufferPread.java | 269 +++++
.../hadoop/hdfs/TestDistributedFileSystem.java | 55 +
.../apache/hadoop/hdfs/TestHAAuxiliaryPort.java | 2 +-
.../apache/hadoop/hdfs/TestMultipleNNPortQOP.java | 219 ++++
.../hdfs/server/datanode/TestBPOfferService.java | 2 +-
.../datanode/TestDataNodeHotSwapVolumes.java | 68 ++
.../datanode/TestDataXceiverBackwardsCompat.java | 2 +-
.../datanode/fsdataset/impl/TestReplicaMap.java | 22 +
.../namenode/TestINodeAttributeProvider.java | 33 +
.../hadoop/hdfs/server/namenode/ha/HATestUtil.java | 20 +-
.../namenode/ha/TestDelegationTokensWithHA.java | 46 +
.../hdfs/tools/TestDFSHAAdminMiniCluster.java | 6 +-
.../src/test/resources/contract/hdfs.xml | 5 +
.../.launches/AllMapredTests.launch | 28 -
.../.eclipse.templates/.launches/JobTracker.launch | 24 -
.../.launches/SpecificTestTemplate.launch | 28 -
.../.launches/TaskTracker.launch | 24 -
.../.eclipse.templates/README.txt | 6 -
.../hadoop/ozone/client/OzoneClientUtils.java | 5 +-
.../hadoop/ozone/client/io/KeyOutputStream.java | 212 ++--
.../hadoop/ozone/client/rpc/OzoneKMSUtil.java | 3 +
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 15 +-
.../hadoop/ozone/client/rpc/TestOzoneKMSUtil.java | 51 +
hadoop-ozone/common/pom.xml | 24 +
hadoop-ozone/common/src/main/bin/ozone | 5 +
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 6 +
.../apache/hadoop/ozone/om/OMMetadataManager.java | 6 +-
.../hadoop/ozone/om/S3SecretManagerImpl.java | 21 +-
.../hadoop/ozone/om/codec/S3SecretValueCodec.java | 57 +
.../hadoop/ozone/om/exceptions/OMException.java | 2 +-
.../ozone/om/ha/OMFailoverProxyProvider.java | 2 +-
.../ozone/om/helpers/OmDeleteVolumeResponse.java | 49 +
.../hadoop/ozone/om/helpers/OmVolumeArgs.java | 19 +-
.../om/helpers/OmVolumeOwnerChangeResponse.java | 56 +
.../hadoop/ozone/om/helpers/S3SecretValue.java | 20 +
.../ozone/om/protocol/OzoneManagerHAProtocol.java | 138 ++-
.../apache/hadoop/ozone/web/utils/OzoneUtils.java | 32 +-
.../src/main/proto/OzoneManagerProtocol.proto | 15 +-
.../ozone/om/codec/TestS3SecretValueCodec.java | 88 ++
hadoop-ozone/dist/Dockerfile | 21 -
.../dist/dev-support/bin/dist-layout-stitching | 5 +-
hadoop-ozone/dist/pom.xml | 108 ++
hadoop-ozone/dist/skaffold.yaml | 25 -
hadoop-ozone/dist/src/main/Dockerfile | 21 +
.../main/compose/ozone-hdfs/docker-compose.yaml | 1 -
.../dist/src/main/compose/ozone-hdfs/docker-config | 2 +-
.../main/compose/ozone-om-ha/docker-compose.yaml | 3 -
.../src/main/compose/ozone-om-ha/docker-config | 2 +-
.../main/compose/ozone-recon/docker-compose.yaml | 1 -
.../src/main/compose/ozone-recon/docker-config | 2 +-
.../src/main/compose/ozone/docker-compose.yaml | 1 -
.../dist/src/main/compose/ozone/docker-config | 2 +-
.../src/main/compose/ozoneblockade/docker-config | 2 +-
.../src/main/compose/ozonefs/docker-compose.yaml | 41 +-
.../dist/src/main/compose/ozonefs/hadoopo3fs.robot | 56 +
.../src/main/compose/ozoneperf/docker-compose.yaml | 1 -
.../src/main/compose/ozones3/docker-compose.yaml | 1 -
.../dist/src/main/compose/ozones3/docker-config | 2 +-
.../dist/src/main/compose/ozonesecure-mr/README.md | 6 +-
.../compose/ozonesecure-mr/docker-compose.yaml | 7 +-
.../src/main/compose/ozonesecure-mr/docker-config | 8 +-
.../main/compose/ozonesecure/docker-compose.yaml | 1 -
.../src/main/compose/ozonesecure/docker-config | 8 +-
.../main/compose/ozonetrace/docker-compose.yaml | 1 -
.../dist/src/main/compose/ozonetrace/docker-config | 2 +-
.../main/k8s/definitions/jaeger/flekszible.yaml | 16 +
.../src/main/k8s/definitions/jaeger/jaeger.yaml | 54 +
.../src/main/k8s/definitions/ozone/config.yaml | 32 +
.../main/k8s/definitions/ozone/datanode-ds.yaml | 43 +
.../definitions/ozone/definitions/profiler.yaml | 27 +
.../definitions/ozone/definitions/prometheus.yaml | 26 +
.../k8s/definitions/ozone/definitions/tracing.yaml | 33 +
.../src/main/k8s/definitions/ozone/flekszible.yaml | 16 +
.../k8s/definitions/ozone/freon/flekszible.yaml | 16 +
.../main/k8s/definitions/ozone/freon/freon.yaml | 40 +
.../main/k8s/definitions/ozone/om-ss-service.yaml | 27 +
.../dist/src/main/k8s/definitions/ozone/om-ss.yaml | 53 +
.../main/k8s/definitions/ozone/s3g-ss-service.yaml | 27 +
.../src/main/k8s/definitions/ozone/s3g-ss.yaml | 38 +
.../main/k8s/definitions/ozone/scm-ss-service.yaml | 27 +
.../src/main/k8s/definitions/ozone/scm-ss.yaml | 46 +
.../definitions/ozone/transformations/config.yaml | 26 +
.../ozone/transformations/emptydir.yaml | 58 +
.../main/k8s/definitions/prometheus/configmap.yaml | 49 +
.../definitions/prometheus/definitions/enable.yaml | 26 +
.../{ => definitions}/prometheus/deployment.yaml | 0
.../k8s/definitions/prometheus/flekszible.yaml | 16 +
.../src/main/k8s/definitions/prometheus/role.yaml | 33 +
.../k8s/definitions/prometheus/rolebinding.yaml | 27 +
.../prometheus/service-account.yaml | 0
.../k8s/{ => definitions}/prometheus/service.yaml | 0
.../src/main/k8s/examples/minikube/LICENSE.header | 15 +
.../k8s/examples/minikube/config-configmap.yaml | 34 +
.../k8s/examples/minikube/datanode-service.yaml | 25 +
.../examples/minikube/datanode-statefulset.yaml | 58 +
.../examples/minikube/flekszible/flekszible.yaml | 45 +
.../examples/minikube/freon/freon-deployment.yaml | 46 +
.../k8s/examples/minikube/om-public-service.yaml | 28 +
.../src/main/k8s/examples/minikube/om-service.yaml | 28 +
.../main/k8s/examples/minikube/om-statefulset.yaml | 67 ++
.../k8s/examples/minikube/s3g-public-service.yaml | 28 +
.../main/k8s/examples/minikube/s3g-service.yaml | 28 +
.../k8s/examples/minikube/s3g-statefulset.yaml | 51 +
.../k8s/examples/minikube/scm-public-service.yaml | 28 +
.../main/k8s/examples/minikube/scm-service.yaml | 28 +
.../k8s/examples/minikube/scm-statefulset.yaml | 67 ++
.../src/main/k8s/examples/ozone-dev/LICENSE.header | 15 +
.../k8s/examples/ozone-dev/config-configmap.yaml | 37 +
.../k8s/examples/ozone-dev/datanode-daemonset.yaml | 63 +
.../examples/ozone-dev/flekszible/flekszible.yaml | 36 +
.../examples/ozone-dev/freon/freon-deployment.yaml | 53 +
.../examples/ozone-dev/jaeger-public-service.yaml | 28 +
.../k8s/examples/ozone-dev/jaeger-service.yaml | 28 +
.../k8s/examples/ozone-dev/jaeger-statefulset.yaml | 42 +
.../k8s/examples/ozone-dev/om-public-service.yaml | 28 +
.../main/k8s/examples/ozone-dev/om-service.yaml | 28 +
.../k8s/examples/ozone-dev/om-statefulset.yaml | 79 ++
.../examples/ozone-dev/prometheus-clusterrole.yaml | 42 +
.../examples/ozone-dev/prometheus-deployment.yaml | 48 +
.../prometheus-operator-clusterrolebinding.yaml | 28 +
.../prometheus-operator-serviceaccount.yaml | 20 +
.../k8s/examples/ozone-dev/prometheus-service.yaml | 26 +
.../ozone-dev/prometheusconf-configmap.yaml | 50 +
.../k8s/examples/ozone-dev/s3g-public-service.yaml | 28 +
.../main/k8s/examples/ozone-dev/s3g-service.yaml | 28 +
.../k8s/examples/ozone-dev/s3g-statefulset.yaml | 58 +
.../k8s/examples/ozone-dev/scm-public-service.yaml | 28 +
.../main/k8s/examples/ozone-dev/scm-service.yaml | 28 +
.../k8s/examples/ozone-dev/scm-statefulset.yaml | 74 ++
.../src/main/k8s/examples/ozone/LICENSE.header | 15 +
.../main/k8s/examples/ozone/config-configmap.yaml | 34 +
.../k8s/examples/ozone/datanode-daemonset.yaml | 56 +
.../k8s/examples/ozone/flekszible/flekszible.yaml | 43 +
.../k8s/examples/ozone/freon/freon-deployment.yaml | 46 +
.../src/main/k8s/examples/ozone/om-service.yaml | 28 +
.../main/k8s/examples/ozone/om-statefulset.yaml | 73 ++
.../src/main/k8s/examples/ozone/s3g-service.yaml | 28 +
.../main/k8s/examples/ozone/s3g-statefulset.yaml | 51 +
.../src/main/k8s/examples/ozone/scm-service.yaml | 28 +
.../main/k8s/examples/ozone/scm-statefulset.yaml | 67 ++
.../dist/src/main/k8s/ozone/config-configmap.yaml | 34 -
.../main/k8s/ozone/datanode-public-service.yaml | 29 -
.../dist/src/main/k8s/ozone/datanode-service.yaml | 29 -
.../src/main/k8s/ozone/datanode-statefulset.yaml | 52 -
.../dist/src/main/k8s/ozone/om-public-service.yaml | 29 -
.../dist/src/main/k8s/ozone/om-service.yaml | 29 -
.../dist/src/main/k8s/ozone/om-statefulset.yaml | 72 --
.../src/main/k8s/ozone/s3g-public-service.yaml | 29 -
.../dist/src/main/k8s/ozone/s3g-service.yaml | 29 -
.../dist/src/main/k8s/ozone/s3g-statefulset.yaml | 49 -
.../src/main/k8s/ozone/scm-public-service.yaml | 29 -
.../dist/src/main/k8s/ozone/scm-service.yaml | 29 -
.../dist/src/main/k8s/ozone/scm-statefulset.yaml | 67 --
.../dist/src/main/k8s/prometheus/configmap.yaml | 51 -
.../k8s/prometheus/prometheus-public-serivce.yaml | 28 -
.../dist/src/main/k8s/prometheus/role.yaml | 31 -
.../dist/src/main/k8s/prometheus/rolebindng.yaml | 27 -
.../main/k8s/prometheus/scm-public-service.yaml | 29 -
.../dist/src/main/smoketest/commonlib.robot | 8 +-
.../dist/src/main/smoketest/createbucketenv.robot | 43 +
.../dist/src/main/smoketest/env-compose.robot | 32 +
.../integration-test/src/test/bin/start-chaos.sh | 35 +
.../TestContainerStateManagerIntegration.java | 4 +-
.../hdds/scm/pipeline/TestSCMPipelineManager.java | 69 +-
.../hadoop/hdds/scm/pipeline/TestSCMRestart.java | 2 +-
.../safemode/TestSCMSafeModeWithPipelineRules.java | 202 ++++
.../apache/hadoop/ozone/MiniOzoneChaosCluster.java | 20 +-
.../org/apache/hadoop/ozone/MiniOzoneCluster.java | 10 +-
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 104 +-
.../hadoop/ozone/MiniOzoneHAClusterImpl.java | 4 +
.../hadoop/ozone/MiniOzoneLoadGenerator.java | 28 +-
.../hadoop/ozone/TestMiniChaosOzoneCluster.java | 20 +-
.../apache/hadoop/ozone/TestMiniOzoneCluster.java | 14 +-
.../hadoop/ozone/TestStorageContainerManager.java | 127 +-
.../ozone/client/rpc/TestBlockOutputStream.java | 16 +-
.../rpc/TestBlockOutputStreamWithFailures.java | 27 +-
.../hadoop/ozone/client/rpc/TestCommitWatcher.java | 296 +++++
.../rpc/TestContainerStateMachineFailures.java | 2 -
.../rpc/TestOzoneClientRetriesOnException.java | 46 +-
.../ozone/container/ContainerTestHelper.java | 17 +-
.../common/impl/TestContainerPersistence.java | 2 +-
.../TestCloseContainerByPipeline.java | 17 +
.../container/ozoneimpl/TestOzoneContainer.java | 5 +-
.../ozoneimpl/TestOzoneContainerWithTLS.java | 5 +-
.../ozoneimpl/TestSecureOzoneContainer.java | 5 +-
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 12 +-
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 172 ++-
.../apache/hadoop/ozone/om/TestScmChillMode.java | 348 ------
.../apache/hadoop/ozone/om/TestScmSafeMode.java | 348 ++++++
.../ozone/ozShell/TestOzoneDatanodeShell.java | 26 +-
.../hadoop/ozone/ozShell/TestOzoneShell.java | 74 +-
.../org/apache/hadoop/ozone/scm/TestSCMMXBean.java | 8 +-
.../hadoop/ozone/scm/node/TestSCMNodeMetrics.java | 6 +-
.../src/test/resources/core-site.xml} | 0
.../src/test/resources/hdfs-site.xml} | 0
.../hdfs/server/datanode/ObjectStoreHandler.java | 41 +-
.../web/storage/DistributedStorageHandler.java | 12 +
.../org/apache/hadoop/ozone/om/BucketManager.java | 29 +-
.../apache/hadoop/ozone/om/BucketManagerImpl.java | 92 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 4 +-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 14 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 189 ++-
.../hadoop/ozone/om/S3BucketManagerImpl.java | 47 +-
.../org/apache/hadoop/ozone/om/VolumeManager.java | 52 +-
.../apache/hadoop/ozone/om/VolumeManagerImpl.java | 289 +++--
.../ozone/om/ratis/OzoneManagerRatisServer.java | 28 +-
.../ozone/om/ratis/OzoneManagerStateMachine.java | 183 ++-
.../protocolPB/OzoneManagerHARequestHandler.java | 48 +
.../OzoneManagerHARequestHandlerImpl.java | 368 ++++++
.../protocolPB/OzoneManagerRequestHandler.java | 6 +-
.../hadoop/ozone/protocolPB/RequestHandler.java | 2 +-
.../ozone/web/ozShell/bucket/BucketCommands.java | 2 +-
.../hadoop/ozone/web/ozShell/keys/KeyCommands.java | 2 +-
.../ozone/web/ozShell/keys/ListKeyHandler.java | 7 +
.../ozone/web/ozShell/token/TokenCommands.java | 2 +-
.../ozone/web/ozShell/volume/VolumeCommands.java | 2 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 14 +-
.../om/ratis/TestOzoneManagerStateMachine.java | 16 +-
hadoop-ozone/ozone-recon-codegen/pom.xml | 70 ++
.../ozone/recon/codegen/JooqCodeGenerator.java | 170 +++
.../recon/codegen/ReconSchemaGenerationModule.java | 39 +
.../ozone/recon/codegen/TableNamingStrategy.java | 48 +
.../hadoop/ozone/recon/codegen/package-info.java | 22 +
.../ozone/recon/schema/ReconSchemaDefinition.java | 34 +
.../recon/schema/UtilizationSchemaDefinition.java | 69 ++
.../hadoop/ozone/recon/schema/package-info.java | 22 +
.../dev-support/findbugsExcludeFile.xml | 28 +
hadoop-ozone/ozone-recon/pom.xml | 146 ++-
.../hadoop/ozone/recon/ReconControllerModule.java | 102 +-
.../hadoop/ozone/recon/ReconServerConfigKeys.java | 25 +
.../org/apache/hadoop/ozone/recon/ReconUtils.java | 2 +-
.../ozone/recon/api/ContainerKeyService.java | 23 +-
.../ozone/recon/api/types/ContainerMetadata.java | 33 +-
.../recon/persistence/DataSourceConfiguration.java | 86 ++
.../persistence/DefaultDataSourceProvider.java | 74 ++
.../recon/persistence/JooqPersistenceModule.java | 110 ++
.../TransactionalMethodInterceptor.java | 76 ++
.../ozone/recon/persistence/package-info.java | 22 +
.../recon/spi/ContainerDBServiceProvider.java | 9 +
.../spi/impl/ContainerDBServiceProviderImpl.java | 30 +-
.../recon/WEB-INF}/web.xml | 0
.../ozone/recon/api/TestContainerKeyService.java | 49 +-
.../recon/persistence/AbstractSqlDatabaseTest.java | 152 +++
.../TestUtilizationSchemaDefinition.java | 160 +++
.../ozone/recon/persistence/package-info.java | 22 +
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 361 ++++++
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 858 ++++++++++++++
.../hadoop/fs/ozone/FilteredClassLoader.java | 1 +
.../apache/hadoop/fs/ozone/OzoneClientAdapter.java | 2 +-
.../hadoop/fs/ozone/OzoneClientAdapterFactory.java | 30 +-
.../hadoop/fs/ozone/OzoneClientAdapterImpl.java | 356 +-----
.../apache/hadoop/fs/ozone/OzoneFileSystem.java | 823 +------------
.../hadoop/fs/ozone/TestOzoneFileInterfaces.java | 2 +-
.../services/org.apache.hadoop.fs.FileSystem | 0
hadoop-ozone/pom.xml | 10 +-
hadoop-ozone/s3gateway/pom.xml | 15 +-
.../ozone/genesis/BenchMarkOzoneManager.java | 6 +-
.../apache/hadoop/ozone/genesis/BenchMarkSCM.java | 6 +-
.../hadoop/ozone/audit/parser/TestAuditParser.java | 7 +-
.../hadoop/ozone/freon/TestDataValidate.java | 14 +-
.../TestDataValidateWithSafeByteOperations.java | 52 +
.../TestDataValidateWithUnsafeByteOperations.java | 52 +
hadoop-ozone/upgrade/pom.xml | 54 +
.../org/apache/hadoop/ozone/upgrade/Balance.java | 38 +
.../org/apache/hadoop/ozone/upgrade/Execute.java | 37 +
.../hadoop/ozone/upgrade/InPlaceUpgrade.java | 45 +
.../java/org/apache/hadoop/ozone/upgrade/Plan.java | 38 +
.../apache/hadoop/ozone/upgrade/package-info.java | 23 +
hadoop-project/pom.xml | 16 +-
hadoop-submarine/hadoop-submarine-core/pom.xml | 10 +
.../base/ubuntu-16.04/Dockerfile.cpu.tf_1.13.1 | 71 ++
.../base/ubuntu-16.04/Dockerfile.cpu.tf_1.8.0 | 69 --
.../base/ubuntu-16.04/Dockerfile.gpu.tf_1.13.1 | 85 ++
.../base/ubuntu-16.04/Dockerfile.gpu.tf_1.8.0 | 67 --
.../src/main/docker/build-all.sh | 8 +-
.../ubuntu-16.04/Dockerfile.cpu.tf_1.13.1 | 22 +
.../ubuntu-16.04/Dockerfile.cpu.tf_1.8.0 | 22 -
.../ubuntu-16.04/Dockerfile.gpu.tf_1.13.1 | 22 +
.../ubuntu-16.04/Dockerfile.gpu.tf_1.8.0 | 22 -
.../README.md | 0
.../cifar10_estimator_tf_1.13.1/cifar10.py | 113 ++
.../cifar10_estimator_tf_1.13.1/cifar10_main.py | 521 ++++++++
.../cifar10_model.py | 0
.../cifar10_estimator_tf_1.13.1/cifar10_utils.py | 153 +++
.../generate_cifar10_tfrecords.py | 118 ++
.../model_base.py | 0
.../cifar10_estimator_tf_1.8.0/cifar10.py | 113 --
.../cifar10_estimator_tf_1.8.0/cifar10_main.py | 521 --------
.../cifar10_estimator_tf_1.8.0/cifar10_utils.py | 154 ---
.../generate_cifar10_tfrecords.py | 114 --
.../yarn/submarine/client/cli/CliConstants.java | 3 +
.../yarn/submarine/client/cli/RunJobCli.java | 80 +-
.../yarn/submarine/client/cli/ShowJobCli.java | 7 +-
.../submarine/client/cli/param/BaseParameters.java | 10 +-
.../client/cli/param/ParametersHolder.java | 315 +++++
.../client/cli/param/RunJobParameters.java | 128 +-
.../submarine/client/cli/param/RunParameters.java | 37 +-
.../submarine/client/cli/param/yaml/Configs.java | 107 ++
.../submarine/client/cli/param/yaml/PsRole.java | 25 +
.../yarn/submarine/client/cli/param/yaml/Role.java | 91 ++
.../submarine/client/cli/param/yaml/Roles.java | 41 +
.../client/cli/param/yaml/Scheduling.java | 32 +
.../submarine/client/cli/param/yaml/Security.java | 50 +
.../yarn/submarine/client/cli/param/yaml/Spec.java | 41 +
.../client/cli/param/yaml/TensorBoard.java | 41 +
.../client/cli/param/yaml/WorkerRole.java | 25 +
.../client/cli/param/yaml/YamlConfigFile.java | 77 ++
.../client/cli/param/yaml/YamlParseException.java | 27 +
.../client/cli/param/yaml/package-info.java | 19 +
.../src/site/markdown/InstallationGuide.md | 214 ++--
.../markdown/InstallationGuideChineseVersion.md | 201 ++--
.../src/site/markdown/QuickStart.md | 43 +-
.../markdown/RunningDistributedCifar10TFJobs.md | 12 +-
.../src/site/markdown/TestAndTroubleshooting.md | 27 +-
.../src/site/markdown/WriteDockerfile.md | 34 +-
.../submarine/client/cli/TestRunJobCliParsing.java | 69 +-
.../client/cli/TestRunJobCliParsingYaml.java | 380 ++++++
.../cli/TestRunJobCliParsingYamlStandalone.java | 205 ++++
.../submarine/client/cli/YamlConfigTestUtils.java | 65 +
.../common/fs/MockRemoteDirectoryManager.java | 2 +
.../runjobcliparsing/envs-are-missing.yaml | 59 +
.../runjobcliparsing/missing-configs.yaml | 41 +
.../security-principal-is-missing.yaml | 61 +
.../runjobcliparsing/some-sections-missing.yaml | 48 +
.../tensorboard-dockerimage-is-missing.yaml | 61 +
.../runjobcliparsing/test-false-values.yaml | 56 +
.../valid-config-with-overrides.yaml | 81 ++
.../resources/runjobcliparsing/valid-config.yaml | 62 +
.../runjobcliparsing/wrong-indentation.yaml | 60 +
.../runjobcliparsing/wrong-property-name.yaml | 60 +
.../hadoop-submarine-tony-runtime/README.md | 25 +
.../hadoop-submarine-tony-runtime/pom.xml | 66 ++
.../submarine/runtimes/tony/TonyJobMonitor.java | 52 +
.../submarine/runtimes/tony/TonyJobSubmitter.java | 97 ++
.../runtimes/tony/TonyRuntimeFactory.java | 55 +
.../yarn/submarine/runtimes/tony/TonyUtils.java | 164 +++
.../runtimes/tony/buider/JobStatusBuilder.java | 61 +
.../runtimes/tony/buider/package-info.java | 14 +
.../yarn/submarine/runtimes/tony/package-info.java | 14 +
.../src/site/markdown/QuickStart.md | 297 +++++
.../src/site/resources/css/site.css | 0
.../src/site/site.xml | 0
.../src/test/java/TestTonyUtils.java | 113 ++
.../hadoop-submarine-yarnservice-runtime/pom.xml | 6 +
.../runtimes/yarnservice/AbstractComponent.java | 99 ++
.../runtimes/yarnservice/FileSystemOperations.java | 201 ++++
.../yarnservice/HadoopEnvironmentSetup.java | 161 +++
.../runtimes/yarnservice/ServiceSpec.java | 27 +
.../yarnservice/ServiceSpecFileGenerator.java | 51 +
.../runtimes/yarnservice/ServiceWrapper.java | 62 +
.../yarnservice/YarnServiceJobSubmitter.java | 860 +-------------
.../runtimes/yarnservice/YarnServiceUtils.java | 94 +-
.../yarnservice/command/AbstractLaunchCommand.java | 64 +
.../yarnservice/command/LaunchCommandFactory.java | 67 ++
.../yarnservice/command/LaunchScriptBuilder.java | 107 ++
.../runtimes/yarnservice/command/package-info.java | 19 +
.../yarnservice/tensorflow/TensorFlowCommons.java | 109 ++
.../tensorflow/TensorFlowServiceSpec.java | 203 ++++
.../command/TensorBoardLaunchCommand.java | 67 ++
.../command/TensorFlowLaunchCommand.java | 87 ++
.../command/TensorFlowPsLaunchCommand.java | 58 +
.../command/TensorFlowWorkerLaunchCommand.java | 59 +
.../tensorflow/command/package-info.java | 19 +
.../tensorflow/component/TensorBoardComponent.java | 96 ++
.../component/TensorFlowPsComponent.java | 73 ++
.../component/TensorFlowWorkerComponent.java | 82 ++
.../tensorflow/component/package-info.java | 20 +
.../yarnservice/tensorflow/package-info.java | 20 +
.../yarn/submarine/utils/ClassPathUtilities.java | 57 +
.../yarn/submarine/utils/DockerUtilities.java | 33 +
.../yarn/submarine/utils/EnvironmentUtilities.java | 120 ++
.../submarine/utils/KerberosPrincipalFactory.java | 95 ++
.../hadoop/yarn/submarine/utils/Localizer.java | 170 +++
.../submarine/utils/SubmarineResourceUtils.java | 51 +
.../hadoop/yarn/submarine/utils/ZipUtilities.java | 82 ++
.../hadoop/yarn/submarine/utils/package-info.java | 19 +
.../yarn/submarine/FileUtilitiesForTests.java | 146 +++
.../cli/yarnservice/ParamBuilderForTest.java | 139 +++
.../cli/yarnservice/TestYarnServiceRunJobCli.java | 1242 +++++---------------
.../TestYarnServiceRunJobCliCommons.java | 79 ++
.../TestYarnServiceRunJobCliLocalization.java | 599 ++++++++++
.../runtimes/yarnservice/TestServiceWrapper.java | 95 ++
.../yarnservice/TestTFConfigGenerator.java | 10 +-
.../command/AbstractLaunchCommandTestHelper.java | 190 +++
.../command/TestLaunchCommandFactory.java | 97 ++
.../command/TestTensorBoardLaunchCommand.java | 104 ++
.../command/TestTensorFlowLaunchCommand.java | 251 ++++
.../tensorflow/component/ComponentTestCommons.java | 90 ++
.../component/TestTensorBoardComponent.java | 125 ++
.../component/TestTensorFlowPsComponent.java | 166 +++
.../component/TestTensorFlowWorkerComponent.java | 215 ++++
.../submarine/utils/TestClassPathUtilities.java | 91 ++
.../submarine/utils/TestEnvironmentUtilities.java | 231 ++++
.../utils/TestKerberosPrincipalFactory.java | 156 +++
.../utils/TestSubmarineResourceUtils.java | 72 ++
hadoop-submarine/pom.xml | 1 +
.../java/org/apache/hadoop/fs/s3a/Constants.java | 11 +
.../fs/s3a/MetadataPersistenceException.java | 40 +
.../java/org/apache/hadoop/fs/s3a/Retries.java | 21 +-
.../org/apache/hadoop/fs/s3a/S3AFileSystem.java | 35 +-
.../org/apache/hadoop/fs/s3a/S3AInputStream.java | 39 +-
.../org/apache/hadoop/fs/s3a/S3ARetryPolicy.java | 4 +
.../java/org/apache/hadoop/fs/s3a/S3AUtils.java | 3 +-
.../apache/hadoop/fs/s3a/WriteOperationHelper.java | 17 +-
.../fs/s3a/s3guard/DynamoDBMetadataStore.java | 24 +
.../hadoop/fs/s3a/s3guard/MetadataStore.java | 2 +
.../org/apache/hadoop/fs/s3a/s3guard/S3Guard.java | 2 +
.../src/site/markdown/tools/hadoop-aws/s3guard.md | 182 ++-
.../src/site/markdown/tools/hadoop-aws/testing.md | 6 +
.../fs/contract/s3a/ITestS3AContractUnbuffer.java | 41 +
.../apache/hadoop/fs/s3a/AbstractS3ATestBase.java | 5 +
.../s3a/ITestS3AMetadataPersistenceException.java | 136 +++
.../org/apache/hadoop/fs/s3a/ITestS3AUnbuffer.java | 66 ++
.../org/apache/hadoop/fs/s3a/S3ATestConstants.java | 4 +
.../org/apache/hadoop/fs/s3a/S3ATestUtils.java | 10 +
.../org/apache/hadoop/fs/s3a/TestS3AUnbuffer.java | 76 ++
.../s3a/s3guard/AbstractS3GuardToolTestBase.java | 6 +-
.../hadoop/fs/s3a/s3guard/DDBCapacities.java | 108 ++
.../fs/s3a/s3guard/ITestDynamoDBMetadataStore.java | 24 +-
.../s3guard/ITestDynamoDBMetadataStoreScale.java | 4 +
.../fs/s3a/s3guard/ITestS3GuardConcurrentOps.java | 4 +-
.../fs/s3a/s3guard/ITestS3GuardToolDynamoDB.java | 133 +--
.../hadoop-aws/src/test/resources/contract/s3a.xml | 5 +
.../hadoop-azure/src/config/checkstyle.xml | 5 +-
.../fs/azurebfs/services/AbfsOutputStream.java | 17 +-
.../hadoop-distcp/src/site/markdown/DistCp.md.vm | 54 +-
hadoop-tools/hadoop-fs2img/pom.xml | 1 -
.../hadoop/yarn/sls/nodemanager/NodeInfo.java | 9 +
.../hadoop/yarn/sls/scheduler/RMNodeWrapper.java | 9 +
.../hadoop-yarn/dev-support/findbugs-exclude.xml | 35 +
.../hadoop-yarn/hadoop-yarn-api/pom.xml | 21 +
.../hadoop/yarn/api/records/ResourceOption.java | 8 +-
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 41 +-
.../hadoop/yarn/api/records/TestResource.java | 22 +-
.../apache/hadoop/yarn/api/records/TestURL.java | 8 +-
.../timelineservice/TestApplicationEntity.java | 12 +-
.../timelineservice/TestTimelineMetric.java | 10 +-
.../resource/TestPlacementConstraintParser.java | 268 ++---
.../api/resource/TestPlacementConstraints.java | 55 +-
.../hadoop/yarn/conf/TestResourceInformation.java | 47 +-
.../hadoop/yarn/util/TestUnitsConversionUtil.java | 143 +--
.../pom.xml | 2 +-
.../src/main/scripts/entrypoint.sh | 45 +-
.../src/main/scripts/setup-image.sh | 5 +
.../pom.xml | 5 +-
.../application/AppCatalogInitializer.java | 54 +
.../application/AppCatalogSolrClient.java | 83 +-
.../appcatalog/application/YarnServiceClient.java | 22 +
.../controller/AppDetailsController.java | 34 +
.../src/main/javascript/app.js | 3 +
.../src/main/javascript/controllers.js | 54 +
.../src/main/webapp/WEB-INF/web.xml | 42 +
.../src/main/webapp/css/bootstrap-hadoop.css | 60 +-
.../src/main/webapp/css/specific.css | 42 +-
.../src/main/webapp/index.html | 23 +-
.../src/main/webapp/partials/details.html | 3 +-
.../src/main/webapp/partials/home.html | 6 +-
.../src/main/webapp/partials/upgrade.html | 114 ++
.../src/main/webapp/theme.html | 2 +-
.../application/TestAppCatalogSolrClient.java | 19 +
.../controller/AppDetailsControllerTest.java | 19 +
.../configsets/exampleCollection/conf/schema.xml | 1 +
.../hadoop-yarn-applications-catalog/pom.xml | 2 +-
.../dev-support/findbugs-exclude.xml | 17 +
.../dev-support/findbugs-exclude.xml | 17 +
.../hadoop-yarn-applications-mawo-core/pom.xml | 173 +++
.../src/assembly/bin.xml | 41 +
.../mawo/server/common/AbstractTask.java | 246 ++++
.../mawo/server/common/CompositeTask.java | 42 +
.../applications/mawo/server/common/DieTask.java | 32 +
.../mawo/server/common/MawoConfiguration.java | 502 ++++++++
.../applications/mawo/server/common/NullTask.java | 33 +
.../mawo/server/common/SimpleTask.java | 57 +
.../applications/mawo/server/common/Task.java | 83 ++
.../applications/mawo/server/common/TaskId.java | 149 +++
.../mawo/server/common/TaskStatus.java | 347 ++++++
.../applications/mawo/server/common/TaskType.java | 45 +
.../mawo/server/common/TeardownTask.java | 48 +
.../mawo/server/common/WorkAssignmentProtocol.java | 65 +
.../mawo/server/common/package-info.java | 20 +
.../applications/mawo/server/master/job/JobId.java | 128 ++
.../mawo/server/master/job/package-info.java | 20 +
.../applications/mawo/server/worker/WorkerId.java | 162 +++
.../mawo/server/worker/package-info.java | 20 +
.../src/main/resources/mawo-default.properties | 41 +
.../mawo/server/common/TestMaWoConfiguration.java | 60 +
.../src/test/resources/mawo.properties | 28 +
.../hadoop-yarn-applications-mawo/pom.xml | 37 +
.../hadoop-yarn-services-core/pom.xml | 16 +-
.../service/utils/ApplicationReportSerDeser.java | 12 +-
.../hadoop/yarn/service/utils/JsonSerDeser.java | 34 +-
.../yarn/service/utils/PublishedConfiguration.java | 12 +-
.../service/utils/SerializedApplicationReport.java | 8 +-
.../hadoop/yarn/service/utils/ServiceApiUtil.java | 10 +-
.../hadoop/yarn/service/ServiceTestUtils.java | 7 +-
.../yarn/service/TestYarnNativeServices.java | 3 +-
.../hadoop/yarn/service/client/TestServiceCLI.java | 13 +-
.../yarn/service/utils/TestServiceApiUtil.java | 3 +-
.../hadoop-yarn/hadoop-yarn-applications/pom.xml | 1 +
.../hadoop-yarn/hadoop-yarn-client/pom.xml | 5 +
.../yarn/client/api/impl/AMRMClientImpl.java | 7 +-
.../client/api/async/impl/TestAMRMClientAsync.java | 3 +-
.../hadoop/yarn/client/api/impl/TestAHSClient.java | 3 +-
.../yarn/client/api/impl/TestAHSv2ClientImpl.java | 34 +-
.../yarn/client/api/impl/TestAMRMClient.java | 3 +-
.../api/impl/TestAMRMClientContainerRequest.java | 4 +-
.../TestOpportunisticContainerAllocationE2E.java | 3 +-
.../yarn/client/api/impl/TestYarnClient.java | 39 +-
.../api/impl/TestYarnClientWithReservation.java | 10 +-
.../apache/hadoop/yarn/client/cli/TestYarnCLI.java | 13 +-
.../hadoop-yarn/hadoop-yarn-common/pom.xml | 5 +
.../AggregatedLogDeletionService.java | 16 +-
.../yarn/logaggregation/LogAggregationUtils.java | 184 ++-
.../LogAggregationFileController.java | 49 +-
.../LogAggregationFileControllerFactory.java | 37 +-
.../ifile/LogAggregationIndexedFileController.java | 7 +
.../server/metrics/ContainerMetricsConstants.java | 9 +
.../src/main/resources/yarn-default.xml | 31 +
.../apache/hadoop/yarn/api/TestResourcePBImpl.java | 3 +-
.../TestAggregatedLogDeletionService.java | 163 ++-
.../logaggregation/TestContainerLogsUtils.java | 5 +-
.../TestLogAggregationIndexFileController.java | 7 +-
.../java/org/apache/hadoop/yarn/util/TestApps.java | 3 +-
.../hadoop/yarn/util/TestConverterUtils.java | 9 +-
.../yarn/util/TestTimelineServiceHelper.java | 4 +-
.../yarn/util/resource/TestResourceUtils.java | 28 +-
.../hadoop-yarn/hadoop-yarn-csi/pom.xml | 5 +
.../yarn/csi/adaptor/TestCsiAdaptorService.java | 13 +-
.../pom.xml | 5 +
.../webapp/TestAHSWebServices.java | 19 +-
.../hadoop-yarn-server-common/pom.xml | 5 +
.../metrics/OpportunisticSchedulerMetrics.java | 141 +++
.../scheduler/OpportunisticContainerAllocator.java | 122 +-
.../policies/manager/BasePolicyManagerTest.java | 9 +-
.../TestOpportunisticContainerAllocator.java | 230 +++-
.../hadoop-yarn-server-nodemanager/pom.xml | 5 +
.../src/CMakeLists.txt | 7 +-
.../yarn/server/nodemanager/NodeManager.java | 8 +-
.../application/ApplicationImpl.java | 2 +-
.../containermanager/container/ContainerImpl.java | 2 +-
.../launcher/ContainerCleanup.java | 6 +-
.../launcher/ContainerRelaunch.java | 9 +-
.../linux/runtime/DockerLinuxContainerRuntime.java | 29 +
.../localizer/LocalizedResource.java | 2 +-
.../resourceplugin/com/nec/NECVEPlugin.java | 310 +++++
.../resourceplugin/com/nec/package-info.java | 19 +
.../timelineservice/NMTimelinePublisher.java | 102 +-
.../container-executor/impl/container-executor.c | 9 +-
.../container-executor/impl/utils/docker-util.c | 26 +-
.../test/utils/test_docker_util.cc | 4 +-
.../TestLinuxContainerExecutorWithMocks.java | 46 +-
.../TestContainerManagerRecovery.java | 13 +-
.../launcher/TestContainerCleanup.java | 13 +
.../launcher/TestContainerLaunch.java | 21 +-
.../linux/resources/TestResourceHandlerModule.java | 6 +-
.../resources/fpga/TestFpgaResourceHandler.java | 7 +-
.../linux/runtime/TestDockerContainerRuntime.java | 50 +
.../localizer/TestResourceLocalizationService.java | 9 +-
.../logaggregation/TestLogAggregationService.java | 43 +-
.../resourceplugin/com/nec/TestNECVEPlugin.java | 413 +++++++
.../nvidia/TestNvidiaGPUPluginForRuntimeV2.java | 31 +-
.../recovery/TestNMLeveldbStateStoreService.java | 32 +-
.../timelineservice/TestNMTimelinePublisher.java | 137 +++
.../nodemanager/webapp/TestNMWebServices.java | 225 ++--
.../hadoop-yarn-server-resourcemanager/pom.xml | 5 +
.../OpportunisticContainerAllocatorAMService.java | 11 +-
.../yarn/server/resourcemanager/RMAuditLogger.java | 5 -
.../resourcemanager/ResourceTrackerService.java | 5 +
.../metrics/TimelineServiceV2Publisher.java | 13 +-
.../rmapp/attempt/RMAppAttemptImpl.java | 3 +-
.../rmcontainer/RMContainerImpl.java | 5 +-
.../yarn/server/resourcemanager/rmnode/RMNode.java | 11 +
.../server/resourcemanager/rmnode/RMNodeImpl.java | 22 +-
.../scheduler/AbstractYarnScheduler.java | 79 +-
.../resourcemanager/scheduler/SchedulerNode.java | 66 ++
.../scheduler/activities/ActivitiesLogger.java | 57 +-
.../scheduler/activities/ActivitiesManager.java | 149 ++-
.../scheduler/activities/AppAllocation.java | 2 +-
.../scheduler/capacity/CSQueue.java | 9 +
.../scheduler/capacity/CSQueueUtils.java | 5 +-
.../scheduler/capacity/CapacityScheduler.java | 15 +-
.../scheduler/capacity/LeafQueue.java | 8 +-
.../scheduler/capacity/ParentQueue.java | 7 +
.../scheduler/fair/FairScheduler.java | 27 +-
.../resourcemanager/webapp/RMWebServices.java | 1 +
.../ApplicationMasterServiceTestBase.java | 4 +
.../yarn/server/resourcemanager/MockNodes.java | 9 +
.../resourcemanager/TestClientRMService.java | 23 +-
.../TestKillApplicationWithRMHA.java | 3 +-
...stOpportunisticContainerAllocatorAMService.java | 152 ++-
.../server/resourcemanager/TestRMAdminService.java | 3 +-
.../yarn/server/resourcemanager/TestRMHA.java | 5 +-
.../yarn/server/resourcemanager/TestRMRestart.java | 5 +-
.../server/resourcemanager/TestRMServerUtils.java | 9 +-
.../TestWorkPreservingRMRestart.java | 16 +-
.../applicationsmanager/TestAMRestart.java | 148 ++-
.../metrics/TestSystemMetricsPublisher.java | 3 +-
.../metrics/TestSystemMetricsPublisherForV2.java | 47 +
...alCapacityPreemptionPolicyPreemptToBalance.java | 4 +-
.../TestFileSystemNodeAttributeStore.java | 13 +-
.../nodelabels/TestRMNodeLabelsManager.java | 76 +-
.../recovery/TestFSRMStateStore.java | 5 +-
.../recovery/TestZKRMStateStore.java | 7 +-
.../reservation/TestReservationSystemUtil.java | 9 +-
.../reservation/planning/TestAlignedPlanner.java | 33 +-
.../rmapp/TestRMAppTransitions.java | 4 +-
.../rmcontainer/TestRMContainerImpl.java | 6 +-
.../scheduler/TestAbstractYarnScheduler.java | 93 ++
.../scheduler/TestSchedulerOvercommit.java | 735 ++++++++++++
.../activities/TestActivitiesManager.java | 272 +++++
.../scheduler/capacity/TestApplicationLimits.java | 26 +-
.../capacity/TestApplicationPriority.java | 8 +-
.../scheduler/capacity/TestCapacityScheduler.java | 281 +++--
.../capacity/TestCapacitySchedulerOvercommit.java | 52 +
.../TestQueueManagementDynamicEditPolicy.java | 2 +-
.../scheduler/fair/TestConfigurableResource.java | 13 +-
.../scheduler/fair/TestContinuousScheduling.java | 8 +-
.../scheduler/fair/TestFSLeafQueue.java | 10 +-
.../scheduler/fair/TestFairScheduler.java | 32 +-
.../fair/TestFairSchedulerOvercommit.java | 46 +
.../scheduler/fifo/TestFifoScheduler.java | 15 +-
.../scheduler/policy/TestFifoOrderingPolicy.java | 10 +-
.../TestFifoOrderingPolicyForPendingApps.java | 8 +-
.../volume/csi/TestVolumeProcessor.java | 9 +-
.../webapp/TestRMWebServicesNodes.java | 5 +-
.../webapp/TestRMWebServicesReservation.java | 3 +-
.../TestRMWebServicesSchedulerActivities.java | 16 +-
...esSchedulerActivitiesWithMultiNodesEnabled.java | 293 +++++
.../hadoop-metrics2-resourcemanager.properties | 23 +
.../src/test/resources/hadoop-metrics2.properties | 23 +
.../clientrm/FederationClientInterceptor.java | 2 +-
.../pom.xml | 5 +
.../timeline/TestEntityGroupFSTimelineStore.java | 3 +-
.../document/entity/TimelineEntityDocument.java | 22 +-
.../document/flowrun/FlowRunDocument.java | 13 +-
.../cosmosdb/CosmosDBDocumentStoreReader.java | 2 +-
.../cosmosdb/CosmosDBDocumentStoreWriter.java | 2 +-
.../pom.xml | 6 +
.../flow/TestHBaseStorageFlowRunCompaction.java | 4 +-
.../storage/apptoflow/AppToFlowTableRW.java | 8 +-
.../hadoop-yarn-server-timelineservice/pom.xml | 6 +
.../collector/TimelineCollector.java | 24 +-
.../collector/TestTimelineCollector.java | 48 +-
.../hadoop-yarn-server-web-proxy/pom.xml | 5 +
.../server/webproxy/TestWebAppProxyServlet.java | 3 +-
.../server/webproxy/amfilter/TestAmFilter.java | 3 +-
.../webproxy/amfilter/TestSecureAmFilter.java | 4 +-
.../src/site/markdown/TimelineServiceV2.md | 8 -
.../src/site/markdown/yarn-service/Examples.md | 64 +
.../src/site/markdown/yarn-service/MasterWorker.md | 36 +
pom.xml | 2 +-
start-build-env.sh | 1 +
922 files changed, 38425 insertions(+), 14073 deletions(-)
create mode 100644
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferPositionedReadable.java
create mode 100644
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractUnbufferTest.java
create mode 100644
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/CommitWatcher.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ByteStringHelper.java
create mode 100644
hadoop-hdds/common/src/main/resources/network-topology-default.yaml
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/conf/SimpleConfiguration.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestYamlSchemaLoader.java
create mode 100644
hadoop-hdds/common/src/test/resources/networkTopologyTestFiles/good.yaml
create mode 100644
hadoop-hdds/common/src/test/resources/networkTopologyTestFiles/middle-leaf.yaml
create mode 100644
hadoop-hdds/common/src/test/resources/networkTopologyTestFiles/multiple-root.yaml
create mode 100644 hadoop-hdds/config/pom.xml
create mode 100644
hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/Config.java
create mode 100644
hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigFileAppender.java
create mode 100644
hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigFileGenerator.java
create mode 100644
hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigGroup.java
create mode 100644
hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigTag.java
create mode 100644
hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigType.java
create mode 100644
hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigurationException.java
create mode 100644
hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/package-info.java
create mode 100644
hadoop-hdds/config/src/main/resources/META-INF/services/javax.annotation.processing.Processor
create mode 100644
hadoop-hdds/config/src/test/java/org/apache/hadoop/hdds/conf/ConfigurationExample.java
create mode 100644
hadoop-hdds/config/src/test/java/org/apache/hadoop/hdds/conf/TestConfigFileAppender.java
create mode 100644
hadoop-hdds/config/src/test/java/org/apache/hadoop/hdds/conf/package-info.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeStopService.java
create mode 100644 hadoop-hdds/docs/content/OzoneSecurityArchitecture.md
delete mode 100644 hadoop-hdds/docs/content/S3Commands.md
create mode 100644 hadoop-hdds/docs/content/SetupSecureOzone.md
create mode 100644 hadoop-hdds/docs/static/prometheus-key-allocate.png
create mode 100644 hadoop-hdds/docs/static/prometheus.png
create mode 100644
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/server/TestServerUtils.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/ChillModeExitRule.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/ChillModeHandler.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/ChillModePrecheck.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/ChillModeRestrictedOps.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/ContainerChillModeRule.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/DataNodeChillModeRule.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/HealthyPipelineChillModeRule.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/OneReplicaPipelineChillModeRule.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/Precheck.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/SCMChillModeManager.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/package-info.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/AbstractContainerReportHandler.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/DeleteContainerCommandWatcher.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ReportHandlerHelper.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationCommandWatcher.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManager.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationQueue.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationRequest.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/InsufficientDatanodesException.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/ContainerSafeModeRule.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/DataNodeSafeModeRule.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/HealthyPipelineSafeModeRule.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/OneReplicaPipelineSafeModeRule.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/Precheck.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/SCMSafeModeManager.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/SafeModeExitRule.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/SafeModeHandler.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/SafeModePrecheck.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/SafeModeRestrictedOps.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/package-info.java
delete mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/chillmode/TestChillModeHandler.java
delete mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/chillmode/TestHealthyPipelineChillModeRule.java
delete mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/chillmode/TestOneReplicaPipelineChillModeRule.java
delete mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/chillmode/TestSCMChillModeManager.java
delete mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/chillmode/package-info.java
delete mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerReportHelper.java
delete mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestReplicationActivityStatus.java
delete mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestReplicationManager.java
delete mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestReplicationQueue.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestHealthyPipelineSafeModeRule.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestOneReplicaPipelineSafeModeRule.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestSCMSafeModeManager.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestSafeModeHandler.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/safemode/package-info.java
delete mode 100644
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/ChillModeCheckSubcommand.java
delete mode 100644
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/ChillModeCommands.java
delete mode 100644
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/ChillModeExitSubcommand.java
create mode 100644
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/SafeModeCheckSubcommand.java
create mode 100644
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/SafeModeCommands.java
create mode 100644
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/SafeModeExitSubcommand.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractUnbuffer.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestByteBufferPread.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultipleNNPortQOP.java
delete mode 100644
hadoop-mapreduce-project/.eclipse.templates/.launches/AllMapredTests.launch
delete mode 100644
hadoop-mapreduce-project/.eclipse.templates/.launches/JobTracker.launch
delete mode 100644
hadoop-mapreduce-project/.eclipse.templates/.launches/SpecificTestTemplate.launch
delete mode 100644
hadoop-mapreduce-project/.eclipse.templates/.launches/TaskTracker.launch
delete mode 100644 hadoop-mapreduce-project/.eclipse.templates/README.txt
create mode 100644
hadoop-ozone/client/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneKMSUtil.java
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/codec/S3SecretValueCodec.java
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmDeleteVolumeResponse.java
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmVolumeOwnerChangeResponse.java
create mode 100644
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/codec/TestS3SecretValueCodec.java
delete mode 100644 hadoop-ozone/dist/Dockerfile
delete mode 100644 hadoop-ozone/dist/skaffold.yaml
create mode 100644 hadoop-ozone/dist/src/main/Dockerfile
create mode 100644 hadoop-ozone/dist/src/main/compose/ozonefs/hadoopo3fs.robot
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/jaeger/flekszible.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/jaeger/jaeger.yaml
create mode 100644 hadoop-ozone/dist/src/main/k8s/definitions/ozone/config.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/ozone/datanode-ds.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/ozone/definitions/profiler.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/ozone/definitions/prometheus.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/ozone/definitions/tracing.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/ozone/flekszible.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/ozone/freon/flekszible.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/ozone/freon/freon.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/ozone/om-ss-service.yaml
create mode 100644 hadoop-ozone/dist/src/main/k8s/definitions/ozone/om-ss.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/ozone/s3g-ss-service.yaml
create mode 100644 hadoop-ozone/dist/src/main/k8s/definitions/ozone/s3g-ss.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/ozone/scm-ss-service.yaml
create mode 100644 hadoop-ozone/dist/src/main/k8s/definitions/ozone/scm-ss.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/ozone/transformations/config.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/ozone/transformations/emptydir.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/prometheus/configmap.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/prometheus/definitions/enable.yaml
rename hadoop-ozone/dist/src/main/k8s/{ =>
definitions}/prometheus/deployment.yaml (100%)
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/prometheus/flekszible.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/prometheus/role.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/definitions/prometheus/rolebinding.yaml
rename hadoop-ozone/dist/src/main/k8s/{ =>
definitions}/prometheus/service-account.yaml (100%)
rename hadoop-ozone/dist/src/main/k8s/{ =>
definitions}/prometheus/service.yaml (100%)
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/minikube/LICENSE.header
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/minikube/config-configmap.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/minikube/datanode-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/minikube/datanode-statefulset.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/minikube/flekszible/flekszible.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/minikube/freon/freon-deployment.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/minikube/om-public-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/minikube/om-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/minikube/om-statefulset.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/minikube/s3g-public-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/minikube/s3g-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/minikube/s3g-statefulset.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/minikube/scm-public-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/minikube/scm-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/minikube/scm-statefulset.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/LICENSE.header
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/config-configmap.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/datanode-daemonset.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/flekszible/flekszible.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/freon/freon-deployment.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/jaeger-public-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/jaeger-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/jaeger-statefulset.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/om-public-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/om-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/om-statefulset.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/prometheus-clusterrole.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/prometheus-deployment.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/prometheus-operator-clusterrolebinding.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/prometheus-operator-serviceaccount.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/prometheus-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/prometheusconf-configmap.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/s3g-public-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/s3g-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/s3g-statefulset.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/scm-public-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/scm-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/scm-statefulset.yaml
create mode 100644 hadoop-ozone/dist/src/main/k8s/examples/ozone/LICENSE.header
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone/config-configmap.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone/datanode-daemonset.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone/flekszible/flekszible.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone/freon/freon-deployment.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone/om-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone/om-statefulset.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone/s3g-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone/s3g-statefulset.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone/scm-service.yaml
create mode 100644
hadoop-ozone/dist/src/main/k8s/examples/ozone/scm-statefulset.yaml
delete mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/config-configmap.yaml
delete mode 100644
hadoop-ozone/dist/src/main/k8s/ozone/datanode-public-service.yaml
delete mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/datanode-service.yaml
delete mode 100644
hadoop-ozone/dist/src/main/k8s/ozone/datanode-statefulset.yaml
delete mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/om-public-service.yaml
delete mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/om-service.yaml
delete mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/om-statefulset.yaml
delete mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/s3g-public-service.yaml
delete mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/s3g-service.yaml
delete mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/s3g-statefulset.yaml
delete mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/scm-public-service.yaml
delete mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/scm-service.yaml
delete mode 100644 hadoop-ozone/dist/src/main/k8s/ozone/scm-statefulset.yaml
delete mode 100644 hadoop-ozone/dist/src/main/k8s/prometheus/configmap.yaml
delete mode 100644
hadoop-ozone/dist/src/main/k8s/prometheus/prometheus-public-serivce.yaml
delete mode 100644 hadoop-ozone/dist/src/main/k8s/prometheus/role.yaml
delete mode 100644 hadoop-ozone/dist/src/main/k8s/prometheus/rolebindng.yaml
delete mode 100644
hadoop-ozone/dist/src/main/k8s/prometheus/scm-public-service.yaml
create mode 100644 hadoop-ozone/dist/src/main/smoketest/createbucketenv.robot
create mode 100644 hadoop-ozone/dist/src/main/smoketest/env-compose.robot
create mode 100755 hadoop-ozone/integration-test/src/test/bin/start-chaos.sh
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestSCMSafeModeWithPipelineRules.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
delete mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestScmChillMode.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestScmSafeMode.java
copy hadoop-ozone/{dist/src/main/conf/ozone-site.xml =>
integration-test/src/test/resources/core-site.xml} (100%)
copy hadoop-ozone/{dist/src/main/conf/ozone-site.xml =>
integration-test/src/test/resources/hdfs-site.xml} (100%)
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerHARequestHandler.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerHARequestHandlerImpl.java
create mode 100644 hadoop-ozone/ozone-recon-codegen/pom.xml
create mode 100644
hadoop-ozone/ozone-recon-codegen/src/main/java/org/hadoop/ozone/recon/codegen/JooqCodeGenerator.java
create mode 100644
hadoop-ozone/ozone-recon-codegen/src/main/java/org/hadoop/ozone/recon/codegen/ReconSchemaGenerationModule.java
create mode 100644
hadoop-ozone/ozone-recon-codegen/src/main/java/org/hadoop/ozone/recon/codegen/TableNamingStrategy.java
create mode 100644
hadoop-ozone/ozone-recon-codegen/src/main/java/org/hadoop/ozone/recon/codegen/package-info.java
create mode 100644
hadoop-ozone/ozone-recon-codegen/src/main/java/org/hadoop/ozone/recon/schema/ReconSchemaDefinition.java
create mode 100644
hadoop-ozone/ozone-recon-codegen/src/main/java/org/hadoop/ozone/recon/schema/UtilizationSchemaDefinition.java
create mode 100644
hadoop-ozone/ozone-recon-codegen/src/main/java/org/hadoop/ozone/recon/schema/package-info.java
create mode 100644 hadoop-ozone/ozone-recon/dev-support/findbugsExcludeFile.xml
create mode 100644
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/persistence/DataSourceConfiguration.java
create mode 100644
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/persistence/DefaultDataSourceProvider.java
create mode 100644
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/persistence/JooqPersistenceModule.java
create mode 100644
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/persistence/TransactionalMethodInterceptor.java
create mode 100644
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/persistence/package-info.java
rename hadoop-ozone/ozone-recon/src/main/resources/{webapps.recon.WEB-INF =>
webapps/recon/WEB-INF}/web.xml (100%)
create mode 100644
hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/persistence/AbstractSqlDatabaseTest.java
create mode 100644
hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/persistence/TestUtilizationSchemaDefinition.java
create mode 100644
hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/persistence/package-info.java
create mode 100644
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneClientAdapterImpl.java
create mode 100644
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
rename hadoop-ozone/ozonefs/src/{main =>
test}/resources/META-INF/services/org.apache.hadoop.fs.FileSystem (100%)
create mode 100644
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/freon/TestDataValidateWithSafeByteOperations.java
create mode 100644
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/freon/TestDataValidateWithUnsafeByteOperations.java
create mode 100644 hadoop-ozone/upgrade/pom.xml
create mode 100644
hadoop-ozone/upgrade/src/main/java/org/apache/hadoop/ozone/upgrade/Balance.java
create mode 100644
hadoop-ozone/upgrade/src/main/java/org/apache/hadoop/ozone/upgrade/Execute.java
create mode 100644
hadoop-ozone/upgrade/src/main/java/org/apache/hadoop/ozone/upgrade/InPlaceUpgrade.java
create mode 100644
hadoop-ozone/upgrade/src/main/java/org/apache/hadoop/ozone/upgrade/Plan.java
create mode 100644
hadoop-ozone/upgrade/src/main/java/org/apache/hadoop/ozone/upgrade/package-info.java
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/docker/base/ubuntu-16.04/Dockerfile.cpu.tf_1.13.1
delete mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/docker/base/ubuntu-16.04/Dockerfile.cpu.tf_1.8.0
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/docker/base/ubuntu-16.04/Dockerfile.gpu.tf_1.13.1
delete mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/docker/base/ubuntu-16.04/Dockerfile.gpu.tf_1.8.0
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/Dockerfile.cpu.tf_1.13.1
delete mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/Dockerfile.cpu.tf_1.8.0
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/Dockerfile.gpu.tf_1.13.1
delete mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/Dockerfile.gpu.tf_1.8.0
rename
hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/{cifar10_estimator_tf_1.8.0
=> cifar10_estimator_tf_1.13.1}/README.md (100%)
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.13.1/cifar10.py
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.13.1/cifar10_main.py
rename
hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/{cifar10_estimator_tf_1.8.0
=> cifar10_estimator_tf_1.13.1}/cifar10_model.py (100%)
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.13.1/cifar10_utils.py
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.13.1/generate_cifar10_tfrecords.py
rename
hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/{cifar10_estimator_tf_1.8.0
=> cifar10_estimator_tf_1.13.1}/model_base.py (100%)
delete mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.8.0/cifar10.py
delete mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.8.0/cifar10_main.py
delete mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.8.0/cifar10_utils.py
delete mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/docker/with-cifar10-models/ubuntu-16.04/cifar10_estimator_tf_1.8.0/generate_cifar10_tfrecords.py
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/ParametersHolder.java
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/Configs.java
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/PsRole.java
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/Role.java
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/Roles.java
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/Scheduling.java
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/Security.java
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/Spec.java
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/TensorBoard.java
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/WorkerRole.java
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/YamlConfigFile.java
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/YamlParseException.java
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/main/java/org/apache/hadoop/yarn/submarine/client/cli/param/yaml/package-info.java
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/TestRunJobCliParsingYaml.java
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/TestRunJobCliParsingYamlStandalone.java
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/YamlConfigTestUtils.java
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/envs-are-missing.yaml
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/missing-configs.yaml
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/security-principal-is-missing.yaml
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/some-sections-missing.yaml
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/tensorboard-dockerimage-is-missing.yaml
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/test-false-values.yaml
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/valid-config-with-overrides.yaml
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/valid-config.yaml
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/wrong-indentation.yaml
create mode 100644
hadoop-submarine/hadoop-submarine-core/src/test/resources/runjobcliparsing/wrong-property-name.yaml
create mode 100644 hadoop-submarine/hadoop-submarine-tony-runtime/README.md
create mode 100644 hadoop-submarine/hadoop-submarine-tony-runtime/pom.xml
create mode 100644
hadoop-submarine/hadoop-submarine-tony-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/tony/TonyJobMonitor.java
create mode 100644
hadoop-submarine/hadoop-submarine-tony-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/tony/TonyJobSubmitter.java
create mode 100644
hadoop-submarine/hadoop-submarine-tony-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/tony/TonyRuntimeFactory.java
create mode 100644
hadoop-submarine/hadoop-submarine-tony-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/tony/TonyUtils.java
create mode 100644
hadoop-submarine/hadoop-submarine-tony-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/tony/buider/JobStatusBuilder.java
create mode 100644
hadoop-submarine/hadoop-submarine-tony-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/tony/buider/package-info.java
create mode 100644
hadoop-submarine/hadoop-submarine-tony-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/tony/package-info.java
create mode 100644
hadoop-submarine/hadoop-submarine-tony-runtime/src/site/markdown/QuickStart.md
copy {hadoop-tools/hadoop-resourceestimator =>
hadoop-submarine/hadoop-submarine-tony-runtime}/src/site/resources/css/site.css
(100%)
copy {hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site =>
hadoop-submarine/hadoop-submarine-tony-runtime}/src/site/site.xml (100%)
create mode 100644
hadoop-submarine/hadoop-submarine-tony-runtime/src/test/java/TestTonyUtils.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/AbstractComponent.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/FileSystemOperations.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/HadoopEnvironmentSetup.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/ServiceSpec.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/ServiceSpecFileGenerator.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/ServiceWrapper.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/command/AbstractLaunchCommand.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/command/LaunchCommandFactory.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/command/LaunchScriptBuilder.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/command/package-info.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/TensorFlowCommons.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/TensorFlowServiceSpec.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/command/TensorBoardLaunchCommand.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/command/TensorFlowLaunchCommand.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/command/TensorFlowPsLaunchCommand.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/command/TensorFlowWorkerLaunchCommand.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/command/package-info.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/component/TensorBoardComponent.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/component/TensorFlowPsComponent.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/component/TensorFlowWorkerComponent.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/component/package-info.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/package-info.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/utils/ClassPathUtilities.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/utils/DockerUtilities.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/utils/EnvironmentUtilities.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/utils/KerberosPrincipalFactory.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/utils/Localizer.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/utils/SubmarineResourceUtils.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/utils/ZipUtilities.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/main/java/org/apache/hadoop/yarn/submarine/utils/package-info.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/FileUtilitiesForTests.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/yarnservice/ParamBuilderForTest.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/yarnservice/TestYarnServiceRunJobCliCommons.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/client/cli/yarnservice/TestYarnServiceRunJobCliLocalization.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/TestServiceWrapper.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/command/AbstractLaunchCommandTestHelper.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/command/TestLaunchCommandFactory.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/command/TestTensorBoardLaunchCommand.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/command/TestTensorFlowLaunchCommand.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/component/ComponentTestCommons.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/component/TestTensorBoardComponent.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/component/TestTensorFlowPsComponent.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/tensorflow/component/TestTensorFlowWorkerComponent.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/utils/TestClassPathUtilities.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/utils/TestEnvironmentUtilities.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/utils/TestKerberosPrincipalFactory.java
create mode 100644
hadoop-submarine/hadoop-submarine-yarnservice-runtime/src/test/java/org/apache/hadoop/yarn/submarine/utils/TestSubmarineResourceUtils.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/MetadataPersistenceException.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractUnbuffer.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AMetadataPersistenceException.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AUnbuffer.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AUnbuffer.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/DDBCapacities.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/main/java/org/apache/hadoop/yarn/appcatalog/application/AppCatalogInitializer.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/main/webapp/partials/upgrade.html
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/dev-support/findbugs-exclude.xml
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/dev-support/findbugs-exclude.xml
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/pom.xml
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/assembly/bin.xml
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/java/org/apache/hadoop/applications/mawo/server/common/AbstractTask.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/java/org/apache/hadoop/applications/mawo/server/common/CompositeTask.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/java/org/apache/hadoop/applications/mawo/server/common/DieTask.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/java/org/apache/hadoop/applications/mawo/server/common/MawoConfiguration.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/java/org/apache/hadoop/applications/mawo/server/common/NullTask.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/java/org/apache/hadoop/applications/mawo/server/common/SimpleTask.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/java/org/apache/hadoop/applications/mawo/server/common/Task.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/java/org/apache/hadoop/applications/mawo/server/common/TaskId.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/java/org/apache/hadoop/applications/mawo/server/common/TaskStatus.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/java/org/apache/hadoop/applications/mawo/server/common/TaskType.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/java/org/apache/hadoop/applications/mawo/server/common/TeardownTask.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/java/org/apache/hadoop/applications/mawo/server/common/WorkAssignmentProtocol.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/java/org/apache/hadoop/applications/mawo/server/common/package-info.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/java/org/apache/hadoop/applications/mawo/server/master/job/JobId.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/java/org/apache/hadoop/applications/mawo/server/master/job/package-info.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/java/org/apache/hadoop/applications/mawo/server/worker/WorkerId.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/java/org/apache/hadoop/applications/mawo/server/worker/package-info.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/main/resources/mawo-default.properties
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/test/java/org/apache/hadoop/applications/mawo/server/common/TestMaWoConfiguration.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/src/test/resources/mawo.properties
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/pom.xml
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/OpportunisticSchedulerMetrics.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/com/nec/NECVEPlugin.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/com/nec/package-info.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/com/nec/TestNECVEPlugin.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerOvercommit.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/activities/TestActivitiesManager.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerOvercommit.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerOvercommit.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesSchedulerActivitiesWithMultiNodesEnabled.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/hadoop-metrics2-resourcemanager.properties
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/hadoop-metrics2.properties
create mode 100644
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/MasterWorker.md
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
