Repository: hadoop Updated Branches: refs/heads/trunk ff0ee84d7 -> 1bc454615
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1bc45461/hadoop-yarn-project/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt deleted file mode 100644 index 4934e89..0000000 --- a/hadoop-yarn-project/CHANGES.txt +++ /dev/null @@ -1,6905 +0,0 @@ -Hadoop YARN Change Log - -Trunk - Unreleased - - INCOMPATIBLE CHANGES - - NEW FEATURES - - YARN-2882. Add an OPPORTUNISTIC ExecutionType. - (Konstantinos Karanasos and Inigo Goiri via kasha) - - IMPROVEMENTS - - YARN-2438. yarn-env.sh cleanup (aw) - - YARN-2472. yarn-daemons.sh should jsut call yarn directly (Masatake Iwasaki - via aw) - - YARN-2437. start-yarn.sh/stop-yarn should give info (Varun Saxena via aw) - - YARN-2796. deprecate sbin/yarn-daemon.sh (aw) - - YARN-2980. Move health check script related functionality to hadoop-common - (Varun Saxena via aw) - - YARN-3168. Convert site documentation from apt to markdown (Gururaj Shetty - via aw) - - YARN-3199. Fair Scheduler documentation improvements (Rohit Agarwal via - aw) - - YARN-2280. Resource manager web service fields are not accessible - (Krisztian Horvath via aw) - - YARN-3261. rewrite resourcemanager restart doc to remove roadmap bits (Gururaj Shetty via aw) - - - OPTIMIZATIONS - - BUG FIXES - - YARN-4680. TimerTasks leak in ATS V1.5 Writer. (Xuan Gong via gtcarrera9) - - YARN-524 TestYarnVersionInfo failing if generated properties doesn't - include an SVN URL. (stevel) - - YARN-1471. The SLS simulator is not running the preemption policy - for CapacityScheduler (Carlo Curino via cdouglas) - - YARN-2436. [post-HADOOP-9902] yarn application help doesn't work (aw) - - YARN-2525. yarn logs command gives error on trunk (Akira AJISAKA via aw) - - YARN-3002. YARN documentation needs updating post-shell rewrite (aw) - - YARN-2428. LCE default banned user list should have yarn (Varun - Saxena via aw) - - YARN-2355. MAX_APP_ATTEMPTS_ENV may no longer be a useful env var - for a container (Darrell Taylor via aw) - - YARN-3915. scmadmin help message correction (Bibin A Chundatt via aw) - -Release 2.9.0 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - YARN-1856. Added cgroups based memory monitoring for containers as another - alternative to custom memory-monitoring. (Varun Vasudev via vinodkv) - - YARN-4462. FairScheduler: Disallow preemption from a queue. (Tao Jie via kasha) - - IMPROVEMENTS - - YARN-4072. ApplicationHistoryServer, WebAppProxyServer, NodeManager and - ResourceManager to support JvmPauseMonitor as a service. - (Sunil G via Stevel) - - YARN-4341. add doc about timeline performance tool usage (Chang Li via - sjlee) - - YARN-4417. Make RM and Timeline-server REST APIs more consistent. - (wtan via jianhe) - - YARN-3480. Remove attempts that are beyond max-attempt limit from state - store. (Jun Gong via jianhe) - - YARN-4522. Queue acl can be checked at app submission. (Jian He via wangda) - - YARN-4438. Implement RM leader election with curator. (Jian He via xgong) - - YARN-4571. Make app id/name available to the yarn authorizer provider for - better auditing. (Jian He via wangda) - - YARN-4551. Address the duplication between StatusUpdateWhenHealthy and - StatusUpdateWhenUnhealthy transitions. (Sunil G via kasha) - - YARN-4553. Add cgroups support for docker containers. - (Sidharta Seethana via vvasudev) - - YARN-4526. Make SystemClock singleton so AppSchedulingInfo could use it. - (kasha) - - YARN-4559. Make leader elector and zk store share the same curator client. - (Jian He via xgong) - - YARN-4603. FairScheduler should mention user requested queuename in error - message when failed in queue ACL check. (Tao Jie via kasha) - - YARN-4496. Improve HA ResourceManager Failover detection on the client. - (Jian He via xgong) - - YARN-3542. Refactored existing CPU cgroups support to use the newer and - integrated ResourceHandler mechanism, and also deprecated the old - LCEResourceHandler inteface hierarchy. (Varun Vasudev via vinodkv) - - YARN-4647. Make RegisterNodeManagerRequestPBImpl thread-safe. (kasha) - - YARN-4649. Add additional logging to some NM state store operations. - (Sidharta Seethana via vvasudev) - - YARN-4625. Make ApplicationSubmissionContext and - ApplicationSubmissionContextInfo more consistent. - (Xuan Gong via vvasudev) - - YARN-4307. Display blacklisted nodes for AM container in the RM web UI. - (Naganarasimha G R via vvasudev) - - YARN-4628. Display application priority in yarn top. - (Bibin A Chundatt via vvasudev) - - YARN-4655. Log uncaught exceptions/errors in various thread pools in YARN. - (Sidharta Seethana via vvasudev) - - YARN-4689. FairScheduler: Cleanup preemptContainer to be more readable. - (Kai Sasaki via kasha) - - YARN-4708. Missing default mapper type in TimelineServer performance test tool - usage. (Kai Sasaki via ozawa) - - YARN-4648. Move preemption related tests from TestFairScheduler to - TestFairSchedulerPreemption. (Kai Sasaki via ozawa) - - YARN-4697. NM aggregation thread pool is not bound by - limits (haibochen via rkanter) - - YARN-4718. Rename variables in SchedulerNode to reduce ambiguity post - YARN-1011. (Inigo Goiri via kasha) - - OPTIMIZATIONS - - BUG FIXES - - YARN-4109. Exception on RM scheduler page loading with labels. - (Mohammad Shahid Khan via rohithsharmaks) - - YARN-4156. TestAMRestart#testAMBlacklistPreventsRestartOnSameNode - assumes CapacityScheduler. (Anubhav Dhoot via kasha) - - YARN-2934. Improve handling of container's stderr. - (Naganarasimha G R via gera) - - YARN-4530. LocalizedResource trigger a NPE Cause the NodeManager exit - (tangshangwen via rohithsharmaks) - - YARN-4535. Fix checkstyle error in CapacityScheduler.java - (Naganarasimha G R via rohithsharmaks) - - YARN-4393. Fix intermittent test failure for TestResourceLocalizationService - #testFailedDirsResourceRelease (Varun Saxana via rohithsharmaks) - - YARN-4550. Some tests in TestContainerLanch fails on non-english locale - environment. (Takashi Ohnishi via stevel) - - YARN-4567. javadoc failing on java 8. (Steve Loughran via aajisaka) - - YARN-3446. FairScheduler headroom calculation should exclude nodes in the - blacklist. (Zhihai Xu via kasha) - - YARN-4611. Fix scheduler load simulator to support multi-layer network - location. (Ming Ma via xgong) - - YARN-4584. RM startup failure when AM attempts greater than max-attempts. - (Bibin A Chundatt via rohithsharmaks) - - YARN-4578. Directories that are mounted in docker containers need to be more - restrictive/container-specific. (Sidharta Seethana via vvasudev) - - YARN-4497. RM might fail to restart when recovering apps whose attempts are missing. - (Jun Gong via rohithsharmaks) - - YARN-4574. Fix random failure in TestAMRMClientOnRMRestart. - (Takashi Ohnishi via rohithsharmaks) - - YARN-4613. Fix test failure in TestClientRMService#testGetClusterNodes. - (Takashi Ohnishi via rohithsharmaks) - - YARN-4612. Fix rumen and scheduler load simulator handle killed tasks properly. - (Ming Ma via xgong) - - YARN-4573. Fix test failure in TestRMAppTransitions#testAppRunningKill and - testAppKilledKilled. (Takashi Ohnishi via rohithsharmaks) - - YARN-4633. Fix random test failure in TestRMRestart#testRMRestartAfterPreemption - (Bibin A Chundatt via rohithsharmaks) - - YARN-4543. Fix random test failure in TestNodeStatusUpdater.testStopReentrant - (Akihiro Suda via rohithsharmaks) - - YARN-4615. Fix random test failure in TestAbstractYarnScheduler#testResource - RequestRecoveryToTheRightAppAttempt. (Sunil G via rohithsharmaks) - - YARN-4594. container-executor fails to remove directory tree when chmod - required (Colin Patrick McCabe via jlowe) - - YARN-4669. Fix logging statements in resource manager's Application class. - (Seethana Sidharta via vvasudev) - - YARN-4629. Distributed shell breaks under strong security. - (Daniel Templeton via stevel) - - YARN-4684. TestYarnCLI#testGetContainers failing in CN locale. - (Bibin A Chundatt via vvasudev) - - YARN-4569. Remove incorrect part of maxResources in FairScheduler - documentation. (Ray Chiang via kasha) - - YARN-4651. Document movetoqueue option in 'YARN Commands' - (Takashi Ohnishi via rohithsharmaks) - - YARN-4729. SchedulerApplicationAttempt#getTotalRequiredResources can throw - an NPE. (kasha) - - YARN-4701. When task logs are not available, port 8041 is referenced - instead of port 8042 (haibochen via rkanter) - - YARN-4566. Fix test failure in TestMiniYarnClusterNodeUtilization. - (Takashi Ohnishi via rohithsharmaks) - - YARN-4731. container-executor should not follow symlinks in - recursive_unlink_children (Colin Patrick McCabe via jlowe) - - YARN-4704. TestResourceManager#testResourceAllocation() fails when using - FairScheduler. (Yufei Gu via kasha) - -Release 2.8.0 - UNRELEASED - - INCOMPATIBLE CHANGES - - YARN-2336. Fair scheduler's REST API returns a missing '[' bracket JSON for - deep queue tree. (Kenji Kikushima and Akira Ajisaka via ozawa) - - NEW FEATURES - - YARN-3360. Add JMX metrics to TimelineDataManager (Jason Lowe via jeagles) - - YARN-3345. Add non-exclusive node label API. (Wangda Tan via jianhe) - - YARN-3365. Enhanced NodeManager to support using the 'tc' tool via - container-executor for outbound network traffic control. (Sidharta Seethana - via vinodkv) - - YARN-1376. NM need to notify the log aggregation status to RM through - heartbeat. (Xuan Gong via junping_du) - - YARN-3348. Add a 'yarn top' tool to help understand cluster usage. (Varun - Vasudev via jianhe) - - YARN-3347. Improve YARN log command to get AMContainer logs as well as - running containers logs. (Xuan Gong via junping_du) - - YARN-3443. Create a 'ResourceHandler' subsystem to ease addition of support - for new resource types on the NM. (Sidharta Seethana via junping_du) - - YARN-3361. CapacityScheduler side changes to support non-exclusive node - labels. (Wangda Tan via jianhe) - - YARN-3318. Create Initial OrderingPolicy Framework and FifoOrderingPolicy. - (Craig Welch via wangda) - - YARN-3326. Support RESTful API for getLabelsToNodes. (Naganarasimha G R - via ozawa) - - YARN-3354. Add node label expression in ContainerTokenIdentifier to support - RM recovery. (Wangda Tan via jianhe) - - YARN-1402. Update related Web UI and CLI with exposing client API to check - log aggregation status. (Xuan Gong via junping_du) - - YARN-3463. Integrate OrderingPolicy Framework with CapacityScheduler. - (Craig Welch via wangda) - - YARN-3410. YARN admin should be able to remove individual application - records from RMStateStore. (Rohith Sharmaks via wangda) - - YARN-3225. New parameter of CLI for decommissioning node gracefully in - RMAdmin CLI. (Devaraj K via junping_du) - - YARN-3366. Enhanced NodeManager to support classifying/shaping outgoing - network bandwidth traffic originating from YARN containers (Sidharta Seethana - via vinodkv) - - YARN-3319. Implement a FairOrderingPolicy. (Craig Welch via wangda) - - YARN-2498. Respect labels in preemption policy of capacity scheduler for - inter-queue preemption. (Wangda Tan via jianhe) - - YARN-2619. Added NodeManager support for disk io isolation through cgroups. - (Varun Vasudev and Wei Yan via vinodkv) - - YARN-3448. Added a rolling time-to-live LevelDB timeline store implementation. - (Jonathan Eagles via zjshen) - - YARN-3505. Node's Log Aggregation Report with SUCCEED should not cached in - RMApps. (Xuan Gong via junping_du) - - YARN-3541. Add version info on timeline service / generic history web UI - and REST API. (Zhijie Shen via xgong) - - YARN-160. Enhanced NodeManager to automatically obtain cpu/memory values from - underlying OS when configured to do so. (Varun Vasudev via vinodkv) - - YARN-41. The RM should handle the graceful shutdown of the NM. (Devaraj K via - junping_du) - - YARN-1012. Report NM aggregated container resource utilization in heartbeat. - (Inigo Goiri via kasha) - - YARN-3116. RM notifies NM whether a container is an AM container or normal - task container. (Giovanni Matteo Fumarola via zjshen) - - YARN-2003. Support for Application priority : Changes in RM and Capacity - Scheduler. (Sunil G via wangda) - - YARN-3656. LowCost: A Cost-Based Placement Agent for YARN Reservations. - (Jonathan Yaniv and Ishai Menache via curino) - - YARN-3852. Add docker container support to container-executor - (Abin Shahab via vvasudev) - - YARN-3853. Add docker container runtime support to LinuxContainterExecutor. - (Sidharta Seethana via vvasudev) - - YARN-3736. Add RMStateStore apis to store and load accepted reservations for - failover (adhoot via asuresh) - - YARN-3948. Display Application Priority in RM Web UI.(Sunil G via rohithsharmaks) - - YARN-3873. PendingApplications in LeafQueue should also use OrderingPolicy. - (Sunil G via wangda) - - YARN-3887. Support changing Application priority during runtime. (Sunil G - via jianhe) - - YARN-4023. Publish Application Priority to TimelineServer. (Sunil G - via rohithsharmaks) - - YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) - - YARN-4055. Report node resource utilization in heartbeat. - (Inigo Goiri via kasha) - - YARN-2923. Support configuration based NodeLabelsProvider Service in Distributed - Node Label Configuration Setup. (Naganarasimha G R) - - YARN-221. NM should provide a way for AM to tell it not to aggregate logs. - (Ming Ma via xgong) - - YARN-4014. Support user cli interface in for Application Priority. - (Rohith Sharma K S via jianhe) - - YARN-3250. Support admin cli interface in for Application Priority. - (Rohith Sharma K S via jianhe) - - YARN-3970. Add REST api support for Application Priority. - (Naganarasimha G R via vvasudev) - - YARN-2884. Added a proxy service in NM to proxy the the communication - between AM and RM. (Kishore Chaliparambil via jianhe) - - YARN-313. Add Admin API for supporting node resource configuration in - command line. (Inigo Goiri, Kenji Kikushima and Junping Du - via junping_du) - - YARN-4034. Render cluster Max Priority in scheduler metrics in RM web - UI. (Rohith Sharma K S via jianhe) - - YARN-3212. RMNode State Transition Update with DECOMMISSIONING state. - (Junping Du via wangda) - - YARN-3866. AM-RM protocol changes to support container resizing. (Meng Ding - via jianhe) - - YARN-1449. AM-NM protocol changes to support container resizing. - (Meng Ding & Wangda Tan via jianhe) - - YARN-1645. ContainerManager implementation to support container resizing. - (Meng Ding & Wangda Tan via jianhe) - - YARN-3867. ContainerImpl changes to support container resizing. (Meng Ding - via jianhe) - - YARN-1643. Make ContainersMonitor support changing monitoring size of an - allocated container. (Meng Ding and Wangda Tan) - - YARN-1644. RM-NM protocol changes and NodeStatusUpdater implementation to - support container resizing. (Meng Ding via jianhe) - - YARN-3868. Recovery support for container resizing. (Meng Ding via jianhe) - - YARN-1651. CapacityScheduler side changes to support container resize. - (Wangda Tan via jianhe) - - YARN-1897. CLI and core support for signal container functionality. - (Ming Ma via xgong) - - YARN-261. Ability to fail AM attempts (Andrey Klochkov and - Rohith Sharma K S via jlowe) - - YARN-3964. Support NodeLabelsProvider at Resource Manager side. - (Dian Fu via devaraj) - - YARN-2556. Tool to measure the performance of the timeline server (Chang Li - via sjlee) - - YARN-4262. Allow whitelisted users to run privileged docker containers. - (Sidharta Seethana via vvasudev) - - YARN-3739. Add reservation system recovery to RM recovery process. - (Subru Krishnan via adhoot) - - YARN-2729. Support script based NodeLabelsProvider Interface in Distributed Node Label - Configuration Setup. (Naganarasimha G R via rohithsharmaks) - - YARN-1510. Make NMClient support change container resources. - (Meng Ding via wangda) - - YARN-1509. Make AMRMClient support send increase container request and - get increased/decreased containers. (Meng Ding via wangda) - - YARN-4184. Remove update reservation state api from state store as its not used by - ReservationSystem (Sean Po via asuresh) - - YARN-4349. Support CallerContext in YARN. (wtan via jianhe) - - YARN-3623. Add a new config to indicate the Timeline Service version. - (Xuan Gong via junping_du) - - YARN-3226. UI changes for decommissioning node. (Sunil G via - junping_du) - - YARN-3458. CPU resource monitoring in Windows. (Inigo Goiri via cnauroth) - - YARN-4234. New put APIs in TimelineClient for ats v1.5. (Xuan Gong via - junping_du) - - YARN-4265. Provide new timeline plugin storage to support fine-grained entity - caching. (Li Lu and Jason Lowe via junping_du) - - YARN-4219. New levelDB cache storage for timeline v1.5. (Li Lu via xgong) - - YARN-4411. RMAppAttemptImpl#createApplicationAttemptReport throws - IllegalArgumentException. (Bibin A Chundatt, yarntime via devaraj) - - YARN-3223. Resource update during NM graceful decommission. (Brook Zhou - via junping_du) - - IMPROVEMENTS - - YARN-644. Basic null check is not performed on passed in arguments before - using them in ContainerManagerImpl.startContainer (Varun Saxena via bobby) - - YARN-1880. Cleanup TestApplicationClientProtocolOnHA - (ozawa via harsh) - - YARN-3357. Move TestFifoScheduler to FIFO package. (Rohith Sharmaks - via devaraj) - - YARN-3356. Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to - track used-resources-by-label. (Wangda Tan via jianhe) - - YARN-3350. YARN RackResolver spams logs with messages at info level. - (Wilfred Spiegelenburg via junping_du) - - YARN-2868. FairScheduler: Metric for latency to allocate first container - for an application. (Ray Chiang via kasha) - - YARN-3397. yarn rmadmin should skip -failover. (J.Andreina via kasha) - - YARN-3288. Document and fix indentation in the DockerContainerExecutor code - - YARN-2495. Allow admin specify labels from each NM (Distributed - configuration for node label). (Naganarasimha G R via wangda) - - YARN-3258. FairScheduler: Need to add more logging to investigate - allocations. (Anubhav Dhoot via ozawa) - - YARN-3428. Debug log resources to be localized for a container. (kasha) - - YARN-3424. Change logs for ContainerMonitorImpl's resourse monitoring - from info to debug. (Anubhav Dhoot via ozawa) - - YARN-2901. Add errors and warning metrics page to RM, NM web UI. - (Varun Vasudev via wangda) - - YARN-3294. Allow dumping of Capacity Scheduler debug logs via - web UI for a fixed time period. (Varun Vasudev via xgong) - - YARN-3293. Track and display capacity scheduler health metrics - in web UI. (Varun Vasudev via xgong) - - YARN-3394. Enrich WebApplication proxy documentation. (Naganarasimha G R - via jianhe) - - YARN-3404. Display queue name on application page. (Ryu Kobayashi via jianhe) - - YARN-2696. Queue sorting in CapacityScheduler should consider node label. - (Wangda Tan via jianhe) - - YARN-3451. Display attempt start time and elapsed time on the web UI. - (Rohith Sharmaks via jianhe) - - YARN-3494. Expose AM resource limit and usage in CS QueueMetrics. (Rohith - Sharmaks via jianhe) - - YARN-3503. Expose disk utilization percentage and bad local and log dir - counts in NM metrics. (Varun Vasudev via jianhe) - - YARN-3511. Add errors and warnings page to ATS. (Varun Vasudev via xgong) - - YARN-3406. Display count of running containers in the RM's Web UI. - (Ryu Kobayashi via ozawa) - - YARN-3363. add localization and container launch time to ContainerMetrics - at NM to show these timing information for each active container. - (zxu via rkanter) - - YARN-3396. Handle URISyntaxException in ResourceLocalizationService. - (Brahma Reddy Battula via junping_du) - - YARN-3491. PublicLocalizer#addResource is too slow. (zxu via rkanter) - - YARN-3523. Cleanup ResourceManagerAdministrationProtocol interface audience. - (Naganarasimha G R via junping_du) - - YARN-3169. Drop YARN's overview document. (Brahma Reddy Battula via ozawa) - - YARN-2784. Make POM project names consistent. (Rohith via devaraj) - - YARN-20. More information for "yarn.resourcemanager.webapp.address" in - yarn-default.xml (Bartosz Åugowski vai tgraves) - - YARN-3593. Add label-type and Improve "DEFAULT_PARTITION" in Node Labels Page. - (Naganarasimha G R via wangda) - - YARN-3018. Unified the default value for the configuration property - yarn.scheduler.capacity.node-locality-delay in code and default xml file. - (Nijel SF via vinodkv) - - YARN-2331. Distinguish shutdown during supervision vs. shutdown for - rolling upgrade. (Jason Lowe via xgong) - - YARN-1050. Document the Fair Scheduler REST API. - (Kenji Kikushima and Roman Shaposhnik via kasha) - - YARN-3271. FairScheduler: Move tests related to max-runnable-apps from - TestFairScheduler to TestAppRunnability. (nijel via kasha) - - YARN-1912. ResourceLocalizer started without any jvm memory control. - (Masatake Iwasaki via xgong) - - YARN-1287. Consolidate MockClocks. - (Sebastian Wong and Anubhav Dhoot via kasha) - - YARN-3395. FairScheduler: Trim whitespaces when using username for - queuename. (Zhihai Xu via kasha) - - YARN-3587. Fix the javadoc of DelegationTokenSecretManager in yarn, etc. - projects. (Gabor Liptak via junping_du) - - YARN-3513. Remove unused variables in ContainersMonitorImpl and add debug - log for overall resource usage by all containers. (Naganarasimha G R via devaraj) - - YARN-3613. TestContainerManagerSecurity should init and start Yarn cluster in - setup instead of individual methods. (nijel via kasha) - - YARN-3579. CommonNodeLabelsManager should support NodeLabel instead of string - label name when getting node-to-label/label-to-label mappings. (Sunil G via wangda) - - YARN-3521. Support return structured NodeLabel objects in REST API (Sunil G via wangda) - - YARN-3362. Add node label usage in RM CapacityScheduler web UI. - (Naganarasimha G R via wangda) - - YARN-3565. NodeHeartbeatRequest/RegisterNodeManagerRequest should use - NodeLabel object instead of String. (Naganarasimha G R via wangda) - - YARN-3583. Support of NodeLabel object instead of plain String - in YarnClient side. (Sunil G via wangda) - - YARN-3684. Changed ContainerExecutor's primary lifecycle methods to use a more - extensible mechanism of context objects. (Sidharta Seethana via vinodkv) - - YARN-3594. WintuilsProcessStubExecutor.startStreamReader leaks streams. - (Lars Francke via junping_du) - - YARN-3647. RMWebServices api's should use updated api from CommonNodeLabelsManager - to get NodeLabel object. (Sunil G via wangda) - - YARN-3581. Deprecate -directlyAccessNodeLabelStore in RMAdminCLI. - (Naganarasimha G R via wangda) - - YARN-3722. Merge multiple TestWebAppUtils into o.a.h.yarn.webapp.util.TestWebAppUtils. - (Masatake Iwasaki via devaraj) - - YARN-3713. Remove duplicate function call storeContainerDiagnostics in - ContainerDiagnosticsUpdateTransition (zxu via rkanter) - - YARN-3467. Expose allocatedMB, allocatedVCores, and runningContainers metrics on - running Applications in RM Web UI. (Anubhav Dhoot via kasha) - - YARN-2392. Add more diags about app retry limits on AM failures. (Steve - Loughran via jianhe) - - YARN-1462. AHS API and other AHS changes to handle tags for completed MR jobs. (xgong) - - YARN-2716. Refactor ZKRMStateStore retry code with Apache Curator. - (Karthik Kambatla via jianhe) - - YARN-3786. Document yarn class path options. - (Brahma Reddy Battula via cnauroth) - - YARN-3787. Allowed generic history service to load a number of applications whose - started time is within the given range. (Xuan Gong via zjshen) - - YARN-3785. Support for Resource as an argument during submitApp call in MockRM - test class. (Sunil G via xgong) - - YARN-3789. Improve logs for LeafQueue#activateApplications(). - (Bibin A Chundatt via devaraj) - - YARN-3148. Allow CORS related headers to passthrough in WebAppProxyServlet. - (Varun Saxena via devaraj) - - YARN-3834. Scrub debug logging of tokens during resource localization. - (Chris Nauroth via xgong) - - YARN-3827. Migrate YARN native build to new CMake framework (Alan Burlison - via Colin P. McCabe) - - YARN-3800. Reduce storage footprint for ReservationAllocation. (Anubhav Dhoot - via curino) - - YARN-3069. Document missing properties in yarn-default.xml. - (Ray Chiang via aajisaka) - - YARN-3844. Make hadoop-yarn-project Native code -Wall-clean (Alan Burlison - via Colin P. McCabe) - - YARN-3026. Move application-specific container allocation logic from - LeafQueue to FiCaSchedulerApp. (Wangda Tan via jianhe) - - YARN-3950. Add unique SHELL_ID environment variable to DistributedShell - (Robert Kanter via jlowe) - - YARN-3965. Add startup timestamp to nodemanager UI (Hong Zhiguo via jlowe) - - YARN-3543. ApplicationReport should be able to tell whether the Application - is AM managed or not. (Rohith Sharma K S via xgong) - - YARN-4004. container-executor should print output of docker logs if the docker - container exits with non-0 exit status. (Varun Vasudev via xgong) - - YARN-3961. Expose pending, running and reserved containers of a queue in REST - api and yarn top (adhoot via asuresh) - - YARN-4019. Add JvmPauseMonitor to ResourceManager and NodeManager. (Robert Kanter - via junping_du) - - YARN-3974. Refactor the reservation system test cases to use parameterized - base test. (subru via curino) - - YARN-3966. Fix excessive loggings in CapacityScheduler. (Jian He via wangda) - - YARN-4026. Refactored ContainerAllocator to accept a list of priorites - rather than a single priority. (Wangda Tan via jianhe) - - YARN-4031. Add JvmPauseMonitor to ApplicationHistoryServer and - WebAppProxyServer (djp via rkanter) - - YARN-4057. If ContainersMonitor is not enabled, only print - related log info one time. (Jun Gong via zxu) - - YARN-1556. NPE getting application report with a null appId. (Weiwei Yang via - junping_du) - - YARN-4121. Fix typos in capacity scheduler documentation. - (Kai Sasaki via vvasudev) - - YARN-4086. Allow Aggregated Log readers to handle HAR files (rkanter) - - YARN-4145. Make RMHATestBase abstract so its not run when running all - tests under that namespace (adhoot via rkanter) - - YARN-2005. Blacklisting support for scheduling AMs. (Anubhav Dhoot via kasha) - - YARN-3717. Expose app/am/queue's node-label-expression to RM web UI / - CLI / REST-API. (Naganarasimha G R via wangda) - - YARN-4149. yarn logs -am should provide an option to fetch all the log files - (Varun Vasudev via xgong) - - YARN-2597. MiniYARNCluster should propagate reason for AHS not starting. - (stevel) - - MAPREDUCE-6478. Add an option to skip cleanupJob stage or ignore cleanup - failure during commitJob. (Junping Du via wangda) - - YARN-3920. FairScheduler container reservation on a node should be - configurable to limit it to large containers (adhoot via asuresh) - - HADOOP-12428. Fix inconsistency between log-level guards and statements. - (Jagadesh Kiran N and Jackie Chang via ozawa) - - YARN-4095. Avoid sharing AllocatorPerContext object in LocalDirAllocator - between ShuffleHandler and LocalDirsHandlerService. (Zhihai Xu via jlowe) - - YARN-4176. Resync NM nodelabels with RM periodically for distributed nodelabels. - (Bibin A Chundatt via wangda) - - YARN-4215. RMNodeLabels Manager Need to verify and replace node labels for the - only modified Node Label Mappings in the request. (Naganarasimha G R via wangda) - - YARN-3943. Use separate threshold configurations for disk-full detection - and disk-not-full detection. (Zhihai Xu via jlowe) - - YARN-4252. Log container-executor invocation details when exit code is non-zero. - (Sidharta Seethana via vvasudev) - - YARN-4258. Add support for controlling capabilities for docker containers. - (Sidharta Seethana via vvasudev) - - YARN-4162. CapacityScheduler: Add resource usage by partition and queue capacity - by partition to REST API. (Naganarasimha G R via wangda) - - YARN-4170. AM need to be notified with priority in AllocateResponse. - (Sunil G via jianhe) - - YARN-4267. Add additional logging to container launch implementations in - container-executor. (Sidharta Seethana via vvasudev) - - YARN-3985. Make ReservationSystem persist state using RMStateStore - reservation APIs. (adhoot via asuresh) - - YARN-4243. Add retry on establishing Zookeeper conenction in - EmbeddedElectorService#serviceInit. (Xuan Gong via junping_du) - - YARN-2913. Fair scheduler should have ability to set MaxResourceDefault for - each queue. (Siqi Li via mingma) - - YARN-4296. DistributedShell Log.info is not friendly. - (Xiaowei Wang via stevel) - - YARN-3738. Add support for recovery of reserved apps running under dynamic - queues (subru via asuresh) - - YARN-4285. Display resource usage as percentage of queue and cluster in the - RM UI (Varun Vasudev via wangda) - - YARN-3216. Max-AM-Resource-Percentage should respect node labels. - (Sunil G via wangda) - - YARN-4310. FairScheduler: Log skipping reservation messages at DEBUG level (asuresh) - - YARN-4279. Mark ApplicationId and ApplicationAttemptId static methods as @Public, - @Unstable. (stevel) - - YARN-3454. Add efficient merge operation to RLESparseResourceAllocation - (Carlo Curino via asuresh) - - YARN-3980. Plumb resource-utilization info in node heartbeat through to the - scheduler. (Inigo Goiri via kasha) - - YARN-4132. Separate configs for nodemanager to resourcemanager connection - timeout and retries (Chang Li via jlowe) - - YARN-4292. ResourceUtilization should be a part of NodeInfo REST API. - (Sunil G via wangda) - - YARN-4405. Support node label store in non-appendable file system. (Wangda - Tan via jianhe) - - YARN-4358. Reservation System: Improve relationship between SharingPolicy - and ReservationAgent. (Carlo Curino via asuresh) - - YARN-3456. Improve handling of incomplete TimelineEntities. (Varun Saxena - via rohithsharmaks) - - YARN-4248. REST API for submit/update/delete Reservations. (curino) - - YARN-3946. Update exact reason as to why a submitted app is in ACCEPTED state to - app's diagnostic message. (Naganarasimha G R via wangda) - - YARN-4309. Add container launch related debug information to container logs - when a container fails. (Varun Vasudev via wangda) - - YARN-4418. AM Resource Limit per partition can be updated to ResourceUsage as well. - (Sunil G via wangda) - - YARN-4207. Add a non-judgemental YARN app completion status. (Rich Haase via sseth) - - YARN-4293. ResourceUtilization should be a part of yarn node CLI. (Sunil G via wangda) - - YARN-4164. Changed updateApplicationPriority API to return the updated - application priority. (Rohith Sharma K S via jianhe) - - YARN-4480. Clean up some inappropriate imports. (Kai Zheng via umamahesh) - - YARN-4290. Add -showDetails option to YARN Nodes CLI to print all nodes reports - information. (Sunil G via wangda) - - YARN-4098. Document ApplicationPriority feature. (Rohith Sharma K S via jianhe) - - YARN-4544. All the log messages about rolling monitoring interval are - shown with WARN level. (Takashi Ohnishi via aajisaka) - - YARN-4537. Pull out priority comparison from fifocomparator and use compound - comparator for FifoOrdering policy. (Rohith Sharma K S via jianhe) - - YARN-4582. Label-related invalid resource request exception should be able to - properly handled by application. (Bibin A Chundatt via wangda) - - YARN-4304. AM max resource configuration per partition to be displayed/updated - correctly in UI and in various partition related metrics. (Sunil G via wangda) - - YARN-4524. Cleanup AppSchedulingInfo. (Karthik Kambatla via wangda) - - YARN-4371. "yarn application -kill" should take multiple application ids - (Sunil G via jlowe) - - YARN-4340. Add "list" API to reservation system. (Sean Po via wangda) - - YARN-4100. Add Documentation for Distributed and Delegated-Centralized - Node Labels feature. (Naganarasimha G R via devaraj) - - YARN-4662. Document some newly added metrics. (Jian He via xgong) - - YARN-4360. Improve GreedyReservationAgent to support "early" allocations, - and performance improvements (curino via asuresh) - - YARN-4420. Add REST API for List Reservations. (Sean Po via curino) - - YARN-4519. Potential deadlock of CapacityScheduler between decrease container - and assign containers. (Meng Ding via jianhe) - - YARN-4138. Roll back container resource allocation after resource - increase token expires. (Meng Ding via jianhe) - - YARN-2575. Create separate ACLs for Reservation create/update/delete/list - ops (Sean Po via asuresh) - - YARN-4682. AMRM client to log when AMRM token updated. - (Prabhu Joseph via stevel) - - YARN-4579. Allow DefaultContainerExecutor container log directory - permissions to be configurable (rchiang via rkanter) - - OPTIMIZATIONS - - YARN-3339. TestDockerContainerExecutor should pull a single image and not - the entire centos repository. (Ravindra Kumar Naik via raviprak) - - YARN-3469. ZKRMStateStore: Avoid setting watches that are not required. - (Jun Gong via kasha) - - YARN-3006. Improve the error message when attempting manual failover with - auto-failover enabled. (Akira AJISAKA via wangda) - - YARN-3547. FairScheduler: Apps that have no resource demand should not participate - scheduling. (Xianyin Xin via kasha) - - YARN-3259. FairScheduler: Trigger fairShare updates on node events. - (Anubhav Dhoot via kasha) - - YARN-2768. Avoid cloning Resource in FSAppAttempt#updateDemand. - (Hong Zhiguo via kasha) - - YARN-3983. Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend - container allocation logic. (Wangda Tan via jianhe) - - YARN-3635. Refactored current queue mapping implementation in CapacityScheduler - to use a generic PlacementManager framework. (Wangda Tan via jianhe) - - YARN-4066. Large number of queues choke fair scheduler. - (Johan Gustavsson via kasha) - - YARN-4720. Skip unnecessary NN operations in log aggregation. - (Jun Gong via mingma) - - BUG FIXES - - YARN-3197. Confusing log generated by CapacityScheduler. (Varun Saxena - via devaraj) - - YARN-3305. Normalize AM resource request on app submission. (Rohith Sharmaks - via jianhe) - - YARN-3205 FileSystemRMStateStore should disable FileSystem Cache to avoid - get a Filesystem with an old configuration. (Zhihai Xu via ozawa) - - YARN-3269. Yarn.nodemanager.remote-app-log-dir could not be configured to - fully qualified path. (Xuan Gong via junping_du) - - YARN-3241. FairScheduler handles "invalid" queue names inconsistently. - (Zhihai Xu via kasha) - - YARN-3383. AdminService should use "warn" instead of "info" to log exception - when operation fails. (Li Lu via wangda) - - YARN-3400. [JDK 8] Build Failure due to unreported exceptions in - RPCUtil (rkanter) - - YARN-3412. RM tests should use MockRM where possible. (kasha) - - YARN-3425. NPE from RMNodeLabelsManager.serviceStop when - NodeLabelsManager.serviceInit failed. (Bibin A Chundatt via wangda) - - YARN-3415. Non-AM containers can be counted towards amResourceUsage of a - Fair Scheduler queue (Zhihai Xu via Sandy Ryza) - - YARN-3435. AM container to be allocated Appattempt AM container shown as null. - (Bibin A Chundatt via xgong) - - YARN-2666. TestFairScheduler.testContinuousScheduling fails Intermittently. - (Zhihai Xu via ozawa) - - YARN-3429. TestAMRMTokens.testTokenExpiry fails Intermittently with - error message:Invalid AMRMToken (zxu via rkanter) - - YARN-3110. Few issues in ApplicationHistory web ui. (Naganarasimha G R via xgong) - - YARN-3459. Fix failiure of TestLog4jWarningErrorMetricsAppender. - (Varun Vasudev via wangda) - - YARN-3266. RMContext#inactiveNodes should have NodeId as map key. - (Chengbing Liu via jianhe) - - YARN-3436. Fix URIs in documantion of YARN web service REST APIs. - (Bibin A Chundatt via ozawa) - - YARN-3021. YARN's delegation-token handling disallows certain trust setups - to operate properly over DistCp. (Yongjun Zhang via jianhe) - - YARN-3136. Fixed a synchronization problem of - AbstractYarnScheduler#getTransferredContainers. (Sunil G via jianhe) - - YARN-3495. Confusing log generated by FairScheduler. - (Brahma Reddy Battula via ozawa) - - YARN-3387. Previous AM's container completed status couldn't pass to current - AM if AM and RM restarted during the same time. (sandflee via jianhe) - - YARN-3444. Fix typo capabililty. (Gabor Liptak via aajisaka) - - YARN-3530. ATS throws exception on trying to filter results without otherinfo. - (zhijie shen via xgong) - - YARN-2740. Fix NodeLabelsManager to properly handle node label modifications - when distributed node label configuration enabled. (Naganarasimha G R via wangda) - - YARN-3517. RM web ui for dumping scheduler logs should be for admins only - (Varun Vasudev via tgraves) - - YARN-3533. Test: Fix launchAM in MockRM to wait for attempt to be scheduled. - (Anubhav Dhoot via jianhe) - - YARN-3564. Fix TestContainerAllocation.testAMContainerAllocationWhenDNSUnavailable - fails randomly. (Jian He via wangda) - - YARN-2893. AMLaucher: sporadic job failures due to EOFException in - readTokenStorageStream. (Zhihai Xu via gera) - - YARN-2454. Fix compareTo of variable UNBOUNDED in o.a.h.y.util.resource.Resources. - (Xu Yang via junping_du) - - YARN-1993. Cross-site scripting vulnerability in TextView.java. (Kenji Kikushima - via ozawa) - - YARN-3097. Logging of resource recovery on NM restart has redundancies - (Eric Payne via jlowe) - - YARN-3375. NodeHealthScriptRunner.shouldRun() check is performing 3 times for - starting NodeHealthScriptRunner. (Devaraj K via wangda) - - YARN-2725. Added test cases of retrying creating znode in ZKRMStateStore. - (Tsuyoshi Ozawa via jianhe) - - YARN-3552. RM Web UI shows -1 running containers for completed apps - (Rohith via jlowe) - - YARN-2123. Progress bars in Web UI always at 100% due to non-US locale. - (Akira AJISAKA via xgong) - - YARN-3343. Increased TestCapacitySchedulerNodeLabelUpdate#testNodeUpdate - timeout. (Rohith Sharmaks via jianhe) - - YARN-3582. NPE in WebAppProxyServlet. (jian he via xgong) - - YARN-3577. Misspelling of threshold in log4j.properties for tests. - (Brahma Reddy Battula via aajisaka) - - YARN-3584. Fixed attempt diagnostics format shown on the UI. (nijel via - jianhe) - - YARN-1832. Fix wrong MockLocalizerStatus#equals implementation. - (Hong Zhiguo via aajisaka) - - YARN-3572. Correct typos in WritingYarnApplications.md. - (Gabor Liptak via aajisaka) - - YARN-3592. Fix typos in RMNodeLabelsManager. (Sunil G via devaraj) - - YARN-3589. RM and AH web UI display DOCTYPE wrongly. (Rohith via ozawa) - - YARN-3600. AM container link is broken (Naganarasimha G R via tgraves) - - YARN-3604. Fixed ZKRMStateStore#removeApplication to also disable watch. - (zhihai xu via jianhe) - - YARN-3473. Fix RM Web UI configuration for some properties (rchiang via rkanter) - - YARN-2206. Updated document for applications REST API response examples. (Kenji - Kikushima and Brahma Reddy Battula via zjshen) - - YARN-3602. TestResourceLocalizationService.testPublicResourceInitializesLocalDir - fails Intermittently due to IOException from cleanup. (zhihai xu via xgong) - - YARN-3629. NodeID is always printed as "null" in node manager initialization log. - (nijel via devaraj) - - YARN-2921. Fix MockRM/MockAM#waitForState sleep too long. - (Tsuyoshi Ozawa via wangda) - - YARN-1519. Check in container-executor if sysconf is implemented before - using it (Radim Kolar and Eric Payne via raviprak) - - YARN-2421. RM still allocates containers to an app in the FINISHING - state (Chang Li via jlowe) - - YARN-3302. TestDockerContainerExecutor should run automatically if it can - detect docker in the usual place (Ravindra Kumar Naik via raviprak) - - YARN-2821. Fixed a problem that DistributedShell AM may hang if restarted. - (Varun Vasudev via jianhe) - - YARN-3654. ContainerLogsPage web UI should not have meta-refresh. (Xuan Gong - via jianhe) - - YARN-3707. RM Web UI queue filter doesn't work. (Wangda Tan via jianhe) - - YARN-3632. Ordering policy should be allowed to reorder an application when - demand changes. (Craig Welch via jianhe) - - YARN-3716. Node-label-expression should be included by - ResourceRequestPBImpl.toString. (Xianyin Xin via wangda) - - YARN-3751. Fixed AppInfo to check if used resources are null. (Sunil G via - zjshen) - - YARN-3762. FairScheduler: CME on FSParentQueue#getQueueUserAclInfo. (kasha) - - YARN-3749. We should make a copy of configuration when init MiniYARNCluster - with multiple RMs. (Chun Chen via xgong) - - YARN-3766. Fixed the apps table column error of generic history web UI. - (Xuan Gong via zjshen) - - YARN-3655. FairScheduler: potential livelock due to maxAMShare limitation - and container reservation. (Zhihai Xu via kasha) - - YARN-3747. TestLocalDirsHandlerService should delete the created test directory logDir2. - (David Moore via devaraj) - - YARN-3778. Fix Yarn resourcemanger CLI usage. (Brahma Reddy Battula via xgong) - - YARN-3794. TestRMEmbeddedElector fails because of ambiguous LOG reference. - (Chengbing Liu via devaraj) - - YARN-3714. AM proxy filter can not get RM webapp address from - yarn.resourcemanager.hostname.rm-id. (Masatake Iwasaki via xgong) - - YARN-3617. Fix WindowsResourceCalculatorPlugin.getCpuFrequency() - returning always -1. (J.Andreina via devaraj) - - YARN-3824. Fix two minor nits in member variable properties - of YarnConfiguration. (Ray Chiang via devaraj) - - YARN-3835. hadoop-yarn-server-resourcemanager test package bundles - core-site.xml, yarn-site.xml (vamsee via rkanter) - - YARN-3790. usedResource from rootQueue metrics may get stale data for FS - scheduler after recovering the container (Zhihai Xu via rohithsharmaks) - - YARN-3826. Race condition in ResourceTrackerService leads to - wrong diagnostics messages. (Chengbing Liu via devaraj) - - YARN-3745. SerializedException should also try to instantiate internal - exception with the default constructor. (Lavkesh Lahngir via devaraj) - - YARN-2871. TestRMRestart#testRMRestartGetApplicationList sometime fails in trunk. - (zhihai xu via xgong) - - YARN-3859. LeafQueue doesn't print user properly for application add. - (Varun Saxena via devaraj) - - YARN-3860. rmadmin -transitionToActive should check the state of non-target node. - (Masatake Iwasaki via junping_du) - - YARN-3695. ServerProxy (NMProxy, etc.) shouldn't retry forever for non - network exception. (Raju Bairishetti via jianhe) - - YARN-3770. SerializedException should also handle java.lang.Error on - de-serialization. (Lavkesh Lahngir via jianhe) - - YARN-3768. ArrayIndexOutOfBoundsException with empty environment variables. - (Zhihai Xu via gera) - - YARN-3823. Fix mismatch in default values for - yarn.scheduler.maximum-allocation-vcores property. (Ray Chiang via devaraj) - - YARN-3830. AbstractYarnScheduler.createReleaseCache may try to clean a null - attempt. (nijel via devaraj) - - YARN-3875. FSSchedulerNode#reserveResource() doesn't print Application Id - properly in log. (Bibin A Chundatt via devaraj) - - YARN-3882. AggregatedLogFormat should close aclScanner and ownerScanner - after create them. (zhihai xu via xgong) - - YARN-3837. javadocs of TimelineAuthenticationFilterInitializer give wrong - prefix for auth options. (Bibin A Chundatt via devaraj) - - YARN-2194. Fix bug causing CGroups functionality to fail on RHEL7. - (Wei Yan via vvasudev) - - YARN-3892. Fixed NPE on RMStateStore#serviceStop when - CapacityScheduler#serviceInit fails. (Bibin A Chundatt via jianhe) - - YARN-3888. ApplicationMaster link is broken in RM WebUI when appstate is NEW. - (Bibin A Chundatt via xgong) - - YARN-3917. getResourceCalculatorPlugin for the default should intercept all - exceptions. (gera) - - YARN-3894. RM startup should fail for wrong CS xml NodeLabel capacity - configuration. (Bibin A Chundatt via wangda) - - YARN-3381. Fix typo InvalidStateTransitonException. - (Brahma Reddy Battula via aajisaka) - - YARN-3453. Ensure preemption logic in FairScheduler uses DominantResourceCalculator - in DRF queues to prevent unnecessary thrashing. (asuresh) - - YARN-3174. Consolidate the NodeManager and NodeManagerRestart documentation - into one. (Masatake Iwasaki via ozawa) - - YARN-3805. Update the documentation of Disk Checker based on YARN-90. - (Masatake Iwasaki via ozawa) - - YARN-3930. FileSystemNodeLabelsStore should make sure edit log file closed when - exception is thrown. (Dian Fu via wangda) - - YARN-3885. ProportionalCapacityPreemptionPolicy doesn't preempt if queue is - more than 2 level. (Ajith S via wangda) - - YARN-3932. SchedulerApplicationAttempt#getResourceUsageReport and UserInfo - should based on total-used-resources. (Bibin A Chundatt via wangda) - - YARN-3954. Fix TestYarnConfigurationFields#testCompareConfigurationClassAgainstXml. - (varun saxena via rohithsharmaks) - - YARN-3956. Fix TestNodeManagerHardwareUtils fails on Mac (Varun Vasudev via wangda) - - YARN-3941. Proportional Preemption policy should try to avoid sending duplicate - PREEMPT_CONTAINER event to scheduler. (Sunil G via wangda) - - YARN-3900. Protobuf layout of yarn_security_token causes errors in other protos - that include it (adhoot via rkanter) - - YARN-3845. Scheduler page does not render RGBA color combinations in IE11. - (Contributed by Mohammad Shahid Khan) - - YARN-3957. FairScheduler NPE In FairSchedulerQueueInfo causing scheduler page to - return 500. (Anubhav Dhoot via kasha) - - YARN-3973. Recent changes to application priority management break - reservation system from YARN-1051. (Carlo Curino via wangda) - - YARN-3958. TestYarnConfigurationFields should be moved to hadoop-yarn-api - module. (Varun Saxena via aajisaka) - - YARN-3846. RM Web UI queue filter is not working for sub queue. - (Mohammad Shahid Khan via jianhe) - - YARN-3982. container-executor parsing of container-executor.cfg broken in - trunk and branch-2. (Varun Vasudev via xgong) - - YARN-3919. NPEs' while stopping service after exception during - CommonNodeLabelsManager#start. (varun saxane via rohithsharmaks) - - YARN-3963. AddNodeLabel on duplicate label addition shows success. - (Bibin A Chundatt via wangda) - - YARN-3971. Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue - on nodelabel recovery. (Bibin A Chundatt via wangda) - - YARN-433. When RM is catching up with node updates then it should not expire - acquired containers. (Xuan Gong via zxu) - - YARN-3992. TestApplicationPriority.testApplicationPriorityAllocation fails - intermittently. (Contributed by Sunil G) - - YARN-3987. Send AM container completed msg to NM once AM finishes. - (sandflee via jianhe) - - YARN-4028. AppBlock page key update and diagnostics value null on recovery - (Bibin A Chundatt via xgong) - - YARN-3986. getTransferredContainers in AbstractYarnScheduler should be present - in YarnScheduler interface instead. (Varun Saxena via rohithsharmaks) - - YARN-4082. Container shouldn't be killed when node's label updated. - (Wangda Tan via vvasudev) - - YARN-4073. Removed unused ApplicationACLsManager in ContainerManagerImpl constructor. - (Naganarasimha G R via rohithsharmaks) - - YARN-4024. YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. - (Hong Zhiguo via wangda) - - YARN-3591. Resource localization on a bad disk causes subsequent containers failure. - (Lavkesh Lahngir via vvasudev) - - YARN-4106. NodeLabels for NM in distributed mode is not updated even after - clusterNodelabel addition in RM. (Bibin A Chundatt via wangda) - - YARN-4115. Reduce loglevel of ContainerManagementProtocolProxy to Debug - (adhoot via rkanter) - - YARN-4151. Fix findbugs errors in hadoop-yarn-server-common module. - (Meng Ding via wangda) - - YARN-4078. Add getPendingResourceRequestForAttempt in YarnScheduler interface. - (Naganarasimha G R via jianhe) - - YARN-4135. Improve the assertion message in MockRM while failing after waiting for the state. - (Nijel S F via rohithsharmaks) - - YARN-4167. NPE on RMActiveServices#serviceStop when store is null. (Bibin A Chundatt via rohithsharmaks) - - YARN-4113. RM should respect retry-interval when uses RetryPolicies.RETRY_FOREVER. - (Sunil G via wangda) - - YARN-4188. Make MoveApplicationAcrossQueues abstract, newInstance static. - (Giovanni Matteo Fumarola via cdouglas) - - YARN-4171. Fix findbugs warnings in YARN-1197 branch. (Wangda Tan via jianhe) - - YARN-4152. NodeManager crash with NPE when LogAggregationService#stopContainer called for - absent container. (Bibin A Chundatt via rohithsharmaks) - - YARN-4044. Running applications information changes such as movequeue is not published to - TimeLine server. (Sunil G via rohithsharmaks) - - YARN-4204. ConcurrentModificationException in FairSchedulerQueueInfo. (adhoot) - - YARN-4141. Runtime Application Priority change should not throw exception - for applications at finishing states (Sunil G via jlowe) - - YARN-4228. FileSystemRMStateStore use IOUtils#close instead of fs#close. (Bibin A Chundatt via rohithsharmaks) - - YARN-4235. FairScheduler PrimaryGroup does not handle empty groups returned - for a user. (Anubhav Dhoot via rohithsharmaks) - - YARN-4140. RM container allocation delayed incase of app submitted to - Nodelabel partition. (Bibin A Chundatt via wangda) - - YARN-4201. AMBlacklist does not work for minicluster. (Jun Gong via zxu) - - YARN-4230. RM crashes with NPE when increasing container resource if there is no headroom left. - (Meng Ding via jianhe) - - YARN-4017. container-executor overuses PATH_MAX. (Sidharta Seethana via vvasudev) - - YARN-4253. Standardize on using PrivilegedOperationExecutor for all - invocations of container-executor in LinuxContainerExecutor. (Sidharta Seethana via vvasudev) - - YARN-4255. container-executor does not clean up docker operation command files. - (Sidharta Seethana via vvasudev) - - YARN-4250. NPE in AppSchedulingInfo#isRequestLabelChanged. (Brahma Reddy Battula via rohithsharmaks) - - YARN-4000. RM crashes with NPE if leaf queue becomes parent queue during restart. - (Varun Saxena via jianhe) - - YARN-4155. TestLogAggregationService.testLogAggregationServiceWithInterval failing - (Bibin A Chundatt via stevel) - - YARN-4270. Limit application resource reservation on nodes for non-node/rack - specific requests (asuresh) - - YARN-4256. YARN fair scheduler vcores with decimal values. (Jun Gong via zxu) - - YARN-4294. [JDK8] Fix javadoc errors caused by wrong reference and illegal - tag. (aajisaka) - - YARN-4289. TestDistributedShell failing with bind exception. - (Brahma Reddy Battula via stevel) - - YARN-3724. Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) - - YARN-4246. NPE while listing app attempt. (Nijel S F via rohithsharmaks) - - YARN-3528. Tests with 12345 as hard-coded port break jenkins. - (Brahma Reddy Battula via ozawa) - - YARN-4223. Fixed findbugs warnings in hadoop-yarn-server-nodemanager project - (varun saxena via rohithsharmaks) - - YARN-4284. condition for AM blacklisting is too narrow (Sangjin Lee via - jlowe) - - YARN-4169. Fix racing condition of TestNodeStatusUpdaterForLabels. - (Naganarasimha G R via wangda) - - YARN-4300. [JDK8] Fix javadoc errors caused by wrong tags. (aajisaka) - - YARN-3573. MiniMRYarnCluster constructor that starts the timeline server - using a boolean should be marked deprecated. (Brahma Reddy Battula via ozawa) - - YARN-4302. SLS not able start due to NPE in SchedulerApplicationAttempt - (Bibin A Chundatt via vvasudev) - - YARN-4251. TestAMRMClientOnRMRestart#testAMRMClientOnAMRMTokenRollOverOnRMRestart - is failing. (Brahma Reddy Battula via ozawa) - - YARN-4130. Duplicate declaration of ApplicationId in RMAppManager#submitApplication method. - (Kai Sasaki via rohithsharmaks) - - YARN-4288. Fixed RMProxy to retry on IOException from local host. - (Junping Du via jianhe) - - YARN-4127. RM fail with noAuth error if switched from failover to non-failover. - (Varun Saxena via jianhe) - - YARN-4345. yarn rmadmin -updateNodeResource doesn't work (Junping Du via - jlowe) - - YARN-4367. SLS webapp doesn't load. (kasha). - - YARN-4298. Fix findbugs warnings in hadoop-yarn-common. - (Sunil G via aajisaka) - - YARN-4387. Fix typo in FairScheduler log message. (Xin Wang via ozawa) - - YARN-4384. updateNodeResource CLI should not accept negative values for resource. - (Junping Du via wangda) - - YARN-4408. Fix issue that NodeManager reports negative running containers. - (Robert Kanter via junping_du) - - YARN-4392. ApplicationCreatedEvent event time resets after RM restart/failover. - (Naganarasimha G R and Xuan Gong via xgong) - - YARN-4431. Not necessary to do unRegisterNM() if NM get stop due to failed to connect - to RM. (Junping Du via rohithsharmaks) - - YARN-4421. Remove dead code in RmAppImpl.RMAppRecoveredTransition. - (Daniel Templeton via rohithsharmaks) - - YARN-4403. (AM/NM/Container)LivelinessMonitor should use monotonic time - when calculating period. (Junping Du via jianhe) - - YARN-4402. TestNodeManagerShutdown And TestNodeManagerResync fails with - bind exception. (Brahma Reddy Battula via jianhe) - - YARN-4440. FSAppAttempt#getAllowedLocalityLevelByTime should init the - lastScheduler time. (Lin Yiqun via zxu) - - YARN-4452. NPE when submit Unmanaged application. (Naganarasimha G R - via junping_du) - - YARN-4225. Add preemption status to yarn queue -status for capacity scheduler. - (Eric Payne via wangda) - - YARN-4416. Deadlock due to synchronised get Methods in AbstractCSQueue. - (Naganarasimha G R via wangda) - - YARN-4461. Redundant nodeLocalityDelay log in LeafQueue (Eric Payne via - jlowe) - - YARN-4454. NM to nodelabel mapping going wrong after RM restart. - (Bibin A Chundatt via wangda) - - YARN-4477. FairScheduler: Handle condition which can result in an - infinite loop in attemptScheduling. (Tao Jie via asuresh) - - YARN-4400. AsyncDispatcher.waitForDrained should be final. (Daniel Templeton - via junping_du) - - YARN-1382. Remove unusableRMNodesConcurrentSet (never used) in NodeListManager - to get rid of memory leak. (Rohith Sharma K S via junping_du) - - YARN-4546. ResourceManager crash due to scheduling opportunity overflow. - (Jason Lowe via junping_du) - - YARN-4479. Change CS LeafQueue pendingOrderingPolicy to hornor recovered apps. - (Rohith Sharma K S via jianhe) - - YARN-4534. Remove the redundant symbol in yarn rmadmin help msg. - (Lin Yiqun via aajisaka) - - YARN-4389. "yarn.am.blacklisting.enabled" and "yarn.am.blacklisting.disable- - failure-threshold" should be app specific rather than a setting for whole - YARN cluster. (Sunil G via rohithsharmaks) - - YARN-4581. AHS writer thread leak makes RM crash while RM is recovering. - (sandflee via junping_du) - - YARN-4538. QueueMetrics pending cores and memory metrics wrong. - (Bibin A Chundatt via wangda) - - YARN-4596. SystemMetricPublisher should not swallow error messages from - TimelineClient#putEntities. (Li Lu via jianhe) - - YARN-4502. Fix two AM containers get allocated when AM restart. - (Vinod Kumar Vavilapalli via wangda) - - YARN-4565. Fix a bug that leads to AM resource limit not hornored when - sizeBasedWeight enabled for FairOrderingPolicy. (wtan via jianhe) - - YARN-4557. Fix improper Queues sorting in PartitionedQueueComparator - when accessible-node-labels=*. (Naganarasimha G R via wangda) - - YARN-4608. Redundant code statement in WritingYarnApplications. - (Kai Sasaki via aajisaka) - - YARN-4605. Spelling mistake in the help message of "yarn applicationattempt" - command. (Weiwei Yang via aajisaka) - - YARN-4614. Fix random failure in TestApplicationPriority#testApplicationPriority - AllocationWithChangeInPriority. (Sunil G via rohithsharmaks) - - YARN-4592. Remove unused GetContainerStatus proto. (Chang Li via aajisaka) - - YARN-4520. Finished app info is unnecessarily persisted in NM state-store - if container is acquired but not lunched on this node. (sandflee via jianeh) - - YARN-4643. Container recovery is broken with delegating container runtime - (Sidharta Seethana via jlowe) - - YARN-4617. LeafQueue#pendingOrderingPolicy should always use fixed ordering - policy instead of using same as active applications ordering policy. - (Rohith Sharma K S via jianhe) - - YARN-4667. RM Admin CLI for refreshNodesResources throws NPE when nothing - is configured. (Naganarasimha G R via devaraj) - - YARN-4654. Yarn node label CLI should parse "=" correctly when trying to - remove all labels on a node. (Naganarasimha G R via rohithsharmaks) - - YARN-4386. refreshNodesGracefully() should send recommission event to active - RMNodes only. (Kuhu Shukla via junping_du) - - YARN-4709. NMWebServices produces incorrect JSON for containers. - (Varun Saxena via vvasudev) - - YARN-4748. ApplicationHistoryManagerOnTimelineStore should not - swallow exceptions on generateApplicationReport. (Li Lu via jianhe) - - YARN-4671. There is no need to acquire CS lock when completing a container. - (Meng Ding via jianhe) - -Release 2.7.3 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - YARN-4287. Capacity Scheduler: Rack Locality improvement (Nathan Roberts via wangda) - - YARN-4492. Add documentation for preemption supported in Capacity - scheduler (Naganarasimha G R via jlowe) - - YARN-4653. Document YARN security model from the perspective of - Application Developers. (Steve Loughran via jianhe) - - OPTIMIZATIONS - - BUG FIXES - - YARN-4326. Fix TestDistributedShell timeout as AHS in MiniYarnCluster no longer - binds to default port 8188. (Meng Ding via wangda) - - YARN-4347. Resource manager fails with Null pointer exception. (Jian He via wangda) - - YARN-4374. RM capacity scheduler UI rounds user limit factor (Chang Li via - jlowe) - - YARN-3849. Too much of preemption activity causing continuos killing of - containers across queues. (Sunil G via wangda) - - YARN-3769. Consider user limit when calculating total pending resource for - preemption policy in Capacity Scheduler. (Eric Payne via wangda) - - YARN-4344. NMs reconnecting with changed capabilities can lead to wrong - cluster resource calculations (Varun Vasudev via jlowe) - - YARN-4365. FileSystemNodeLabelStore should check for root dir existence on - startup (Kuhu Shukla via jlowe) - - YARN-4380. TestResourceLocalizationService.testDownloadingResourcesOnContainerKill - fails intermittently. (Varun Saxena via ozawa) - - YARN-4398. Remove unnecessary synchronization in RMStateStore. (Ning Ding via jianhe) - - YARN-3840. Resource Manager web ui issue when sorting application by id - (with application having id > 9999) ï¼Mohammad Shahid Khan & Varun Saxena - via jianhe) - - YARN-4422. Generic AHS sometimes doesn't show started, node, or logs on App page - (Eric Payne via jeagles) - - YARN-4439. Clarify NMContainerStatus#toString method. (Jian He via xgong) - - YARN-4452. NPE when submit Unmanaged application. (Naganarasimha G R via - junping_du) - - YARN-4315. NaN in Queue percentage for cluster apps page. (Bibin A Chundatt via wangda) - - YARN-4510. Fix SLS startup failure caused by NPE. (Bibin A Chundatt via wangda) - - YARN-4546. ResourceManager crash due to scheduling opportunity overflow. - (Jason Lowe via junping_du) - - YARN-3695. ServerProxy (NMProxy, etc.) shouldn't retry forever for non - network exception. (Raju Bairishetti via jianhe) - - YARN-4414. Nodemanager connection errors are retried at multiple levels - (Chang Li via jlowe) - - YARN-4581. AHS writer thread leak makes RM crash while RM is recovering. - (sandflee via junping_du) - - YARN-4610. Reservations continue looking for one app causes other apps to - starve (jlowe) - - YARN-4598. Invalid event: RESOURCE_FAILED at - CONTAINER_CLEANEDUP_AFTER_KILL (tangshangwen via jlowe) - - YARN-4428. Redirect RM page to AHS page when AHS turned on and RM page is - not available (Chang Li via jlowe) - - YARN-3102. Decommisioned Nodes not listed in Web UI (Kuhu Shukla via - jlowe) - - YARN-3602. TestResourceLocalizationService.testPublicResourceInitializesLocalDir - fails Intermittently due to IOException from cleanup. (zhihai xu via xgong) - - YARN-4707. Remove the extra char (>) from SecureContainer.md. - (Brahma Reddy Battula via aajisaka) - - YARN-2046. Out of band heartbeats are sent only on container kill and - possibly too early (Ming Ma via jlowe) - - YARN-4723. NodesListManager$UnknownNodeId ClassCastException (Kuhu Shukla - via jlowe) - -Release 2.7.2 - 2016-01-25 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - YARN-4009. CORS support for ResourceManager REST API. ( Varun Vasudev via jeagles) - - YARN-3170. YARN architecture document needs updating. (Brahma Reddy Battula - via ozawa) - - YARN-3967. Fetch the application report from the AHS if the RM does not know about it. - (Mit Desai via xgong) - - YARN-2801. Add documentation for node labels feature. (Wangda Tan and Naganarasimha - G R via ozawa) - - YARN-2513. Host framework UIs in YARN for use with the ATS (jeagles) - - OPTIMIZATIONS - - BUG FIXES - - YARN-3793. Several NPEs when deleting local files on NM recovery (Varun - Saxena via jlowe) - - YARN-3508. Prevent processing preemption events on the main RM dispatcher. - (Varun Saxena via wangda) - - YARN-3690. [JDK8] 'mvn site' fails. (Brahma Reddy Battula via aajisaka) - - YARN-3905. Application History Server UI NPEs when accessing apps run after - RM restart (Eric Payne via jeagles) - - YARN-3535. Scheduler must re-request container resources when RMContainer transitions - from ALLOCATED to KILLED (rohithsharma and peng.zhang via asuresh) - - YARN-3878. AsyncDispatcher can hang while stopping if it is configured for - draining events on stop. (Varun Saxena via jianhe) - - YARN-3969. Allow jobs to be submitted to reservation that is active - but does not have any allocations. (subru via curino) - - YARN-3925. ContainerLogsUtils#getContainerLogFile fails to read container - log files from full disks. (zhihai xu via jlowe) - - YARN-3857: Memory leak in ResourceManager with SIMPLE mode. - (mujunchao via zxu) - - YARN-3893. Both RM in active state when Admin#transitionToActive failure - from refeshAll() (Bibin A Chundatt via rohithsharmaks) - - YARN-4103. RM WebServices missing scheme for appattempts logLinks. - (Jonathan Eagles via vvasudeb) - - YARN-4105. Capacity Scheduler headroom for DRF is wrong (Chang Li via - jlowe) - - YARN-4096. App local logs are leaked if log aggregation fails to initialize - for the app. (Jason Lowe via zxu) - - YARN-4136. LinuxContainerExecutor loses info when forwarding - ResourceHandlerException. (Bibin A Chundatt via vvasudev) - - YARN-3697. FairScheduler: ContinuousSchedulingThread can fail to shutdown. - (Zhihai Xu via kasha) - - YARN-4126. RM should not issue delegation tokens in unsecure mode. - (Bibin A Chundatt via jianhe) - - YARN-4158. Remove duplicate close for LogWriter in - AppLogAggregatorImpl#uploadLogsForContainers (Zhihai Xu via jlowe) - - YARN-3433. Jersey tests failing with Port in Use -again. - (Brahma Reddy Battula) - - YARN-3975. WebAppProxyServlet should not redirect to RM page if AHS is - enabled (Mit Desai via jlowe) - - YARN-3624. ApplicationHistoryServer should not reverse the order of the - filters it gets. (Mit Desai via xgong) - - YARN-4180. AMLauncher does not retry on failures when talking to NM. - (adhoot) - - YARN-3619. ContainerMetrics unregisters during getMetrics and leads to - ConcurrentModificationException (Zhihai Xu via jlowe) - - YARN-4209. RMStateStore FENCED state doesnât work due to updateFencedState called - by stateMachine.doTransition. (Zhihai Xu via rohithsharmaks) - - YARN-4041. Slow delegation token renewal can severely prolong RM recovery - (Sunil G via jlowe) - - YARN-2902. Killing a container that is localizing can orphan resources in - the DOWNLOADING state (Varun Saxena via jlowe) - - YARN-4313. Race condition in MiniMRYarnCluster when getting history server - address. (Jian He via xgong) - - YARN-3580. [JDK8] TestClientRMService.testGetLabelsToNodes fails. (Robert Kanter - via junping_du) - - YARN-4312. TestSubmitApplicationWithRMHA fails on branch-2.7 and branch-2.6 - as some of the test cases time out. (Varun Saxena via ozawa) - - YARN-4320. TestJobHistoryEventHandler fails as AHS in MiniYarnCluster no longer - binds to default port 8188. (Varun Saxena via ozawa) - - YARN-4354. Public resource localization fails with NPE. (Jason Lowe via - junping_du) - -Release 2.7.1 - 2015-07-06 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - YARN-3243. CapacityScheduler should pass headroom from parent to children - to make sure ParentQueue obey its capacity limits. (Wangda Tan via jianhe) - - YARN-3489. RMServerUtils.validateResourceRequests should only obtain queue - info once. (Varun Saxena via wangda) - - YARN-3539. Updated timeline server documentation and marked REST APIs evolving. - (Steve Loughran via zjshen) - - YARN-3723. Need to clearly document primaryFilter and otherInfo value type. - (Zhijie Shen via xgong) - - YARN-3711. Documentation of ResourceManager HA should explain configurations - about listen addresses. (Masatake Iwasaki via ozawa) - - OPTIMIZATIONS - - BUG FIXES - - YARN-3497. ContainerManagementProtocolProxy modifies IPC timeout conf - without making a copy. (Jason Lowe via jianhe) - - YARN-2605. [RM HA] Rest api endpoints doing redirect incorrectly. - (Xuan Gong via stevel) - - YARN-3522. Fixed DistributedShell to instantiate TimeLineClient as the - correct user. (Zhijie Shen via jianhe) - - YARN-3351. AppMaster tracking URL is broken in HA. (Anubhav Dhoot via kasha) - - YARN-3382. Some of UserMetricsInfo metrics are incorrectly set to root - queue metrics. (Rohit Agarwal via jianhe) - - YARN-3472. Fixed possible leak in DelegationTokenRenewer#allTokens. - (Rohith Sharmaks via jianhe) - - YARN-3465. Use LinkedHashMap to preserve order of resource requests. - (Zhihai Xu via kasha) - - YARN-3516. killing ContainerLocalizer action doesn't take effect when - private localizer receives FETCH_FAILURE status.(zhihai xu via xgong) - - YARN-3485. FairScheduler headroom calculation doesn't consider - maxResources for Fifo and FairShare policies. (kasha) - - YARN-3301. Fixed the format issue of the new RM attempt web page. - (Xuan Gong via jianhe) - - YARN-3385. Fixed a race-condition in ResourceManager's ZooKeeper based - state-store to avoid crashing on duplicate deletes. (Zhihai Xu via vinodkv) - - YARN-3358. Audit log not present while refreshing Service ACLs. - (Varun Saxena via devaraj) - - YARN-3476. Nodemanager can fail to delete local logs if log aggregation - fails (Rohith via jlowe) - - YARN-3434. Interaction between reservations and userlimit can result in - significant ULF violation (tgraves) - - YARN-3626. On Windows localized resources are not moved to the front - of the classpath when they should be. (Craig Welch via xgong) - - YARN-3457. NPE when NodeManager.serviceInit fails and stopRecoveryStore called. - (Bibin A Chundatt via ozawa) - - YARN-3537. NPE when NodeManager.serviceInit fails and stopRecoveryStore - invoked (Brahma Reddy Battula via jlowe) - - YARN-3601. Fix UT TestRMFailover.testRMWebAppRedirect. (Weiwei Yang via xgong) - - YARN-3677. Fix findbugs warnings in yarn-server-resourcemanager. - (Vinod Kumar Vavilapalli via ozawa) - - YARN-3681. yarn cmd says "could not find main class 'queue'" in windows. - (Craig Welch and Varun Saxena via xgong) - - YARN-3609. Load node labels from storage inside RM serviceStart. (Wangda - Tan via jianhe) - - YARN-3694. Fix dead link for TimelineServer REST API. - (Jagadesh Kiran N via aajisaka) - - YARN-3646. Applications are getting stuck some times in case of retry - policy forever. (Raju Bairishetti via devaraj) - - YARN-3675. FairScheduler: RM quits when node removal races with - continuous-scheduling on the same node. (Anubhav Dhoot via kasha) - - YARN-3701. Isolating the error of generating a single app report when - getting all apps from generic history service. (Zhijie Shen via xgong) - - YARN-2238. filtering on UI sticks even if I move away from the page. - (Jian He via xgong) - - YARN-3686. CapacityScheduler should trim default_node_label_expression. - (Sunil G via wangda) - - YARN-3764. CapacityScheduler should forbid moving LeafQueue from one parent - to another. (Wangda Tan via jianhe) - - YARN-3804. Both RM are on standBy state when kerberos user not in yarn.admin.acl - (Varun Saxena via xgong) - - YARN-3842. NMProxy should retry on NMNotYetReadyException. - (Robert Kanter via kasha) - - YARN-3809. Failed to launch new attempts because - ApplicationMasterLauncher's threads all hang (Jun Gong via jlowe) - -Release 2.7.0 - 2015-04-20 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - YARN-2179. [YARN-1492] Initial cache manager structure and context. - (Chris Trezzo via kasha) - - YARN-2180. [YARN-1492] In-memory backing store for cache manager. - (Chris Trezzo via kasha) - - YARN-2183. [YARN-1492] Cleaner service for cache manager. - (Chris Trezzo and Sangjin Lee via kasha) - - YARN-2186. [YARN-1492] Node Manager uploader service for cache manager. - (Chris Trezzo and Sangjin Lee via kasha) - - YARN-2236. [YARN-1492] Shared Cache uploader service on the Node - Manager. (Chris Trezzo and Sangjin Lee via kasha) - - YARN-2188. [YARN-1492] Client service for cache manager. - (Chris Trezzo and Sangjin Lee via kasha) - - YARN-2189. [YARN-1492] Admin service for cache manager. - (Chris Trezzo via kasha) - - YARN-2765. Added leveldb-based implementation for RMStateStore. (Jason Lowe - via jianhe) - - YARN-2203. [YARN-1492] Web UI for cache manager. (Chris Trezzo via kasha) - - YARN-2738. [YARN-2574] Add FairReservationSystem for FairScheduler. - (Anubhav Dhoot via kasha) - - YARN-2881. [YARN-2574] Implement PlanFollower for FairScheduler. - (Anubhav Dhoot via kasha) - - YARN-2427. Added the API of moving apps between queues in RM web services. - (Varun Vasudev via zjshen) - - YARN-2217. [YARN-1492] Shared cache client side changes. - (Chris Trezzo via kasha) - - YARN-2616 [YARN-913] Add CLI client to the registry to list, view - and manipulate entries. (Akshay Radia via stevel) - - YARN-2994. Document work-preserving RM restart. (Jian He via ozawa) - - YARN-2786. Created a yarn cluster CLI and seeded with one command for listing - node-labels collection. (Wangda Tan via vinodkv) - - IMPROVEMENTS - - YARN-3005. [JDK7] Use switch statement for String instead of if-else - statement in RegistrySecurity.java (Kengo Seki via aajisaka) - - YARN-2950. Change message to mandate, not suggest JS requirement on UI. - (Dustin Cote via harsh) - - YARN-2891. Failed Container Executor does not provide a clear error - message. (Dustin Cote via harsh) - - YARN-1979. TestDirectoryCollection fails when the umask is unusual. - (Vinod Kumar Vavilapalli and Tsuyoshi OZAWA via junping_du) - - YARN-2641. Decommission nodes on -refreshNodes instead of next - NM-RM heartbeat. (Zhihai Xu via kasha) - - YARN-2742. FairSchedulerConfiguration should allow extra spaces - between value and unit. (Wei Yan via kasha) - - YARN-2712. TestWorkPreservingRMRestart: Augment FS tests with - queue and headroom checks. (Tsuyoshi Ozawa via kasha) - - YARN-2735. diskUtilizationPercentageCutoff and diskUtilizationSpaceCutoff - are initialized twice in DirectoryCollection. (Zhihai Xu via kasha) - - YARN-570. Time strings are formated in different timezone. - (Akira Ajisaka and Peng Zhang via kasha) - - YARN-2780. Log aggregated resource allocation in rm-appsummary.log (Eric - Payne via jlowe) - - YARN-2690. [YARN-2574] Make ReservationSystem and its dependent classes - independent of Scheduler type. (Anubhav Dhoot via kasha) - - YARN-2157. Added YARN metrics in the documentaion. (Akira AJISAKA via - jianhe) - - YARN-2802. ClusterMetrics to include AM launch and register delays. - (Zhihai Xu via kasha) - - YARN-2375. Allow enabling/disabling timeline server per framework. - (Mit Desai via jeagles) - - YARN-2604. Scheduler should consider max-allocation-* in conjunction - with the largest node. (Robert Kanter via kasha) - - YARN-2679. Add metric for container launch duration. (Zhihai Xu via kasha) - - YARN-2669. FairScheduler: queue names shouldn't allow periods - (Wei Yan via Sandy Ryza) - - YARN-2404. Removed ApplicationAttemptState and ApplicationState class in - RMStateStore. (Tsuyoshi OZAWA via jianhe) - - YARN-2165. Added the sanity check for the numeric configuration values of - the timeline service. (Vasanth kumar RJ via zjshen) - - YARN-2907. SchedulerNode#toString should print all resource detail instead - of only memory. (Rohith via junping_du) - - YARN-2136. Changed RMStateStore to ignore store opearations when fenced. - (Varun Saxena via jianhe) - - YARN-1156. Enhance NodeManager AllocatedGB and AvailableGB metrics - for aggregation of decimal values. (Tsuyoshi OZAWA via junping_du) - - YARN-2056. Disable preemption at Queue level (Eric Payne via jlowe) - - YARN-2762. Fixed RMAdminCLI to trim and check node-label related arguments - before sending to RM. (Rohith Sharmaks via jianhe) - - YARN-2972. DelegationTokenRenewer thread pool never expands. (Jason Lowe - via junping_du) - - YARN-2949. Add documentation for CGroups (Varun Vasudev via junping_du) - - YARN-2970. NodeLabel operations in RMAdmin CLI get missing in help command. - (Varun Saxena via junping_du) - - YARN-2837. Support TimeLine server to recover delegation token when - restarting. (Zhijie Shen via jianhe) - - YARN-2993. Several fixes (missing acl check, error log msg ...) and some - refinement in AdminService. (Yi Liu via junping_du) - - YARN-2943. Added node-labels page on RM web UI. (Wangda Tan via jianhe) - - YARN-2998. Abstract out scheduler independent PlanFollower components. - (Anubhav Dhoot via kasha) - - YARN-2360. Fair Scheduler: Display dynamic fair share for queues on the - scheduler page. (Ashwin Shankar and Wei Yan via kasha) - - YARN-2880. Added a test to make sure node labels will be recovered - if RM restart is enabled. (Rohith Sharmaks via jianhe) - - YARN-2996. Improved synchronization and I/O operations of FS- and Mem- - RMStateStore. (Yi Liu via zjshen) - - YARN-2956. Added missing links in YARN documentation. (Masatake Iwasaki via - jianhe) - - YARN-2957. Create unit test to automatically compare YarnConfiguration - and yarn-default.xml. (rchiang via rkanter) - - YARN-2643. Don't create a new DominantResourceCalculator on every - FairScheduler.allocate call. (kasha via rkanter) - - YARN-3019. Make work-preserving-recovery the default mechanism for RM - recovery. (Jian He via junping_du) - - YARN-2807. Option "--forceactive" not works as described in usage of - "yarn rmadmin -transitionToActive". (Masatake Iwasaki via xgong) - - YARN-2984. Metrics for container's actual memory usage. (kasha) - - YARN-2800. Remove MemoryNodeLabelsStore and add a way to enable/disable - node labels feature. (Wangda Tan via ozawa) - - YARN-3086. Make NodeManager memory configurable in MiniYARNCluster. - (Robert Metzger via ozawa) - - YARN-2897. CrossOriginFilter needs more log statements (Mit Desai via - jeagles) - - YARN-3028. Better syntax for replaceLabelsOnNode in RMAdmin CLI - (Rohith Sharmaks via wangda) - - YARN-2932. Add entry for "preemptable" status (enabled/disabled) to - scheduler web UI and queue initialize/refresh logging. - (Eric Payne via wangda) - - YARN-3108. ApplicationHistoryServer doesn't process -D arguments (Chang Li - via jeagles) - - YARN-2808. Made YARN CLI list attemptâs finished containers of a running - application. (Naganarasimha G R via zjshen) - - YARN-3085. Application summary should include the application type (Rohith - via jlowe) - - YARN-3022. Expose Container resource information from NodeManager for - monitoring (adhoot via ranter) - - YARN-3075. NodeLabelsManager implementation to retrieve label to node - mapping (Varun Saxena via wangda) - - YARN-1393. SLS: Add how-to-use instructions. (Wei Yan via kasha) - - YARN-1723. AMRMClientAsync missing blacklist addition and removal - functionality. (Bartosz Åugowski via sseth) - - YARN-3123. Made YARN CLI show a single completed container even if the app - is running. (Naganarasimha G R via zjshen) - - YARN-1582. Capacity Scheduler: add a maximum-allocation-mb setting per - queue (Thomas Graves via jlowe) - - YARN-1904. Ensure exceptions thrown in ClientRMService & - ApplicationHistoryClientService are uniform when application-attempt is - not found. (zjshen via acmurthy) - - YARN-3144. Configuration for making delegation token failures to timeline - server not-fatal (Jonathan Eagles via jlowe) - - YARN-3155. Refactor the exception handling code for TimelineClientImpl's - retryOn method (Li Lu via wangda) - - YARN-3100. Made YARN authorization pluggable. (Jian He via zjshen) - - YARN-2683. [YARN-913] registry config options: document and move to - core-default. (stevel) - - YARN-1237. Description for yarn.nodemanager.aux-services in - yarn-default.xml is misleading. (Brahma Reddy Battula via ozawa) - - YARN-3157. Refactor the exception handling in ConverterUtils#to*Id. - (Bibin A Chundatt via ozawa) - - YARN-3147. Clean up RM web proxy code. (Steve Loughran via xgong) - - YARN-2079. Recover NonAggregatingLogHandler state upon nodemanager - restart. (Jason Lowe via junping_du) - - YARN-3158. Correct log messages in ResourceTrackerService. - (Varun Saxena via xgong) - - YARN-3179. Update use of Iterator to Iterable in RMAdminCLI and - CommonNodeLabelsManager. (Ray Chiang via xgong) - - YARN-3182. Cleanup switch statement in ApplicationMasterLauncher#handle(). - (Ray Chiang via ozawa) - - YARN-3203. Correct a log message in AuxServices. (Brahma Reddy Battula - via ozawa) - - YARN-1299. Improve a log message in AppSchedulingInfo by adding application - id. (Ashutosh Jindal and Devaraj K via ozawa) - - YARN-1514. Utility to benchmark ZKRMStateStore#loadState for RM HA. - (Tsuyoshi OZAWA via jianhe) - - YARN-3076. Add API/Implementation to YarnClient to retrieve label-to-node - mapping. (Varun Saxena via wangda) - - YARN-2799. Cleanup TestLogAggregationService based on the change in YARN-90. - (Zhihai Xu via junping_du) - - YARN-3237. AppLogAggregatorImpl fails to log error cause. - (Rushabh S Shah via xgong) - - YARN-3236. Cleanup RMAuthenticationFilter#AUTH_HANDLER_PROPERTY. - (zhihai xu via xgong) - - YARN-2797. TestWorkPreservingRMRestart should use ParametrizedSchedulerTestBase - (Karthik Kambatla via xgong) - - YARN-2797. Add -help to yarn logs and nodes CLI command. - (Jagadesh Kiran N via devaraj) - - YARN-3217. Remove httpclient dependency from hadoop-yarn-server-web-proxy. - (Brahma Reddy Battula via ozawa). - - YARN-3255. RM, NM, JobHistoryServer, and WebAppProxyServer's main() - should support generic options. (shv) - - YARN-2820. Retry in FileSystemRMStateStore when FS's operations fail - due to IOException. (Zhihai Xu via ozawa) - - YARN-3262. Surface application outstanding resource requests table - in RM web UI. (Jian He via wangda) - - YARN-3281. Added RMStateStore to StateMachine visualization list. - (Chengbing Liu via jianhe) - - YARN-3272. Surface container locality info in RM web UI. - (Jian He via wangda) - - YARN-3122. Metrics for container's actual CPU usage. - (Anubhav Dhoot via kasha) - - YARN-2190. Added CPU and memory limit options to the default container - executor for Windows containers. (Chuan Liu via jianhe) - - YARN-3296. Mark ResourceCalculatorProcessTree class as Public for configurable - resource monitoring. (Hitesh Shah via junping_du) - - YARN-3187. Documentation of Capacity Scheduler Queue mapping based on user - or group. (Gururaj Shetty via jianhe) - - YARN-2854. Updated the documentation of the timeline service and the generic - history service. (Naganarasimha G R via zjshen) - - YARN-2777. Mark the end of individual log in aggregated log. - (Varun Saxena via xgong) - - YARN-3273. Improve scheduler UI to facilitate scheduling analysis and - debugging. (Rohith Sharmaks via jianhe) - - OPTIMIZATIONS - - YARN-2990. FairScheduler's delay-scheduling always waits for node-local and - rack-local delays, even for off-rack-only requests. (kasha) - - BUG FIXES - - YARN-3071. Remove invalid char from sample conf in doc of FairScheduler. - (Masatake Iwasaki via aajisaka) - - YARN-2254. TestRMWebServicesAppsModification should run against both - CS and FS. (Zhihai Xu via kasha) - - YARN-2713. "RM Home" link in NM should point to one of the RMs in an - HA setup. (kasha) - - YARN-2857. ConcurrentModificationException in ContainerLogAppender - (Mohammad Kamrul Islam via jlowe) - - YARN-2432. RMStateStore should process the pending events before close. - (Varun Saxena via jianhe) - - YARN-1703. Fixed ResourceManager web-proxy to close connections correctly. - (Rohith Sharma via vinodkv) - - YARN-2870. Updated the command to run the timeline server in the document. - (Masatake Iwasaki via zjshen) - - YARN-2878. Fix DockerContainerExecutor.apt.vm formatting. (Abin Shahab via - jianhe) - - YARN-2315. FairScheduler: Set current capacity in addition to capacity. - (Zhihai Xu via kasha) - - YARN-2697. Remove useless RMAuthenticationHandler. (Haosong Huang via zjshen) - - YARN-2461. Fix PROCFS_USE_SMAPS_BASED_RSS_ENABLED property in - YarnConfiguration. (rchiang via rkanter) - - YARN-2869. CapacityScheduler should trim sub queue names when parse - configuration. (Wangda Tan via jianhe) - - YARN-2927. [YARN-1492] InMemorySCMStore properties are inconsistent. - (Ray Chiang via kasha) - - YARN-2931. PublicLocalizer may fail until directory is initialized by - LocalizeRunner. (Anubhav Dhoot via kasha) - - YARN-2930. Fixed TestRMRestart#testRMRestartRecoveringNodeLabelManager - intermittent failure. (Wangda Tan via jianhe) - - YARN-2924. Fixed RMAdminCLI to not convert node labels to lower case. - (Wangda Tan via jianhe) - - YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in - SchedulerApplicationAttempt ctor. (devaraj) - - YARN-2912 Jersey Tests failing with port in use. (varun saxena via stevel) - - YARN-2356. yarn status command for non-existent application/application - attempt/container is too verbose. (Sunil G via devaraj) - - YARN-2914. [YARN-1492] Potential race condition in Singleton implementation of - SharedCacheUploaderMetrics, CleanerMetrics, ClientSCMMetrics. (Varun Saxena via kasha) - - YARN-2945. FSLeafQueue#assignContainer - document the reason for using both write and - read locks. (Tsuyoshi Ozawa via kasha) - - YARN-2944. InMemorySCMStore can not be instantiated with ReflectionUtils#newInstance. - (Chris Trezzo via kasha) - - YARN-2675. containersKilled metrics is not updated when the container is killed - during localization. (Zhihai Xu via kasha) - - YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) - - YARN-2977. Fixed intermittent TestNMClient failure. - (Junping Du via ozawa) - - YARN-2939. Fix new findbugs warnings in hadoop-yarn-common. (Li Lu via junping_du) - - YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Li Lu - via junping_du) - - YARN-2937. Fixed new findbugs warnings in hadoop-yarn-nodemanager. (Varun Saxena - via zjshen) - - YARN-2946. Fixed potential deadlock in RMStateStore. (Rohith Sharmaks via - jianhe) - - YARN-2988. Graph#save() may leak file descriptors. (Ted Yu via ozawa) - - YARN-2938. Fixed new findbugs warnings in hadoop-yarn-resourcemanager and - hadoop-yarn-applicationhistoryservice. (Varun Saxena via zjshen) - - YARN-2987. Fixed ClientRMService#getQueueInfo to check against queue and - app ACLs. (Varun Saxena via jianhe) - - YARN-2991. Fixed DrainDispatcher to reuse the draining code path in - AsyncDispatcher. (Rohith Sharmaks via zjshen) - - YARN-2958. Made RMStateStore not update the last sequence number when updating the - delegation token. (Varun Saxena via zjshen) - - YARN-2230. Fixed few configs description in yarn-default.xml. (Vijay Bhat - via jianhe) - - YARN-3010. Fixed findbugs warning in AbstractYarnScheduler. (Yi Liu via - jianhe) - - YARN-2936. Changed YARNDelegationTokenIdentifier to set proto fields on - getProto method. (Varun Saxena via jianhe) - - YARN-3014. Replaces labels on a host should update all NM's labels on that - host. (Wangda Tan via jianhe) - - YARN-3027. Scheduler should use totalAvailable resource from node instead of - availableResource for maxAllocation. (adhoot via rkanter) - - YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs. - (Zhijie Shen via jianhe) - - YARN-3064. TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync - failure with allocation timeout. (Jian He via junping_du) - - YARN-2815. Excluded transitive dependency of JLine in hadoop-yarn-server-common. - (Ferdinand Xu via zjshen) - - YARN-3070. TestRMAdminCLI#testHelp fails for transitionToActive command. - (Contributed by Junping Du) - - YARN-3015. yarn classpath command should support same options as hadoop - classpath. (Contributed by Varun Saxena) - - YARN-2933. Capacity Scheduler preemption policy should only consider capacity - without labels temporarily. (Mayank Bansal via wangda) - - YARN-2731. Fixed RegisterApplicationMasterResponsePBImpl <TRUNCATED>
