Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.
The "Release-2.6.1-Working-Notes" page has been changed by VinodKumarVavilapalli: https://wiki.apache.org/hadoop/Release-2.6.1-Working-Notes Comment: Creating the list of ordered commits for 2.6.1 New page: Ordered list of commits to cherrypick ||946463efefec9031cacb21d5a5367acd150ef904 ||HDFS-7213. processIncrementalBlockReport performance degradation. Contributed by Eric Payn || || ||842a54a5f66e76eb79321b66cc3b8820fe66c5cd ||HDFS-7235. DataNode#transferBlock should report blocks that don't exist using reportBadBlo || || ||8bfef590295372a48bd447b1462048008810ee17 ||HDFS-7263. Snapshot read can reveal future bytes for appended files. Contributed by Tao Lu || || ||ec2621e907742aad0264c5f533783f0f18565880 ||HDFS-7035. Make adding a new data directory to the DataNode an atomic operation and improv || || ||beb184ac580b0d89351a3f3a7201da34a26db1c1 ||YARN-2856. Fixed RMAppImpl to handle ATTEMPT_KILLED event at ACCEPTED state on app recover || || ||ad140d1fc831735fb9335e27b38d2fc040847af1 ||YARN-2816. NM fail to start with NPE during container recovery. Contributed by Zhihai Xu ( || || ||242fd0e39ad1c5d51719cd0f6c197166066e3288 ||YARN-2414. RM web UI: app page will crash if app is failed before any attempt has been cre || || ||2e15754a92c6589308ccbbb646166353cc2f2456 ||HDFS-7225. Remove stale block invalidation work when DN re-registers with different UUID. || || ||db31ef7e7f55436bbf88c6d93e2273c4463ca9f0 ||YARN-2865. Fixed RM to always create a new RMContext when transtions from StandBy to Activ || || ||8d8eb8dcec94e92d94eedef883cdece8ba333087 ||HDFS-7425. NameNode block deletion logging uses incorrect appender. Contributed by Chris N || || ||946df98dce18975e37a6a14744ca7a5429f019ce ||HDFS-4882. Prevent the Namenode's LeaseManager from looping forever in checkLeases (Ravi P || || ||ae35b0e14d3438237f4b5d3b5d5268d45e549846 ||YARN-2906. CapacitySchedulerPage shows HTML tags for a queue's Active Users. Contributed b || || ||f6d1bf5ed1cf647d82e676df15587de42b1faa42 ||HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification pipe is full (zhao || || ||38ea1419f60d2b8176dba4931748f1f0e52ca84e ||YARN-2905. AggregatedLogsBlock page can infinitely loop if the aggregated log file is corr || || ||d21ef79707a0f32939d9a5af4fed2d9f5fe6f2ec ||YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Co || || ||06552a15d5172a2b0ad3d61aa7f9a849857385aa ||HDFS-7446. HDFS inotify should have the ability to determine what txid it has read up to ( || || ||d6f3d4893d750f19dd8c539fe28eecfab2a54576 ||YARN-2894. Fixed a bug regarding application view acl when RM fails over. Contributed by R || || ||25be97808b99148412c0efd4d87fc750db4d6607 ||YARN-2874. Dead lock in DelegationTokenRenewer which blocks RM to execute any further apps || || ||dabdd2d746d1e1194c124c5c7fe73fcc025e78d2 ||HADOOP-11343. Overflow is not properly handled in caclulating final iv for AES CTR. Contri || || ||deaa172e7a2ab09656cc9eb431a3e68a73e0bd96 ||HADOOP-11368. Fix SSLFactory truststore reloader thread leak in KMSClientProvider. Contrib || || ||a037d6030b5ae9422fdb265f5e4880d515be9e37 ||HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes (Noah Lorang via || || ||1986ea8dd223267ced3e3aef69980b46e2fef740 ||YARN-2910. FSLeafQueue can throw ConcurrentModificationException. (Wilfred Spiegelenburg v || || ||e4f9ddfdbcdab1eacf568e8689448ffc10bbc2aa ||HDFS-7503. Namenode restart after large deletions can cause slow processReport (Arpit Agar || || ||41f0d20fcb4fec5b932b8947a44f93345205222c ||YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contrib || || ||b521d91c0f5b6d114630b6727f6a01db56dba4f1 ||HADOOP-11238. Update the NameNode's Group Cache in the background when possible (Chris Li || || ||7c7bccfc31ede6f4afc043d81c8204046148c02e ||MAPREDUCE-6166. Reducers do not validate checksum of map outputs when fetching directly to || || ||2d832ad2eb87e0ce7c50899c54d05f612666518a ||YARN-2964. FSLeafQueue#assignContainer - document the reason for using both write and read || || ||dda1fc169db2e69964cca746be4ff8965eb8b56f ||HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via Colin P. McCabe) (che || || ||173664d70f0ed3b1852b6703d32e796778fb1c78 ||YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie). Contributed by || || ||bcaf15e2fa94db929b8cd11ed7c07085161bf950 ||HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are pre || || ||9180d11b3bbb2a49127d5d25f53b38c5113bf7ea ||YARN-2952. Fixed incorrect version check in StateStore. Contributed by Rohith Sharmaks (ch || || ||8b398a66ca3728f47363fc8b2fcf7e556e6bbf5a ||YARN-2340. Fixed NPE when queue is stopped during RM restart. Contributed by Rohith Sharma || || ||ca0349b87ab1b2d0d2b9dc93de7806d26713165c ||YARN-2992. ZKRMStateStore crashes due to session expiry. Contributed by Karthik Kambatla ( || || ||c116743bdda2b1792bf872020a5e2b14d772ac60 ||YARN-2922. ConcurrentModificationException in CapacityScheduler's LeafQueue. Contributed b || || ||e7e6173049adca2a2ae0e1231adcaca8168bec27 ||YARN-2997. Fixed NodeStatusUpdater to not send alreay-sent completed container statuses on || || ||f0acb7c2a284db61640efee15a1648c6c26d24f5 ||HDFS-7579. Improve log reporting during block report rpc failure. Contributed by Charles L || || ||4b589e7cfa27bd042e228bbbcf1c3b75b2aeaa57 ||HDFS-7182. JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. || || ||75e4e55e12b2faa521af7c23fddcba06a9ce661d ||HDFS-7596. NameNode should prune dead storages from storageMap. Contributed by Arpit Agarw || || ||33534a0c9aef5024aa6f340e7ee24930c8fa8ed5 ||HDFS-7533. Datanode sometimes does not shutdown on receiving upgrade shutdown command. Con || || ||2e4df8710435c8362506fe944a935e74ad5919c0 ||HADOOP-11350. The size of header buffer of HttpServer is too small when HTTPS is enabled. || || ||1d9d166c0beb56aa45e65f779044905acff25d88 ||HDFS-7575. Upgrade should generate a unique storage ID for each volume. (Contributed by Ar || || ||7b69719455a1a374c9417417ef0c8d7ba6bf593f ||HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy user. Contribute || || ||24f0d56afb853a371f905c4569d82d09d89cf13e ||HDFS-7676. Fix TestFileTruncate to avoid bug of HDFS-7611. Contributed by Konstantin Shvac || || ||8100c8a68c32978a177af9a3e6639f6de533886d ||YARN-3011. Possible IllegalArgumentException in ResourceLocalizationService might lead NM || || ||12522fd9cbd8da8c040a5b7bb71fcdaa256daf89 ||YARN-3103. AMRMClientImpl does not update AMRM token properly. Contributed by Jason Lowe ( || || ||e8300957a75353849219d616cfac08b08c182db2 ||HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks in the blocksMap || || ||43b3b43cea1f620ce66521bcc1c4b6aec264aa9a ||MAPREDUCE-6230. Fixed RMContainerAllocator to update the new AMRMToken service name proper || || ||c428d303f67bef3a7df12153947c6b0199a0938b ||HDFS-7707. Edit log corruption due to delayed block removal again. Contributed by Yongjun || || ||5807afed0a4f08b6b2acd88f424bace506e03707 ||HDFS-7733. NFS: readdir/readdirplus return null directory attribute on failure. (Contribut || || ||3d36d4737c160d7dc8829e9dd6b801ef6726c0c0 ||HADOOP-11506. Configuration variable expansion regex expensive for long values. (Gera Sheg || || ||b1aad1d941c5855e5dd5f5e819e9218d8fc266ee ||MAPREDUCE-6237. Multiple mappers with DBInputFormat don't work because of reusing conectio || || ||61466809552f96a83aa19446d4d59cecd0d2cad5 ||YARN-3094. Reset timer for liveness monitors after RM recovery. Contributed by Jun Gong (c || || ||a1bf7aecf7d018c5305fa3bd7a9e3ef9af3155c1 ||HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause DataNode to registe || || ||fd75b8c9cadd069673afc80a0fc5661d779897bd ||YARN-2246. Made the proxy tracking URL always be http(s)://proxy addr:port/proxy/<appId> t || || ||ba18adbb27c37a8fa92223a412ce65eaa462d18b ||YARN-3207. Secondary filter matches entites which do not have the key being filtered for. || || ||6c01e586198a3c3ebaa7561778c124ae62553246 ||HADOOP-11295. RPC Server Reader thread can't shutdown if RPCCallQueue is full. Contributed || || ||b9157f92fc3e008e4f3029f8feeaf6acb52eb76f ||HDFS-7788. Post-2.6 namenode may not start up with an image containing inodes created with || || ||187e081d5a8afe1ddfe5d7b5e7de7a94512aa53e ||HADOOP-11604. Prevent ConcurrentModificationException while closing domain sockets during || || ||fefeba4ac8bed44ce2dd0d3c4f0a99953ff8d4df ||YARN-3238. Connection timeouts to nodemanagers are retried at multiple levels. Contributed || || ||657a6e389b3f6eae43efb11deb6253c3b1255a51 ||HDFS-7009. Active NN and standby NN have different live nodes. Contributed by Ming Ma. (c || || ||8346427929d2b6f2fd3fa228d77f3cf596ef0306 ||HDFS-7831. Fix the starting index and end condition of the loop in FileDiffList.findEarlie || || ||efb7e287f45c6502f293456034a37d9209a917be ||HDFS-7763. fix zkfc hung issue due to not catching exception in a corner case. Contributed || || ||257087417e424e628f090b6b648ccb3b9c880250 ||YARN-3239. WebAppProxy does not support a final tracking url which has query fragments and || || ||12fe3afcd3ea224909bd125f5c6a7410bf2601b0 ||HDFS-7843. A truncated file is corrupted after rollback from a rolling upgrade. || || ||6090f51725e2b44d794433ed72a1901fae2ba7e3 ||HDFS-7871. NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contrib || || ||888a44563819ba910dc3cc10d10ee0fb8f05db61 ||YARN-3222. Fixed NPE on RMNodeImpl#ReconnectNodeTransition when a node is reconnected with || || ||721d7b574126c4070322f70ec5b49a7b8558a4c7 ||YARN-3231. FairScheduler: Changing queueMaxRunningApps interferes with pending jobs. (Siqi || || ||0d62e948877e5d50f1b6fbe735a94ac6da5ff472 ||YARN-3242. Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving events for || || ||b569c3ab1cb7e328dde822f6b2405d24b9560e3a ||HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream should be non static. || || ||a5f3fb4dc14503bf7c454a48cf954fb0d6710de2 ||YARN-1809. Synchronize RM and TimeLineServer Web-UIs. Contributed by Zhijie Shen and Xuan || || ||994dadb9ba0a3b87b6548e6e0801eadd26554d55 ||HDFS-7885. Datanode should not trust the generation stamp provided by client. Contributed || || ||56c2050ab7c04e9741bcba9504b71e5a54d09eea ||YARN-3227. Timeline renew delegation token fails when RM user's TGT is expired. Contribute || || ||a94d23762e2cf4211fe84661eb67504c7072db49 ||YARN-3287. Made TimelineClient put methods do as the correct login context. Contributed by || || ||eefca23e8c5e474de1e25bf2ec8a5b266bbe8cfe ||HDFS-7830. DataNode does not release the volume lock when adding a volume fails. (Lei Xu v || || ||813c93cb250d6d556604fe98845b979970bd5e18 ||HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt synchronization. (Se || || ||44aedad5ddc8069a6dba3eaf66ed54d612b21208 ||YARN-3267. Timelineserver applies the ACL rules after applying the limit on the number of || || ||af80a98ace50934284dde417efc802bb094c8b4e ||HDFS-7926. NameNode implementation of ClientProtocol.truncate(..) is not idempotent. Contr || || ||5a5b2446485531f12d37f3d4ca791672b9921872 ||HDFS-7587. Edit log corruption can happen if append fails with a quota violation. Contribu || || ||219eb22c1571f76df32967a930049d983cbf5024 ||HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade starts (Zhe Zha || || ||90164ffd84f6ef56e9f8f99dcc7424a8d115dbae ||HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu) || || ||8e142d27cbddfa1a1c83c5f8752bd14ac0a13612 ||YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. (Brahma Reddy || || ||cbdcdfad6de81e17fb586bc2a53b37da43defd79 ||YARN-3393. Getting application(s) goes wrong when app finishes before starting the attempt || || ||fe693b72dec703ecbf4ab3919d61d06ea8735a9e ||HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp provided by t || || ||2f46ee50bd4efc82ba3d30bd36f7637ea9d9714e ||HDFS-7960. The full block report should prune zombie storages even if they're not empty. C || || ||c4cedfc1d601127430c70ca8ca4d4e2ee2d1003d ||HDFS-7742. Favoring decommissioning node for replication can cause a block to stay underre || || ||beb0fd0d601aff0ba993c2d48b83fe52edfb9065 ||HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infini || || ||cacadea632f7ab6fe4fdb1432e1a2c48e8ebd55f ||MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal to a reducer. Contri || || ||a827089905524e10638c783ba908a895d621911d ||HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very || || ||f0324738c9db4f45d2b1ec5cfb46c5f2b7669571 ||HDFS-8072. Reserved RBW space is not released if client terminates while writing block. (A || || ||7e622076d41a85fc9a8600fb270564a085f5cd83 ||HDFS-8046. Allow better control of getContentSummary. Contributed by Kihwal Lee. (cherry p || || ||1ff3fd33ed6f2ac09c774cc42b0107c5dbd9c19d ||YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token renewal of app || || ||3de0bf9a35094f3c2ac216992d861729251b6a3d ||MAPREDUCE-6300. Task list sort by task id broken. Contributed by Siqi Li. (cherry picked || || ||38b031d6bab8527698bd186887d301bd6a63cf01 ||HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to finalize upgrade. Con || || ||e7cbecddc3e7ca5386c71aa4deb67f133611415c ||YARN-3493. RM fails to come up with error "Failed to load/recover state" when mem settings || || ||3316cd4357ff6ccc4c76584813092adb1c2b4d43 ||YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue ( || || ||756c2542930756fef1cbff82056b418070f8d55f ||MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options which is a regressi || || ||961051e569151d68d90f91055c8678c034c20207 ||HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infini || || ||1544c63602089b690e850e0e30af4589513a2371 ||HADOOP-11812. Implement listLocatedStatus for ViewFileSystem to speed up split calculation || || ||a6a5d1d6b5ee76c829ba7b54a4ad619f7b986681 ||HADOOP-11730. Regression: s3n read failure recovery broken. (Takenori Sato via stevel) || || ||788b76761d5dfadf688406d50169e95401fe5d33 ||HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there is an I/O error || || ||4045c41afe440b773d006e962bf8a5eae3fdc284 ||YARN-3464. Race condition in LocalizerRunner kills localizer before localizing all resourc || || ||32dc13d907a416049bdb7deff429725bd6dbcb49 ||MAPREDUCE-6324. Fixed MapReduce uber jobs to not fail the udpate of AM-RM tokens when they || || ||58970d69de8a1662e4548cd6d4ca460dd70562f8 ||HADOOP-11491. HarFs incorrectly declared as requiring an authority. (Brahma Reddy Battula || || ||87c2d915f1cc799cb4020c945c04d3ecb82ee963 ||MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge. Contributed by || || ||01bdfd794cf460ae0a399649eaae54676d101214 ||HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infini || || ||e68e8b3b5cff85bfd8bb5b00b9033f63577856d6 ||HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. (sure || || ||4e1f2eb3955a97a70cf127dc97ae49201a90f5e0 ||HDFS-7980. Incremental BlockReport will dramatically slow down namenode startup. Contribu || || ||d817fbb34d6e34991c6e512c20d71387750a98f4 ||YARN-2918. RM should not fail on startup if queue's configured labels do not exist in clus || || ||802a5775f3522c57c60ae29ecb9533dbbfecfe76 ||HDFS-7894. Rolling upgrade readiness is not updated in jmx until query command is issued. || || ||f264a5aeede7e144af11f5357c7f901993de8e12 ||HDFS-8245. Standby namenode doesn't process DELETED_BLOCK if the addblock request is in ed || || ||50778f9d458443c2cbfead4502df4e1204c4d567 ||HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infini || || ||fb5b0ebb459cc8812084090a7ce7ac29e2ad147c ||MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between copySucceeded() in || || ||a81ad814610936a02e55964fbe08f7b33fe29b23 ||YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when exceptions happen in || || ||802676e1be350785d8c0ad35f6676eeb85b2467b ||YARN-3526. ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster. Cont || || ||536b9ee6d6e5b8430fda23cbdcfd859c299fa8ad ||HDFS-8404. Pending block replication can get stuck using older genstamp. Contributed by Na || || ||d3193fd1d7395bf3e7c8dfa70d1aec08b0f147e6 ||Move YARN-2918 from 2.8.0 to 2.7.1 (cherry picked from commit 03f897fd1a3779251023bae3582 || || ||839f81a6326b2f8b3d5183178382c1551b0bc259 ||YARN-3700. Made generic history service load a number of latest applications according to || || ||25db34127811fbadb9a698fa3a76e24d426fb0f6 ||HDFS-8431. hdfs crypto class not found in Windows. Contributed by Anu Engineer. (cherry p || || ||33648268ce0f79bf51facafa3d151612e3d00ddb ||HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop. Contr || || ||17fb442a4c4e43105374c97fccd68dd966729a19 ||HDFS-7609. Avoid retry cache collision when Standby NameNode loading edits. Contributed by || || ||4fee8b320276bac86278e1ae0a3397592a78aa18 ||YARN-2900. Application (Attempt and Container) Not Found in AHS results in Internal Server || || ||a3734f67d35e714690ecdf21d80bce8a355381e3 ||YARN-3725. App submission via REST API is broken in secure mode due to Timeline DT service || || ||17e369511dd18d6e25e10ddb0cd9511f7ec8f469 ||HDFS-8486. DN startup may cause severe data loss (Daryn Sharp via Colin P. McCabe) (cher || || ||066e45bcb667bb0c37ef70fd297b24e4f26383eb ||HDFS-8270. create() always retried with hardcoded timeout when file already exists with op || || ||752caa95a40d899e1bf98bc907e91aec2bb57073 ||YARN-3585. NodeManager cannot exit on SHUTDOWN event triggered and NM recovery is enabled. || || ||78d626fa892415023827e35ad549636e2a83275d ||YARN-3733. Fix DominantRC#compare() does not work as expected if cluster resource is empty || || ||344b7509153cdd993218cd5104c7e5c07cd35d3c ||Add missing test file of YARN-3733 (cherry picked from commit 405bbcf68c32d8fd8a83e46e686 || || ||80697e4f324948ec32b4cad3faccba55287be652 ||HADOOP-7139. Allow appending to existing SequenceFiles (Contributed by kanaka kumar avvaru || || ||cbd11681ce8a51d187d91748b67a708681e599de ||HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve || || ||15b1800b1289d239cbebc5cfd66cfe156d45a2d3 ||YARN-3832. Resource Localization fails on a cluster due to existing cache directories. Con || || ||55427fb66c6d52ce98b4d68a29b592a734014c28 ||HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed || || ||0221d19f4e398c386f4ca3990b0893562aa8dacf ||YARN-3850. NM fails to read files from full disks which can lead to container logs being l || || ||516bbf1c20547dc513126df0d9f0934bb65c10c7 ||HDFS-7314. When the DFSClient lease cannot be renewed, abort open-for-write files rather t || || ||c31e3ba92132f232bd56b257f3854ffe430fbab9 ||YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected || || ||d2b941f94a835f7bdde7714d21a470b505aa582b ||HDFS-8850. VolumeScanner thread exits with exception if there is no block pool to be scann || || ||5950c1f6f8ac6f514f8d2e8bfbd1f71747b097de ||HADOOP-11932. MetricsSinkAdapter may hang when being stopped. Contributed by Brahma Reddy || ||
