Very happy to have CR from Uber leading today's discussion. Here's today's sync meeting notes. https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit 8/5/2019
CR Hota (Uber) gave an update on Router Based Federation Attendee: Cloudera (Weichiu, Adam), Uber (CR Hota) and Target (Craig) Rename: There is a change in Hive that upon exception, do a copy instead. How/where can the community help:it already support all NN APIs, running in production, mostly now is the efficiency improvement. How to migrate from non-federation to RBF —> easy. still use hfs:// scheme. Will have to update metadata (HMS) How to migrate from ViewFS based federation to RBF —> ViewFS use view’s:// so it’ll be harder to migrate. View FS based is limited to 4 namespaces. There is no such limit in RBF. Uber is already at 5 namespaces. Cluster utilization rebalancer. Not a priority at Uber because of UDestinty. Router HA supported. All routers’ state is synchronized. (Uber: has 10 routers in one cluster) Latency compare to single Namenode which is bottlenecked in 1 NN lock. Read-only name node help solve this problem too. Presto is more latency sensitive. So Uber made a change to support “read-only router” In general, very negligible latency. If there is, just add more routers. Uber doesn’t want to manage 4-5 thousand clusters. They want to manage some set of 1000 thousand clusters in the future. Isolation There is a current problem. Very important for production deployment. See HDFS-14090: fairness in router. Let me know your feedback. Is this the right topic you are looking for? Do you want to present other topics? Development discussion, demos, best practices are welcomed. Best, Weichiu
