Thanks Yiqun for sharing with us this morning.
The following is my note taken today. Feel free to update the Google doc in
case I missed something.


Attendee: Yiqun, Weichiu, Chao, Matt, Craig, Pifta

Yiqun presented the configurations and liras that help optimize and
stabilize large scale clusters at eBay.

HDFS-13183 was brought up. It runs well at eBay. Even though community
think Consistent Read from Standby would supersede this improvement,
there’s still value to add it since there can be users who do not use CRFS
or are using older releases.

eBay: Queue time < 10 ms

Uber avg Queue time > 100 ms

eBay is looking into consistent read from standby soon. RBF or Ozone
appears to be a big change so not considered. Additionally, federation is
running well at eBay so not looking into RBF now.

Looking into upgrading to Hadoop 2.9 or Hadoop 3

Ozone tested by Pinduoduo and JD in China.

Talked about recent upstream Hadoop development: Namenode fine grained
locking, OpenTracing, JDK11

On Wed, Oct 30, 2019 at 9:30 AM Wei-Chiu Chuang <> wrote:

> Gentle reminder. Yiqun will present in 30 minutes!
> On Mon, Oct 28, 2019 at 7:41 PM Wei-Chiu Chuang <>
> wrote:
>> Hello, I am super stoked to have Yiqun Lin with us this Wednesday morning
>> Oct 30 US Pacific 10am/CET (Budapest) 6pm/ IST (Banglore) 10:30pm/ CST
>> (Beijing) Oct 31 1am / JST (Tokyo) 2am to talk about “HDFS Cluster
>> Optimization in eBay” — Yiqun happens to be in the bay area this week and
>> this is the same talk that he is going to present Tuesday night at Yahoo
>> this week.
>> HDFS Cluster Optimization in eBay
>> Yiqun Lin, Hadoop Team, eBay + Apache Hadoop Committer / PMC member
>>> On eBay, we have many large HDFS clusters with thousands of nodes. We
>>> face many stability/data availability problems in our cluster. Today we
>>> want to share some optimizations we did in the system layer or HDFS level
>>> to improve our clusters. Besides, that makes our cluster more stable than
>>> before.
>> Past meeting notes and zoom link:
>> Best,
>> Weichiu

Reply via email to