[DISCUSS] fate of branch-2.9

Hi, Following the discussion to end branch-2.8, I want to start a discussion around what's next with branch-2.9. I am hesitant to use the word "end of life" but consider these facts: * 2.9.0 was released Dec 17, 2017. * 2.9.2, the last 2.9.x release, went out Nov 19 2018, which is more than 15

This week's Hadoop storage community call

Hi! I'd like to use this week's community call as an opportunity to drive the releases forward. RMs: Gabor and Brahma, would you be able to join the call? March 4th (Wednesday) US Pacific: 10am, GMT 6pm, India: 11:30pm Please join via Zoom: https://cloudera.zoom.us/j/880548968 Past meeting

[VOTE] EOL Hadoop branch-2.8

ude on March 9th (Mon) 11am PST. Please feel free to share your thoughts. Thanks, Weichiu On Mon, Feb 24, 2020 at 10:28 AM Wei-Chiu Chuang wrote: > Looking at the EOL policy wiki: > https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches > > The Hado

Re: HDFS-13616 : batch listing of multiple directories

Steve, You made a great point and I'm sorry this API was implemented without consideration of other FS implementation. Thank you for your direct feedback. async -- yes builder -- yes cancellable -- totally agree There are good use cases for this API though -- Impala and Presto both require lots

[通知] 建立 user-zh 郵件列表

修改仍應以英文在 *-dev@, JIRAs and GitHub上進行。 這個郵件列表目前已經可以使用了，我們的網站也將再更新後加入此郵件列表。任何人都可藉由發信至 user-zh-subscr...@hadoop.apache.org 訂閱此列表。非訂閱者的信件經審核後可發出。 - 莊偉赳（Apache Hadoop PMC代表） On Fri, Feb 28, 2020 at 9:30 AM Wei-Chiu Chuang wrote: > Hi! > > Apache Hadoop welcomes contributors from around

[ANNOUNCE] Creation of user-zh mailing list

che.org. Non-subscribers may also post messages after the moderators' approvals. - Wei-Chiu Chuang (on behalf of the Apache Hadoop PMC)

Yang Yun's patch needs reviews

I see a new contributor, Yang Yun, posting a number of patches to the Apache JIRA. https://issues.apache.org/jira/issues/?jql=resolution%20%3D%20Unresolved%20AND%20assignee%20%3D%20hadoop_yangyun%20AND%20project%20%3D%2012310942 Some of them look pretty good. Yesterday evening at the community

[jira] [Resolved] (HDFS-14668) Support Fuse with Users from multiple Security Realms

[ https://issues.apache.org/jira/browse/HDFS-14668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14668. Fix Version/s: 3.2.2 3.1.4 3.3.0 Resolution

This week's Hadoop storage community online sync (APAC Mandarin)

Hi! It's that time again. I'd like to lead this week's APAC Mandarin community sync discussion. There are a few things to discuss/announce: (1) user-zh mailing list. (2) Fate of Hadoop 2.x / Hadoop 3.x adoption. (3) Apache jiras pending reviews Zoom link: https://cloudera.zoom.us/j/880548968

Re: [DISCUSS] EOL Hadoop branch-2.8

PM Akira Ajisaka > > > wrote: > > > > > > > Thanks Wei-Chiu for starting the discussion, > > > > > > > > +1 for the EoL. > > > > > > > > -Akira > > > > > > > > On Tue, Feb 18, 2020 at 4:59 PM Ayush Saxe

Re: [DISCUSS] EOL Hadoop branch-2.8

PM Ayush Saxena wrote: > > > > > Thanx Wei-Chiu for initiating this > > > +1 for marking 2.8 EOL > > > > > > -Ayush > > > > > > > On 17-Feb-2020, at 11:14 PM, Wei-Chiu Chuang > > wrote: > > > > > > > > The la

Re: This week's Hadoop storage community online meetup (APAC)

.com/calendar/b/3?cid=aGFkb29wLmNvbW11bml0eS5zeW5jLnVwQGdtYWlsLmNvbQ > ) > > On Thu, Feb 13, 2020 at 11:47 AM Wei-Chiu Chuang > wrote: > > > Thanks for joining the call last night/yesterday. > > > > Please find the video recording here: > > > > > https://cloudera.zoom.

Re: [DISCUSS] hadoop-thirdparty 1.0.0 release

+1 On Fri, Feb 21, 2020 at 1:22 AM Akira Ajisaka wrote: > Thanks Vinayakumar for starting the discussion, > > +1 for the release plan. > I think the release vote timeframe is now 5 days, not 7 days. > > -Akira > > On Fri, Feb 21, 2020 at 3:56 PM Vinayakumar B > wrote: > > > Hi All, > > > >

[DISCUSS] EOL Hadoop branch-2.8

The last Hadoop 2.8.x release, 2.8.5, was GA on September 15th 2018. It's been 17 months since the release and the community by and large have moved up to 2.9/2.10/3.x. With Hadoop 3.3.0 over the horizon, is it time to start the EOL discussion and reduce the number of active branches?

Re: This week's Hadoop storage community online meetup (APAC)

://drive.google.com/open?id=1IUPtknaPUeKIL74TpNt-R6CK5ICz5veW On Wed, Feb 12, 2020 at 9:50 PM Wei-Chiu Chuang wrote: > Gentle reminder for this event. > Siyao will lead the session first, followed by a demo. > > Zoom link: https://cloudera.zoom.us/j/880548968 > > Past meeting

Re: This week's Hadoop storage community online meetup (APAC)

Gentle reminder for this event. Siyao will lead the session first, followed by a demo. Zoom link: https://cloudera.zoom.us/j/880548968 Past meeting minutes: https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit?usp=sharing On Mon, Feb 10, 2020 at 1:42 PM Wei-Chiu

This week's Hadoop storage community online meetup (APAC)

Hi! I would like to spend this week's session to discuss Distributed Tracing in Hadoop. We had a session at the last Bay Area Hadoop meetup back in June discussing the Distributed Tracing work we have been doing. I'd like to share our latest update with the APAC community. Zoom link:

Re: Restrict Frequency of BlockReport To Namenode startup and failover

Hey Ayush, Thanks a lot for your proposal. Do you mean the Full Block Report that is sent out every 6 hours per DataNode? Someone told me they reduced the frequency of FBR to 24 hours and it seems okay. One of the purposes of FBR was to prevent bugs in incremental block report implementation.

Re: Alternative Decommission Monitor Implementation

@Akira Ajisaka you said you'd be interested, right? Are you planning to adopt this feature? On Tue, Feb 4, 2020 at 10:41 AM Stephen O'Donnell wrote: > Hi All, > > In https://issues.apache.org/jira/browse/HDFS-14854 we committed a new > decommission monitor to trunk, which is disabled by

Re: Introduce Read Write Lock to Datanode

Thanks for initiating this discussion here. I am +1 to the general approach proposed. With DN getting denser, this is necessary more than ever. On Tue, Feb 4, 2020 at 10:33 AM Stephen O'Donnell wrote: > I would like to reopen an old topic, which is to introduce a Read Write > lock to the

[INFO] New branch created: HDFS-14978_ec_conversion (Re: [DISCUSS] Feature branch for HDFS-14978 In-place Erasure Coding Conversion)

Thank you. FYI, I went ahead and created the branch HDFS-14978_ec_conversion On Fri, Jan 24, 2020 at 5:07 AM Ayush Saxena wrote: > +1 for the feature branch > > -Ayush > > > On 24-Jan-2020, at 3:04 AM, Wei-Chiu Chuang > wrote: > > > > Hi we are working on a

AWS promotional credits for open source projects

https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/ Any one interested? I think it can be a useful resource especially for the S3 cloud connector.

This week's Hadoop storage community online meetup

Hi! After a bit of a hiatus, I'd like to revive the regular community sync! As usual, this call is scheduled at US pacific time 10am 1/22/2020 (Wednesday), GMT 6pm, India 11:30pm and Beijing 2am 1/23 (Thursday) Agenda for this week: (1) Hadoop 3.3.0 release plan (2) Use RocksDB to keep NameNode

Re: Hadoop 3.1.4 Release Plan Proposal

Sounds good to me. Thanks for doing this! On Mon, Jan 20, 2020 at 4:04 AM Gabor Bota wrote: > Hi All, > > Based on the discussion on the topic "Hadoop 2019 Release Planning" I > volunteer to do the next 3.1 Hadoop release, version 3.1.4. > > You can find the blocker/critical issues and all

[jira] [Resolved] (HDFS-14126) DataNode DirectoryScanner holding global lock for too long

[ https://issues.apache.org/jira/browse/HDFS-14126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14126. Resolution: Duplicate I believe HDFS-14476 solves the same issue and so I'll mark

Re: [DISCUSS] Hadoop 2019 Release Planning

I'm curious about the fate of branch-2.8 and branch-2.9. 2.9.2 is over a year old (released on 11/19/2018) 2.8.5 is over a year old too (released on 9/10/2018) It appears to me most of the development in 2.x is focusing on stabilizing 2.10 and I wonder if there are people still on 2.8/2.9.

[jira] [Reopened] (HDFS-14476) lock too long when fix inconsistent blocks between disk and in-memory

[ https://issues.apache.org/jira/browse/HDFS-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-14476: > lock too long when fix inconsistent blocks between disk and in-mem

Re: [DISCUSS] Guidelines for Code cleanup JIRAs

+1 On Thu, Jan 9, 2020 at 9:33 AM epa...@apache.org wrote: > There was some discussion on > https://issues.apache.org/jira/browse/YARN-9052 > about concerns surrounding the costs/benefits of code cleanup JIRAs. This > email > is to get the discussion going within a wider audience. > > The

Re: Reminder: Hadoop Storage Online Meetup tomorrow (Hadoop 2->3 upgrade)

about the session this time. I served as the translator this time and I need your feedback to improve next time. On Fri, Jan 3, 2020 at 10:01 PM Wei-Chiu Chuang wrote: > > Hi, it was a well attended session with more than 40 attendees joined! > Thanks Fei Hui for giving us such a g

Re: Reminder: Hadoop Storage Online Meetup tomorrow (Hadoop 2->3 upgrade)

/write EC blocks! On Wed, Jan 1, 2020 at 7:42 PM Wei-Chiu Chuang wrote: > Hi, > This is a gentle reminder for tomorrow's online meetup. Fei Hui from DiDi > is going to give a presentation about DiDi's Hadoop 2 -> Hadoop 3 upgrade > experience. > > We will extend this s

Reminder: Hadoop Storage Online Meetup tomorrow (Hadoop 2->3 upgrade)

Hi, This is a gentle reminder for tomorrow's online meetup. Fei Hui from DiDi is going to give a presentation about DiDi's Hadoop 2 -> Hadoop 3 upgrade experience. We will extend this session to 1 hour. Fei will speak in Mandarin and I will help translate. So non-Mandarin speakers feel free to

Status of Hadoop 2.9.x

Hi! Duo found out that the last 2.9.x release, 2.9.2 was GA in Nov 2018, which is a little more than a year ago. Are we still interested in maintaining that branch? Any volunteer likes to continue it? As far as I know, Twitter just recently completed upgrade to 2.9 so I imagine there are still

Re: Next Hadoop Storage Online Meetup (APAC Mandarin)

19, 2019 at 4:03 PM Eric Badger wrote: > For those of us that don't speak Mandarin, would someone be able to take > notes in English? I'm very interested in hearing about the experience in > moving from Hadoop 2.x to 3.x. > > Eric > > On Thu, Dec 19, 2019 at 2:07 PM Wei

Next Hadoop Storage Online Meetup (APAC Mandarin)

As you have probably aware, DiDi upgrade a large cluster from Hadoop 2 to Hadoop3 recently. Fei Hui from DiDi graciously agreed to speak to us their upgrade experience at the next APAC Mandarin Online meetup which is in two weeks. So stay tuned! Time/Date: Jan 1 10PM (US west coast PST) / Jan 2

[jira] [Created] (HDFS-15052) WebHDFS getTrashRoot leads to OOM due to FileSystem object creation

Wei-Chiu Chuang created HDFS-15052: -- Summary: WebHDFS getTrashRoot leads to OOM due to FileSystem object creation Key: HDFS-15052 URL: https://issues.apache.org/jira/browse/HDFS-15052 Project

[jira] [Created] (HDFS-15047) Document the new decommission monitor (HDFS-14854)

Wei-Chiu Chuang created HDFS-15047: -- Summary: Document the new decommission monitor (HDFS-14854) Key: HDFS-15047 URL: https://issues.apache.org/jira/browse/HDFS-15047 Project: Hadoop HDFS

[jira] [Created] (HDFS-15046) Backport HDFS-7060 to branch-2.10

Wei-Chiu Chuang created HDFS-15046: -- Summary: Backport HDFS-7060 to branch-2.10 Key: HDFS-15046 URL: https://issues.apache.org/jira/browse/HDFS-15046 Project: Hadoop HDFS Issue Type

[jira] [Resolved] (HDFS-14476) lock too long when fix inconsistent blocks between disk and in-memory

[ https://issues.apache.org/jira/browse/HDFS-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14476. Resolution: Fixed Push into trunk. > lock too long when fix inconsistent blocks betw

Re: APAC Mandarin community sync happening now

performance. Haiyang stated they tested EC performance with Java codec (w/o Intel ISA-L library) so performance suffered. Weichiu: use the latest Intel ISA-L library gives the best performance. On Wed, Dec 4, 2019 at 9:59 PM Wei-Chiu Chuang wrote: > Join Zoom Meeting > > https://cloudera.

APAC Mandarin community sync happening now

Join Zoom Meeting https://cloudera.zoom.us/j/880548968

Re: [DISCUSS] Enable github security notifications to all Hadoop committers

t; from various Hadoop alias each day so I don't mind adding a few more. >> >> Completely agreed. >> >> -Akira >> >> On Thu, Oct 24, 2019 at 3:48 PM Bharat Viswanadham >> wrote: >> >> > +1 (binding). I am interested in receiving these n

Re: This Week's APAC Hadoop storage community sync

block files. Hierarchical cache storage. - Alibaba evaluated HBase BucketCache with Pmem. There’s a benchmark report somewhere that Rakesh On Wed, Nov 20, 2019 at 9:13 PM Wei-Chiu Chuang wrote: > Friendly reminder: this event is happening in an hour! > > On Mon, Nov 18, 2019

Re: This Week's APAC Hadoop storage community sync

Friendly reminder: this event is happening in an hour! On Mon, Nov 18, 2019 at 11:38 PM Wei-Chiu Chuang wrote: > I'm really bad at converting time across different time zones. > > Here's the correction: > Pacific Time: Wed Nov 20 10pm > Beijing Time: Thu Nov 21 2pm > India T

Re: [DISCUSS] Move HDDS JIRA notification to ozone-dev@/ozone-issues@

On Tue, Nov 19, 2019 at 7:04 AM Wei-Chiu Chuang wrote: > Hi, > > Looking at the Notification settings of HDDS project, it is still sending > notification emails to hdfs-dev@/hdfs-issues@. Can we move them to > ozone-dev@ and ozone-issues@? > > Weichiu >

[DISCUSS] Move HDDS JIRA notification to ozone-dev@/ozone-issues@

Hi, Looking at the Notification settings of HDDS project, it is still sending notification emails to hdfs-dev@/hdfs-issues@. Can we move them to ozone-dev@ and ozone-issues@? Weichiu - To unsubscribe, e-mail:

[jira] [Resolved] (HDFS-12866) Recursive delete of a large directory or snapshot makes namenode unresponsive

[ https://issues.apache.org/jira/browse/HDFS-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-12866. Resolution: Duplicate Resolve as it duplicates HDFS-11225 > Recursive delete of a la

[jira] [Resolved] (HDFS-11225) NameNode crashed because deleteSnapshot held FSNamesystem lock too long

[ https://issues.apache.org/jira/browse/HDFS-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-11225. Fix Version/s: 3.1.0 Resolution: Fixed [~shashikant] completed the subtasks. I'm

This Week's APAC Hadoop storage community sync

Hi! I am happy to have Feilong from Intel to join us and talk about the Storage Class Memory support for HDFS ( HDFS-13762 ). This is a new feature that will land in the next Hadoop minor release (Hadoop 3.3.0) Date/Time: Wednesday Nov 20 10pm

Today's Hadoop storage online community sync

-pR1UxEBRrrTiUkX3CxstfKY/edit?usp=sharing Steve L.: People need to care about nightly failures. Zhenyu: add ARM test build in nightly or periodically, and then precommit builds. On Fri, Nov 8, 2019 at 3:58 PM Wei-Chiu Chuang wrote: > Hi, > > I am happy to invite Zhenyu to join us to t

Re: Next Wednesday (Nov 13) Hadoop storage online community sync

Just a reminder. This online sync is starting in 3 minutes. On Fri, Nov 8, 2019 at 3:58 PM Wei-Chiu Chuang wrote: > Hi, > > I am happy to invite Zhenyu to join us to talk about the recent proposal > of supporting ARM/aarch64 for Hadoop. > > November 13 (Wednesday) US

[jira] [Resolved] (HDFS-14959) [SBNN read] access time should be turned off

[ https://issues.apache.org/jira/browse/HDFS-14959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14959. Resolution: Fixed Merged the PR to trunk and cherry pick the commit to branch-3.2

[jira] [Created] (HDFS-14978) In-place Erasure Coding Conversion

Wei-Chiu Chuang created HDFS-14978: -- Summary: In-place Erasure Coding Conversion Key: HDFS-14978 URL: https://issues.apache.org/jira/browse/HDFS-14978 Project: Hadoop HDFS Issue Type: New

Next Wednesday (Nov 13) Hadoop storage online community sync

Hi, I am happy to invite Zhenyu to join us to talk about the recent proposal of supporting ARM/aarch64 for Hadoop. November 13 (Wednesday) US Pacific Time 10am / November 13 (Wednesday) Bangalore 11:30pm) / November 14 (Thursday) Beijing 2am. Previous meeting notes:

Notes from Hadoop storage community online sync

Thanks @Xiaoyu Yao for giving us a great status update on Ozone! We had a pretty large group yesterday. Here's my notes for your reference: https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit?usp=sharing 11/6/2019 ~20 contributors joined the discussion.Weichiu,

[jira] [Created] (HDFS-14959) [SBNN read] access time should be turned off

Wei-Chiu Chuang created HDFS-14959: -- Summary: [SBNN read] access time should be turned off Key: HDFS-14959 URL: https://issues.apache.org/jira/browse/HDFS-14959 Project: Hadoop HDFS Issue

Topic for this Wednesday's APAC Hadoop storage community sync: Ozone

Hi! There have been a number of questions especially from the Chinese community regarding Ozone lately. I am happy to have @Xiaoyu Yao who graciously agreed to talk to us the current status of Ozone this Wednesday even (US) / Thursday (APAC). Time: US Pacific: Wednesday (Nov 6) 10pm Beijing:

[jira] [Resolved] (HDFS-14925) rename operation should check nest snapshot

[ https://issues.apache.org/jira/browse/HDFS-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14925. Fix Version/s: 3.2.2 3.1.4 Resolution: Fixed Thanks! merged

Re: This week's Hadoop storage community online sync

Bay so not looking into RBF now. Looking into upgrading to Hadoop 2.9 or Hadoop 3 Ozone tested by Pinduoduo and JD in China. Talked about recent upstream Hadoop development: Namenode fine grained locking, OpenTracing, JDK11 On Wed, Oct 30, 2019 at 9:30 AM Wei-Chiu Chuang wrote: > Gentle remind

Re: This week's Hadoop storage community online sync

Gentle reminder. Yiqun will present in 30 minutes! On Mon, Oct 28, 2019 at 7:41 PM Wei-Chiu Chuang wrote: > Hello, I am super stoked to have Yiqun Lin with us this Wednesday morning > Oct 30 US Pacific 10am/CET (Budapest) 6pm/ IST (Banglore) 10:30pm/ CST > (Beijing) Oct 31 1am / JST (T

This week's Hadoop storage community online sync

Hello, I am super stoked to have Yiqun Lin with us this Wednesday morning Oct 30 US Pacific 10am/CET (Budapest) 6pm/ IST (Banglore) 10:30pm/ CST (Beijing) Oct 31 1am / JST (Tokyo) 2am to talk about “HDFS Cluster Optimization in eBay” — Yiqun happens to be in the bay area this week and this is the

[jira] [Resolved] (HDFS-14308) DFSStripedInputStream curStripeBuf is not freed by unbuffer()

[ https://issues.apache.org/jira/browse/HDFS-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14308. Fix Version/s: 3.2.2 3.1.4 3.3.0 Resolution

Re: Reminder: APAC Hadoop storage community sync

. There are internal teams that rely on the access time stats. We discussed this and mentioned Uber stated the same issue in the January Hadoop meetup. Didi’s Hadoop team is interested in Ozone after all clusters are upgraded to Hadoop 3. On Wed, Oct 23, 2019 at 3:55 PM Wei-Chiu Chuang wrote: > PDT 1

Re: [DISCUSS] Remove Ozone and Submarine from Hadoop repo

+1 filed HADOOP-16670 for stripping the Submarine code. On Thu, Oct 24, 2019 at 12:14 PM Subru Krishnan wrote: > +1. > > Thanks, > Subru > > On Thu, Oct 24, 2019 at 12:51 AM 张铎(Duo Zhang) > wrote: > > > +1 > > > > Akira Ajisaka

Fwd: Cloud Recording - HDFS sync is now available

This today's HDFS sync recording. Siyao's Dynamometer presentation starts at around 18th minute. -- Forwarded message - From: Zoom Date: Thu, Oct 24, 2019 at 12:19 PM Subject: Cloud Recording - HDFS sync is now available To: Hi Wei-Chiu Chuang, Your cloud recording is now

[DISCUSS] Enable github security notifications to all Hadoop committers

Hi, I raised INFRA-19327 to enable github security notification. How do people feel if we enable this notification to all committers? I already have hundreds of incoming emails from various Hadoop alias each day so I don't mind adding a few more.

HDFS sync

Hi folks, I don't want to add meetings unnecessarily. But I would like to see the oncall engineer of the week to join and summarize the support cases (CDH engineering escalations, HDP EARs). Ideally any one with a case open should join too, but let's start with this and see how it goes. Yes I am

Reminder: APAC Hadoop storage community sync

PDT 10pm Wednesday = tonight, CST 1pm Thursday = today. Feel free to join Zoom and chat. Join Zoom Meeting https://cloudera.zoom.us/j/880548968 Past sessions: https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit Also heads-up, On November 20/21, Feilong from

Re: How should we do about dependency update?

ical. I don't know how often a CVE > comes along for things like our logging API dependency, for example. > But downstream folks should definitely not rely on dependencies we use > for internal service, so I'm surprised that a version change for Jetty > would impact downstream. > >

[jira] [Resolved] (HDFS-12049) Recommissioning live nodes stalls the NN

[ https://issues.apache.org/jira/browse/HDFS-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-12049. Resolution: Duplicate > Recommissioning live nodes stalls the

How should we do about dependency update?

Hi Hadoop developers, I've always had this question and I don't know the answer. For the last few months I finally spent time to deal with the vulnerability reports from our internal dependency check tools. Say in HADOOP-16152 we update Jetty

Meeting notes from today's Hadoop storage community sync

Here's today's notes for future reference: https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit?usp=sharing 10/16/2019 Attendee: Weichiu, Cynthia, Craig, Stephen, Akira, David Stephen introduced upgrade domain, which was developed at Twitter. Cloudera is going to

Re: Reminder: This Wednesday's community sync

Gentle reminder: community sync happening in 3 hours. On Mon, Oct 14, 2019 at 9:07 AM Wei-Chiu Chuang wrote: > Hadoop devs, > > This Wednesday (PDT 10am, EDT 1pm, BST 6pm, CEST 7pm), @Stephen O'Donnell > is going to share with us the projects he's > spent most time on rec

Reminder: This Wednesday's community sync

Hadoop devs, This Wednesday (PDT 10am, EDT 1pm, BST 6pm, CEST 7pm), @Stephen O'Donnell is going to share with us the projects he's spent most time on recently, DataNode decommissioning improvement (HDFS-14854 ) and Upgrade Domain support

Official Apache Slack Channel for Hadoop projects

Hi Hadoop devs, In case you don't know, there is an official ASF slack, and there's a HDFS channel in it. This is the slack workplace managed by Apache Infra. Please see this wiki to get invite: https://cwiki.apache.org/confluence/display/INFRA/Slack+Guest+Invites or DM me to get an invite.

Hadoop storage online sync (Mandarin) happening in 2 hours

I would like to talk about Erasure Coding development this time around. But feel free to join and chime in. Join via Zoom: https://docs.google.com/document/d/1XkrcyVil_ORV1UP-JhosGzK8qWGXXX3wuplo4RtC7u0/edit Past meeting minutes:

Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

Hi I am late to this but I am keen to understand more. To be exact, how can we better use the thirdparty repo? Looking at HBase as an example, it looks like everything that are known to break a lot after an update get shaded into the hbase-thirdparty artifact: guava, netty, ... etc. Is it the

Please cherry pick commits to lower branches

I spent the whole last week cherry picking commits from trunk/branch-3.2 to branch-3.1 (should've done this prior to 3.1.4 code freeze). There were about 50-60 of them, many of them are conflict-free, and several of them are critical bug fixes. If your commit stays in trunk, it'll be useless for

Re: CVE-2018-11768: HDFS FSImage Corruption

Per Apache security vulnerability report policy https://www.apache.org/security/committers.html we do not make public JIRAs for vulnerability reports. The CVE itself has nothing to do with rollbacks. However, the fix itself changes fsimage format and you won't be able to rollback to a 2.7

[jira] [Reopened] (HDFS-14216) NullPointerException happens in NamenodeWebHdfs

[ https://issues.apache.org/jira/browse/HDFS-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-14216: Reopen for branch-3.1. The only thing different is the LOG class change. Can't use

[jira] [Reopened] (HDFS-14754) Erasure Coding : The number of Under-Replicated Blocks never reduced

[ https://issues.apache.org/jira/browse/HDFS-14754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-14754: > Erasure Coding : The number of Under-Replicated Blocks never redu

Reminder: Today's storage community online sync

Just like to send a reminder. Sync is happening today 10am pacific time. To connect, check out instructions: https://docs.google.com/document/d/1XkrcyVil_ORV1UP-JhosGzK8qWGXXX3wuplo4RtC7u0/edit Past sync meeting notes:

[jira] [Reopened] (HDFS-14192) Track missing DFS operations in Statistics and StorageStatistics

[ https://issues.apache.org/jira/browse/HDFS-14192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-14192: Reopen for the branch-3.1 backport. > Track missing DFS operations in Statist

[jira] [Resolved] (HDFS-14418) Remove redundant super user priveledge checks from namenode.

[ https://issues.apache.org/jira/browse/HDFS-14418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14418. Resolution: Fixed > Remove redundant super user priveledge checks from namen

[jira] [Reopened] (HDFS-14418) Remove redundant super user priveledge checks from namenode.

[ https://issues.apache.org/jira/browse/HDFS-14418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-14418: Reopen to add this in branch-3.1 > Remove redundant super user priveledge checks f

[jira] [Resolved] (HDFS-7134) Replication count for a block should not update till the blocks have settled on Datanodes

[ https://issues.apache.org/jira/browse/HDFS-7134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-7134. --- Resolution: Cannot Reproduce Resolve as cannot reproduce. > Replication count for a bl

[jira] [Reopened] (HDFS-7134) Replication count for a block should not update till the blocks have settled on Datanodes

[ https://issues.apache.org/jira/browse/HDFS-7134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-7134: --- > Replication count for a block should not update till the blocks have settled > on Dat

[jira] [Resolved] (HDFS-14564) Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable

[ https://issues.apache.org/jira/browse/HDFS-14564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14564. Fix Version/s: 3.3.0 Resolution: Fixed Thanks [~stakiar] for the patch and [~smeng

Re: [DISCUSS] GitHub PRs without JIRA number

Thanks for doing this! It also looks like "Scan Now" also triggers builds for non-committer's PRs. That's great! On Tue, Sep 10, 2019 at 10:21 PM 张铎(Duo Zhang) wrote: > Actually the job for testing PR is here... > > https://builds.apache.org/job/hadoop-multibranch/ > > I've added the 'Change

[jira] [Resolved] (HDFS-14774) RBF: Improve RouterWebhdfsMethods#chooseDatanode() error handling

[ https://issues.apache.org/jira/browse/HDFS-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14774. Resolution: Not A Problem Thanks CR. I'm resolving it. > RBF: Impr

Re: [VOTE] Moving Submarine to a separate Apache project proposal

+1 I've involved myself in Submarine dev and I'd like to be included in the future. Thanks On Sat, Sep 7, 2019 at 5:27 AM Owen O'Malley wrote: > Since you don't have any Apache Members, I'll join to provide Apache > oversight. > > .. Owen > > On Fri, Sep 6, 2019 at 1:38 PM Owen O'Malley >

Re: Hadoop Storage online sync in an hour

ink again for this meeting? > > Thanks, > Aaron > > On Wed, Sep 4, 2019 at 9:31 AM Matt Foley wrote: > >> Sorry I won’t be able to come today; a work meeting interferes. >> —Matt >> >> On Sep 4, 2019, at 9:10 AM, Wei-Chiu Chuang wrote: >> >> It's a short we

Re: [DISCUSS] GitHub PRs without JIRA number

+general@ On Wed, Aug 28, 2019 at 6:42 AM Wei-Chiu Chuang wrote: > I don't think our GitHub integration supports those commands. Ozone has > its own github integration that can test individual PRs though. > > > > On Tue, Aug 27, 2019 at 12:40 PM Iñigo Goiri wrote: > &g

Hadoop Storage online sync in an hour

It's a short week so I didn't set up a predefined topic to discuss. What should we be discussing? How about Erasure Coding? I'm starting to see tricky EC bug reports coming in lately, so looks like folks are using it in production. Should we be thinking about the next step for EC in addition to

[jira] [Resolved] (HDFS-14706) Checksums are not checked if block meta file is less than 7 bytes

[ https://issues.apache.org/jira/browse/HDFS-14706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14706. Resolution: Fixed Done. Reverted the commits and pushed 08 patch to trunk branch-3.2

[jira] [Reopened] (HDFS-14706) Checksums are not checked if block meta file is less than 7 bytes

[ https://issues.apache.org/jira/browse/HDFS-14706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-14706: Reopen. For future reference, I committed the wrong patch. I'm going to revert and re-apply

[jira] [Created] (HDFS-14800) Data race between block report and recoverLease()

Wei-Chiu Chuang created HDFS-14800: -- Summary: Data race between block report and recoverLease() Key: HDFS-14800 URL: https://issues.apache.org/jira/browse/HDFS-14800 Project: Hadoop HDFS

Re: branch-3. Was it a created by mistake?

ssion thread for more info i.e https://s.apache.org/zla4u > > -Rohith Sharma K S > > On Wed, 28 Aug 2019 at 04:54, Wei-Chiu Chuang > > wrote: > > > I just realized there is a branch-3 in the Hadoop repo. Was this created > by > > mistake? > > > > I don't

Re: [DISCUSS] GitHub PRs without JIRA number

enkins trigger. > I've seen the "retest this" and others: > https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin > https://github.com/jenkinsci/ghprb-plugin/blob/master/README.md > > > > On Tue, Aug 27, 2019 at 10:47 AM Wei-Chiu Chuang > wrote:

branch-3. Was it a created by mistake?

I just realized there is a branch-3 in the Hadoop repo. Was this created by mistake? I don't think we've decided to create a branch-3. It also looks like the branch is rarely used. The last commit was commit bf90a27b51b1f1ac102fa861eb28025d21aad19b (origin/branch-3, branch-3) Author: Chen Liang

[DISCUSS] GitHub PRs without JIRA number

Hi, There are hundreds of GitHub PRs pending review. Many of them just sit there wasting Jenkins resources. I suggest: (1) close PRs that went stale (i.e. doesn't compile). Or even close PRs that hasn't been reviewed for more than a year. (1) close PRs that doesn't have a JIRA number. No one is

<    1   2   3   4   5   6   7   8   >