Re: [VOTE] create ozone-dev and ozone-issues mailing lists
+1 > On Oct 27, 2019, at 1:25 AM, Elek, Marton wrote: > > > As discussed earlier in the thread of "Hadoop-Ozone repository mailing list > configurations" [1] I suggested to solve the current misconfiguration problem > with creating separated mailing lists (dev/issues) for Hadoop Ozone. > > It would have some additional benefit: for example it would make easier to > follow the Ozone development and future plans. > > Here I am starting a new vote thread (open for at least 72 hours) to collect > more feedback about this. > > Please express your opinion / vote. > > Thanks a lot, > Marton > > [1] > https://lists.apache.org/thread.html/dc66a30f48a744534e748c418bf7ab6275896166ca5ade11560ebaef@%3Chdfs-dev.hadoop.apache.org%3E > > - > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/491/ No changes - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: This week's Hadoop storage community online sync
Thanks Yiqun for sharing with us this morning. The following is my note taken today. Feel free to update the Google doc in case I missed something. https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit?usp=sharing 11/30/2019 10/30 Attendee: Yiqun, Weichiu, Chao, Matt, Craig, Pifta Yiqun presented the configurations and liras that help optimize and stabilize large scale clusters at eBay. HDFS-13183 was brought up. It runs well at eBay. Even though community think Consistent Read from Standby would supersede this improvement, there’s still value to add it since there can be users who do not use CRFS or are using older releases. eBay: Queue time < 10 ms Uber avg Queue time > 100 ms eBay is looking into consistent read from standby soon. RBF or Ozone appears to be a big change so not considered. Additionally, federation is running well at eBay so not looking into RBF now. Looking into upgrading to Hadoop 2.9 or Hadoop 3 Ozone tested by Pinduoduo and JD in China. Talked about recent upstream Hadoop development: Namenode fine grained locking, OpenTracing, JDK11 On Wed, Oct 30, 2019 at 9:30 AM Wei-Chiu Chuang wrote: > Gentle reminder. Yiqun will present in 30 minutes! > > On Mon, Oct 28, 2019 at 7:41 PM Wei-Chiu Chuang > wrote: > >> Hello, I am super stoked to have Yiqun Lin with us this Wednesday morning >> Oct 30 US Pacific 10am/CET (Budapest) 6pm/ IST (Banglore) 10:30pm/ CST >> (Beijing) Oct 31 1am / JST (Tokyo) 2am to talk about “HDFS Cluster >> Optimization in eBay” — Yiqun happens to be in the bay area this week and >> this is the same talk that he is going to present Tuesday night at Yahoo >> this week. >> >> HDFS Cluster Optimization in eBay >> >> >> >> Yiqun Lin, Hadoop Team, eBay + Apache Hadoop Committer / PMC member >>> On eBay, we have many large HDFS clusters with thousands of nodes. We >>> face many stability/data availability problems in our cluster. Today we >>> want to share some optimizations we did in the system layer or HDFS level >>> to improve our clusters. Besides, that makes our cluster more stable than >>> before. >> >> >> Past meeting notes and zoom link: >> >> https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit?usp=sharing >> >> Best, >> Weichiu >> >
Re: This week's Hadoop storage community online sync
Gentle reminder. Yiqun will present in 30 minutes! On Mon, Oct 28, 2019 at 7:41 PM Wei-Chiu Chuang wrote: > Hello, I am super stoked to have Yiqun Lin with us this Wednesday morning > Oct 30 US Pacific 10am/CET (Budapest) 6pm/ IST (Banglore) 10:30pm/ CST > (Beijing) Oct 31 1am / JST (Tokyo) 2am to talk about “HDFS Cluster > Optimization in eBay” — Yiqun happens to be in the bay area this week and > this is the same talk that he is going to present Tuesday night at Yahoo > this week. > > HDFS Cluster Optimization in eBay > > > > Yiqun Lin, Hadoop Team, eBay + Apache Hadoop Committer / PMC member >> On eBay, we have many large HDFS clusters with thousands of nodes. We >> face many stability/data availability problems in our cluster. Today we >> want to share some optimizations we did in the system layer or HDFS level >> to improve our clusters. Besides, that makes our cluster more stable than >> before. > > > Past meeting notes and zoom link: > > https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit?usp=sharing > > Best, > Weichiu >
Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/ No changes -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:[line 335] Failed junit tests : hadoop.util.TestReadWriteDiskValidator hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.namenode.TestNamenodeCapacityReport hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.yarn.client.api.impl.TestTimelineClientV2Impl hadoop.registry.secure.TestSecureLogins hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 hadoop.yarn.sls.TestSLSRunner cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt [328K] cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/diff-compile-cc-root-jdk1.8.0_222.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/diff-compile-javac-root-jdk1.8.0_222.txt [308K] checkstyle: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/diff-checkstyle-root.txt [16M] hadolint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/diff-patch-shellcheck.txt [72K] shelldocs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/diff-patch-shelldocs.txt [8.0K] whitespace: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/whitespace-tabs.txt [1.3M] xml: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/xml.txt [8.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/diff-javadoc-javadoc-root-jdk1.7.0_95.txt [16K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/diff-javadoc-javadoc-root-jdk1.8.0_222.txt [1.1M] unit: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [160K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [232K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt [24K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/490/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K]
