Re: [Discuss] Ozone moving to Beta tag
+1 for the Beta. Thanks Shashi On Mon, Feb 24, 2020 at 9:33 AM Mukul Kumar Singh wrote: > +1 for the Beta. Data pipeline and Ozone manager improvement have > certainly helped in the latest runs. > > Thanks, > > Mukul > > On 24/02/20 8:44 am, Bharat Viswanadham wrote: > > +1 for Beta given major performance improvement work went in Ozone > Manager > > and Datanode Pipeline. > > > > I have been testing Teragen runs and now we have consistent runs and > > performance is almost near to HDFS with disaggregated Storage and compute > > cluster. > > > > > > > > Thanks, > > Bharat > > > > > > On Sun, Feb 23, 2020 at 6:35 PM Sammi Chen wrote: > > > >> +1, Impressive performance achievement on OzoneManager, let's move to > >> Beta. > >> > >> Bests, > >> Sammi Chen > >> > >> On Thu, Feb 20, 2020 at 4:17 AM Anu Engineer > wrote: > >> > >>> Hi All, > >>> > >>> > >>> I would like to propose moving Ozone from 'Alpha' tags to 'Beta' tags > >> when > >>> we do future releases. Here are a couple of reasons why I think we > should > >>> make this move. > >>> > >>> > >>> > >>> 1. Ozone Manager or the Namenode for Ozone scales to more than 1 > >> billion > >>> keys. We tested this in our labs in an organic fashion; that is, we > >> were > >>> able to create more than 1 billion keys from external clients with > no > >>> loss > >>> in performance. > >>> 2. The ozone Manager meets the performance and resource constraints > >> that > >>> we set out to achieve. We were able to sustain the same throughput > at > >>> Ozone > >>> manager for over three days that took us to get this 1 billion > keys. > >>> That > >>> is, we did not have to shut down or resize memory for the namenode > as > >> we > >>> went through this exercise. > >>> 3. The most critical, we did this experiment with 64GB of memory > >>> allocation in JVM and 64 GB of RAM off-heap allocation. That is, > the > >>> Ozone > >>> Manager was able to achieve this scale with far less memory > footprint > >>> than > >>> HDFS. > >>> 4. Ozone's performance is at par with HDFS when running workloads > like > >>> Hive ( > >>> > >>> > >> > https://blog.cloudera.com/benchmarking-ozone-clouderas-next-generation-storage-for-cdp/ > >>> ) > >>> 5. We have been able to run long-running clusters with Ozone. > >>> > >>> > >>> Having achieved these goals, I propose that we move from the planned > >>> 0.4.2-Alpha release to 0.5.0-Beta as our next release. If we hear no > >>> concerns about this, we would like to move Ozone from Alpha to Beta > >>> releases. > >>> > >>> > >>> Thanks > >>> > >>> Anu > >>> > >>> > >>> P.S. I am CC-ing HDFS dev since many people who are interested in Ozone > >>> still have not subscribed to Ozone dev lists. My apologies if it feels > >> like > >>> spam, I promise that over time we will become less noisy in the HDFS > >>> channel. > >>> > >>> > >>> PPS. I know lots of you will want to know more specifics; Our blog > >> presses > >>> are working overtime and I promise you that you will get to see all the > >>> details pretty soon. > >>> > > - > To unsubscribe, e-mail: ozone-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: ozone-dev-h...@hadoop.apache.org > >
Re: [Discuss] Ozone moving to Beta tag
+1 for the Beta. Data pipeline and Ozone manager improvement have certainly helped in the latest runs. Thanks, Mukul On 24/02/20 8:44 am, Bharat Viswanadham wrote: +1 for Beta given major performance improvement work went in Ozone Manager and Datanode Pipeline. I have been testing Teragen runs and now we have consistent runs and performance is almost near to HDFS with disaggregated Storage and compute cluster. Thanks, Bharat On Sun, Feb 23, 2020 at 6:35 PM Sammi Chen wrote: +1, Impressive performance achievement on OzoneManager, let's move to Beta. Bests, Sammi Chen On Thu, Feb 20, 2020 at 4:17 AM Anu Engineer wrote: Hi All, I would like to propose moving Ozone from 'Alpha' tags to 'Beta' tags when we do future releases. Here are a couple of reasons why I think we should make this move. 1. Ozone Manager or the Namenode for Ozone scales to more than 1 billion keys. We tested this in our labs in an organic fashion; that is, we were able to create more than 1 billion keys from external clients with no loss in performance. 2. The ozone Manager meets the performance and resource constraints that we set out to achieve. We were able to sustain the same throughput at Ozone manager for over three days that took us to get this 1 billion keys. That is, we did not have to shut down or resize memory for the namenode as we went through this exercise. 3. The most critical, we did this experiment with 64GB of memory allocation in JVM and 64 GB of RAM off-heap allocation. That is, the Ozone Manager was able to achieve this scale with far less memory footprint than HDFS. 4. Ozone's performance is at par with HDFS when running workloads like Hive ( https://blog.cloudera.com/benchmarking-ozone-clouderas-next-generation-storage-for-cdp/ ) 5. We have been able to run long-running clusters with Ozone. Having achieved these goals, I propose that we move from the planned 0.4.2-Alpha release to 0.5.0-Beta as our next release. If we hear no concerns about this, we would like to move Ozone from Alpha to Beta releases. Thanks Anu P.S. I am CC-ing HDFS dev since many people who are interested in Ozone still have not subscribed to Ozone dev lists. My apologies if it feels like spam, I promise that over time we will become less noisy in the HDFS channel. PPS. I know lots of you will want to know more specifics; Our blog presses are working overtime and I promise you that you will get to see all the details pretty soon. - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [Discuss] Ozone moving to Beta tag
+1 for Beta given major performance improvement work went in Ozone Manager and Datanode Pipeline. I have been testing Teragen runs and now we have consistent runs and performance is almost near to HDFS with disaggregated Storage and compute cluster. Thanks, Bharat On Sun, Feb 23, 2020 at 6:35 PM Sammi Chen wrote: > +1, Impressive performance achievement on OzoneManager, let's move to > Beta. > > Bests, > Sammi Chen > > On Thu, Feb 20, 2020 at 4:17 AM Anu Engineer wrote: > > > Hi All, > > > > > > I would like to propose moving Ozone from 'Alpha' tags to 'Beta' tags > when > > we do future releases. Here are a couple of reasons why I think we should > > make this move. > > > > > > > >1. Ozone Manager or the Namenode for Ozone scales to more than 1 > billion > >keys. We tested this in our labs in an organic fashion; that is, we > were > >able to create more than 1 billion keys from external clients with no > > loss > >in performance. > >2. The ozone Manager meets the performance and resource constraints > that > >we set out to achieve. We were able to sustain the same throughput at > > Ozone > >manager for over three days that took us to get this 1 billion keys. > > That > >is, we did not have to shut down or resize memory for the namenode as > we > >went through this exercise. > >3. The most critical, we did this experiment with 64GB of memory > >allocation in JVM and 64 GB of RAM off-heap allocation. That is, the > > Ozone > >Manager was able to achieve this scale with far less memory footprint > > than > >HDFS. > >4. Ozone's performance is at par with HDFS when running workloads like > >Hive ( > > > > > https://blog.cloudera.com/benchmarking-ozone-clouderas-next-generation-storage-for-cdp/ > >) > >5. We have been able to run long-running clusters with Ozone. > > > > > > Having achieved these goals, I propose that we move from the planned > > 0.4.2-Alpha release to 0.5.0-Beta as our next release. If we hear no > > concerns about this, we would like to move Ozone from Alpha to Beta > > releases. > > > > > > Thanks > > > > Anu > > > > > > P.S. I am CC-ing HDFS dev since many people who are interested in Ozone > > still have not subscribed to Ozone dev lists. My apologies if it feels > like > > spam, I promise that over time we will become less noisy in the HDFS > > channel. > > > > > > PPS. I know lots of you will want to know more specifics; Our blog > presses > > are working overtime and I promise you that you will get to see all the > > details pretty soon. > > >
Re: [Discuss] Ozone moving to Beta tag
+1, Impressive performance achievement on OzoneManager, let's move to Beta. Bests, Sammi Chen On Thu, Feb 20, 2020 at 4:17 AM Anu Engineer wrote: > Hi All, > > > I would like to propose moving Ozone from 'Alpha' tags to 'Beta' tags when > we do future releases. Here are a couple of reasons why I think we should > make this move. > > > >1. Ozone Manager or the Namenode for Ozone scales to more than 1 billion >keys. We tested this in our labs in an organic fashion; that is, we were >able to create more than 1 billion keys from external clients with no > loss >in performance. >2. The ozone Manager meets the performance and resource constraints that >we set out to achieve. We were able to sustain the same throughput at > Ozone >manager for over three days that took us to get this 1 billion keys. > That >is, we did not have to shut down or resize memory for the namenode as we >went through this exercise. >3. The most critical, we did this experiment with 64GB of memory >allocation in JVM and 64 GB of RAM off-heap allocation. That is, the > Ozone >Manager was able to achieve this scale with far less memory footprint > than >HDFS. >4. Ozone's performance is at par with HDFS when running workloads like >Hive ( > > https://blog.cloudera.com/benchmarking-ozone-clouderas-next-generation-storage-for-cdp/ >) >5. We have been able to run long-running clusters with Ozone. > > > Having achieved these goals, I propose that we move from the planned > 0.4.2-Alpha release to 0.5.0-Beta as our next release. If we hear no > concerns about this, we would like to move Ozone from Alpha to Beta > releases. > > > Thanks > > Anu > > > P.S. I am CC-ing HDFS dev since many people who are interested in Ozone > still have not subscribed to Ozone dev lists. My apologies if it feels like > spam, I promise that over time we will become less noisy in the HDFS > channel. > > > PPS. I know lots of you will want to know more specifics; Our blog presses > are working overtime and I promise you that you will get to see all the > details pretty soon. >
Re: [Discuss] Ozone moving to Beta tag
+1 for beta given major improvements. On Wed, Feb 19, 2020 at 4:17 PM Jitendra Pandey wrote: > +1. Given massive improvements in performance and stability, ozone is ready > for beta. > > On Wed, Feb 19, 2020 at 1:04 PM Salvatore LaMendola (BLOOMBERG/ 919 3RD A) > < > slamendo...@bloomberg.net> wrote: > > > +1 on moving to beta. This move makes sense to me. > > > > I've tested each point release so far in 0.4, including building several > > master snapshots along the way, and I agree with what Anu is saying below > > with regard to Ozone's stability having improved beyond "alpha" state. > > > > > > From: aengin...@apache.org At: 02/19/20 15:17:38To: > > ozone-...@hadoop.apache.org, hdfs-dev@hadoop.apache.org > > Subject: [Discuss] Ozone moving to Beta tag > > > > Hi All, > > > > > > I would like to propose moving Ozone from 'Alpha' tags to 'Beta' tags > when > > we do future releases. Here are a couple of reasons why I think we should > > make this move. > > > > > >1. Ozone Manager or the Namenode for Ozone scales to more than 1 > billion > >keys. We tested this in our labs in an organic fashion; that is, we > were > >able to create more than 1 billion keys from external clients with no > > loss > >in performance. > >2. The ozone Manager meets the performance and resource constraints > that > >we set out to achieve. We were able to sustain the same throughput at > > Ozone > >manager for over three days that took us to get this 1 billion keys. > > That > >is, we did not have to shut down or resize memory for the namenode as > we > >went through this exercise. > >3. The most critical, we did this experiment with 64GB of memory > >allocation in JVM and 64 GB of RAM off-heap allocation. That is, the > > Ozone > >Manager was able to achieve this scale with far less memory footprint > > than > >HDFS. > >4. Ozone's performance is at par with HDFS when running workloads like > >Hive ( > > > > > > > https://blog.cloudera.com/benchmarking-ozone-clouderas-next-generation-storage-f > > or-cdp/ > > < > https://blog.cloudera.com/benchmarking-ozone-clouderas-next-generation-storage-for-cdp/ > > > >) > >5. We have been able to run long-running clusters with Ozone. > > > > > > Having achieved these goals, I propose that we move from the planned > > 0.4.2-Alpha release to 0.5.0-Beta as our next release. If we hear no > > concerns about this, we would like to move Ozone from Alpha to Beta > > releases. > > > > > > Thanks > > > > Anu > > > > > > P.S. I am CC-ing HDFS dev since many people who are interested in Ozone > > still have not subscribed to Ozone dev lists. My apologies if it feels > like > > spam, I promise that over time we will become less noisy in the HDFS > > channel. > > > > > > PPS. I know lots of you will want to know more specifics; Our blog > presses > > are working overtime and I promise you that you will get to see all the > > details pretty soon. > > > > > > >
Re: [Discuss] Ozone moving to Beta tag
+1. Given massive improvements in performance and stability, ozone is ready for beta. On Wed, Feb 19, 2020 at 1:04 PM Salvatore LaMendola (BLOOMBERG/ 919 3RD A) < slamendo...@bloomberg.net> wrote: > +1 on moving to beta. This move makes sense to me. > > I've tested each point release so far in 0.4, including building several > master snapshots along the way, and I agree with what Anu is saying below > with regard to Ozone's stability having improved beyond "alpha" state. > > > From: aengin...@apache.org At: 02/19/20 15:17:38To: > ozone-...@hadoop.apache.org, hdfs-dev@hadoop.apache.org > Subject: [Discuss] Ozone moving to Beta tag > > Hi All, > > > I would like to propose moving Ozone from 'Alpha' tags to 'Beta' tags when > we do future releases. Here are a couple of reasons why I think we should > make this move. > > >1. Ozone Manager or the Namenode for Ozone scales to more than 1 billion >keys. We tested this in our labs in an organic fashion; that is, we were >able to create more than 1 billion keys from external clients with no > loss >in performance. >2. The ozone Manager meets the performance and resource constraints that >we set out to achieve. We were able to sustain the same throughput at > Ozone >manager for over three days that took us to get this 1 billion keys. > That >is, we did not have to shut down or resize memory for the namenode as we >went through this exercise. >3. The most critical, we did this experiment with 64GB of memory >allocation in JVM and 64 GB of RAM off-heap allocation. That is, the > Ozone >Manager was able to achieve this scale with far less memory footprint > than >HDFS. >4. Ozone's performance is at par with HDFS when running workloads like >Hive ( > > > https://blog.cloudera.com/benchmarking-ozone-clouderas-next-generation-storage-f > or-cdp/ > <https://blog.cloudera.com/benchmarking-ozone-clouderas-next-generation-storage-for-cdp/> >) >5. We have been able to run long-running clusters with Ozone. > > > Having achieved these goals, I propose that we move from the planned > 0.4.2-Alpha release to 0.5.0-Beta as our next release. If we hear no > concerns about this, we would like to move Ozone from Alpha to Beta > releases. > > > Thanks > > Anu > > > P.S. I am CC-ing HDFS dev since many people who are interested in Ozone > still have not subscribed to Ozone dev lists. My apologies if it feels like > spam, I promise that over time we will become less noisy in the HDFS > channel. > > > PPS. I know lots of you will want to know more specifics; Our blog presses > are working overtime and I promise you that you will get to see all the > details pretty soon. > > >
Re:[Discuss] Ozone moving to Beta tag
+1 on moving to beta. This move makes sense to me. I've tested each point release so far in 0.4, including building several master snapshots along the way, and I agree with what Anu is saying below with regard to Ozone's stability having improved beyond "alpha" state. From: aengin...@apache.org At: 02/19/20 15:17:38To: ozone-...@hadoop.apache.org, hdfs-dev@hadoop.apache.org Subject: [Discuss] Ozone moving to Beta tag Hi All, I would like to propose moving Ozone from 'Alpha' tags to 'Beta' tags when we do future releases. Here are a couple of reasons why I think we should make this move. 1. Ozone Manager or the Namenode for Ozone scales to more than 1 billion keys. We tested this in our labs in an organic fashion; that is, we were able to create more than 1 billion keys from external clients with no loss in performance. 2. The ozone Manager meets the performance and resource constraints that we set out to achieve. We were able to sustain the same throughput at Ozone manager for over three days that took us to get this 1 billion keys. That is, we did not have to shut down or resize memory for the namenode as we went through this exercise. 3. The most critical, we did this experiment with 64GB of memory allocation in JVM and 64 GB of RAM off-heap allocation. That is, the Ozone Manager was able to achieve this scale with far less memory footprint than HDFS. 4. Ozone's performance is at par with HDFS when running workloads like Hive ( https://blog.cloudera.com/benchmarking-ozone-clouderas-next-generation-storage-f or-cdp/ ) 5. We have been able to run long-running clusters with Ozone. Having achieved these goals, I propose that we move from the planned 0.4.2-Alpha release to 0.5.0-Beta as our next release. If we hear no concerns about this, we would like to move Ozone from Alpha to Beta releases. Thanks Anu P.S. I am CC-ing HDFS dev since many people who are interested in Ozone still have not subscribed to Ozone dev lists. My apologies if it feels like spam, I promise that over time we will become less noisy in the HDFS channel. PPS. I know lots of you will want to know more specifics; Our blog presses are working overtime and I promise you that you will get to see all the details pretty soon.
[Discuss] Ozone moving to Beta tag
Hi All, I would like to propose moving Ozone from 'Alpha' tags to 'Beta' tags when we do future releases. Here are a couple of reasons why I think we should make this move. 1. Ozone Manager or the Namenode for Ozone scales to more than 1 billion keys. We tested this in our labs in an organic fashion; that is, we were able to create more than 1 billion keys from external clients with no loss in performance. 2. The ozone Manager meets the performance and resource constraints that we set out to achieve. We were able to sustain the same throughput at Ozone manager for over three days that took us to get this 1 billion keys. That is, we did not have to shut down or resize memory for the namenode as we went through this exercise. 3. The most critical, we did this experiment with 64GB of memory allocation in JVM and 64 GB of RAM off-heap allocation. That is, the Ozone Manager was able to achieve this scale with far less memory footprint than HDFS. 4. Ozone's performance is at par with HDFS when running workloads like Hive ( https://blog.cloudera.com/benchmarking-ozone-clouderas-next-generation-storage-for-cdp/ ) 5. We have been able to run long-running clusters with Ozone. Having achieved these goals, I propose that we move from the planned 0.4.2-Alpha release to 0.5.0-Beta as our next release. If we hear no concerns about this, we would like to move Ozone from Alpha to Beta releases. Thanks Anu P.S. I am CC-ing HDFS dev since many people who are interested in Ozone still have not subscribed to Ozone dev lists. My apologies if it feels like spam, I promise that over time we will become less noisy in the HDFS channel. PPS. I know lots of you will want to know more specifics; Our blog presses are working overtime and I promise you that you will get to see all the details pretty soon.
