Re: Official Apache Slack Channel for Hadoop projects

[Yufei Gu]

Thanks Wei-Chiu. Just join both hdfs and yarn channel. Yes, there is a yarn
channel. There are only 3 members in the yarn channel.

Best,

Yufei

`This is not a contribution`


On Fri, Oct 11, 2019 at 4:35 PM Wei-Chiu Chuang  wrote:

> Hi Hadoop devs,
>
> In case you don't know, there is an official ASF slack, and there's a HDFS
> channel in it. This is the slack workplace managed by Apache Infra.
>
> Please see this wiki to get invite:
> https://cwiki.apache.org/confluence/display/INFRA/Slack+Guest+Invites or
> DM
> me to get an invite.
>
> Once you get access to the ASF workplace, search for #hdfs channel. There
> is also an #ozone channel, #hadoop, #submarine-dev, #submarine-user. I
> don't see a #yarn channel, but I can create one (not sure if who is
> eligible for creating channels, PMC or committers or any one?)
>
> We will not use Slack channel to vote on project decisions/vote, but it
> might be an easier way to find me. Right now the channels are quite dry.
> Let's see if we can revive them.
>
> Weichiu
>

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/472/

No changes

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14904) Balancer should pick nodes based on utilization in each iteration

[Leon Gao (Jira)]

Leon Gao created HDFS-14904:
---

 Summary: Balancer should pick nodes based on utilization in each 
iteration
 Key: HDFS-14904
 URL: https://issues.apache.org/jira/browse/HDFS-14904
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: balancer  mover
Reporter: Leon Gao
Assignee: Leon Gao


In each iteration, balancer should pick nodes with the highest/lowest usage 
first.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop Ozone 0.4.1-alpha

[Hanisha Koneru]

Thank you Nanda for putting up the RC.

+1 binding.

Verified the following:
  - Built from source
  - Deployed to 5 node cluster and ran smoke tests.
  - Ran sanity checks

Thanks
Hanisha

> On Oct 4, 2019, at 10:42 AM, Nanda kumar  wrote:
> 
> Hi Folks,
> 
> I have put together RC0 for Apache Hadoop Ozone 0.4.1-alpha.
> 
> The artifacts are at:
> https://home.apache.org/~nanda/ozone/release/0.4.1/RC0/
> 
> The maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1238/
> 
> The RC tag in git is at:
> https://github.com/apache/hadoop/tree/ozone-0.4.1-alpha-RC0
> 
> And the public key used for signing the artifacts can be found at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> This release contains 363 fixes/improvements [1].
> Thanks to everyone who put in the effort to make this happen.
> 
> *The vote will run for 7 days, ending on October 11th at 11:59 pm IST.*
> Note: This release is alpha quality, it’s not recommended to use in
> production but we believe that it’s stable enough to try out the feature
> set and collect feedback.
> 
> 
> [1] https://s.apache.org/yfudc
> 
> Thanks,
> Team Ozone

Official Apache Slack Channel for Hadoop projects

[Wei-Chiu Chuang]

Hi Hadoop devs,

In case you don't know, there is an official ASF slack, and there's a HDFS
channel in it. This is the slack workplace managed by Apache Infra.

Please see this wiki to get invite:
https://cwiki.apache.org/confluence/display/INFRA/Slack+Guest+Invites or DM
me to get an invite.

Once you get access to the ASF workplace, search for #hdfs channel. There
is also an #ozone channel, #hadoop, #submarine-dev, #submarine-user. I
don't see a #yarn channel, but I can create one (not sure if who is
eligible for creating channels, PMC or committers or any one?)

We will not use Slack channel to vote on project decisions/vote, but it
might be an easier way to find me. Right now the channels are quite dry.
Let's see if we can revive them.

Weichiu

[jira] [Created] (HDDS-2286) Add a log info in ozone client and scm to print the exclusion list during allocate block

[Shashikant Banerjee (Jira)]

Shashikant Banerjee created HDDS-2286:
-

 Summary: Add a log info in ozone client and scm to print the 
exclusion list during allocate block
 Key: HDDS-2286
 URL: https://issues.apache.org/jira/browse/HDDS-2286
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Affects Versions: 0.5.0
Reporter: Shashikant Banerjee






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Hadoop 2.10.0 release plan

[Jonathan Hung]

Edit: seems a 2.10.0 blocker was reopened (HDFS-14305). I'll continue
watching this jira and start the release once this is resolved.

Jonathan Hung


On Thu, Oct 10, 2019 at 5:13 PM Jonathan Hung  wrote:

> Hi folks, as of now all 2.10.0 blockers have been resolved [1]. So I'll
> start the release process soon (cutting branches, updating target versions,
> etc).
>
> [1] https://issues.apache.org/jira/issues/?filter=12346975
>
> Jonathan Hung
>
>
> On Mon, Aug 26, 2019 at 10:19 AM Jonathan Hung 
> wrote:
>
>> Hi folks,
>>
>> As discussed previously (e.g. [1], [2]) we'd like to do a 2.10.0 release
>> soon. Some features/big-items we're targeting for this release:
>>
>>- YARN resource types/GPU support (YARN-8200
>>)
>>- Selective wire encryption (HDFS-13541
>>)
>>- Rolling upgrade support from 2.x to 3.x (e.g. HDFS-14509
>>)
>>
>> Per [3] sounds like there's concern around upgrading dependencies as well.
>>
>> We created a public jira filter here (
>> https://issues.apache.org/jira/issues/?filter=12346975) marking all
>> blockers for 2.10.0 release. If you have other jiras that should be 2.10.0
>> blockers, please mark "Target Version/s" as "2.10.0" and add label
>> "release-blocker" so we can track it through this filter.
>>
>> We're targeting a release at end of September.
>>
>> Please share any thoughts you have about this. Thanks!
>>
>> [1] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg29461.html
>> [2]
>> https://www.mail-archive.com/mapreduce-dev@hadoop.apache.org/msg21293.html
>> [3] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg33440.html
>>
>>
>> Jonathan Hung
>>
>

[jira] [Resolved] (HDDS-2213) Reduce key provider loading log level in OzoneFileSystem#getAdditionalTokenIssuers

[Arpit Agarwal (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal resolved HDDS-2213.
-
Fix Version/s: 0.5.0
   Resolution: Fixed

I've merged this.

> Reduce key provider loading log level in 
> OzoneFileSystem#getAdditionalTokenIssuers
> --
>
> Key: HDDS-2213
> URL: https://issues.apache.org/jira/browse/HDDS-2213
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Vivek Ratnavel Subramanian
>Assignee: Shweta
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> OzoneFileSystem#getAdditionalTokenIssuers log an error when secure client 
> tries to collect ozone delegation token to run MR/Spark jobs but ozone file 
> system does not have a kms provider configured. In this case, we simply 
> return null provider here in the code below. This is a benign error and we 
> should reduce the log level to debug level.
> {code:java}
> KeyProvider keyProvider;
>  try {
>   keyProvider = getKeyProvider(); }
> catch (IOException ioe) {
>   LOG.error("Error retrieving KeyProvider.", ioe);
>   return null;
> }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [EXTERNAL] Re: [VOTE] Release Apache Hadoop Ozone 0.4.1-alpha

[Dinesh Chitlangia]

+1 (non-binding)
-Verified signatures
-Built from sources
-Smoke tests passes
-Verified audit logs

Thank you Nanda for organizing this release.

-Dinesh



On Fri, Oct 11, 2019 at 11:47 AM Matthew Sharp  wrote:

> +1 (non-binding)
>
>
> - Started the pseudo docker compose cluster.  Ran ozone freon and looked
> through UI metrics and configurations
>
> - Ran the full smoke test suite that Martin mentioned below, noticed a few
> warnings but all tests passed
>
>
>
>
>
>
> On 10/10/19, 5:36 AM, "Elek, Marton"  wrote:
>
>
>
>
>
> +1
>
>
>
> Thank you Nanda the enormous work to make this release happen.
>
>
>
>
>
>
>
>   * GPG Signatures are fine
>
>   * SHA512 signatures are fine
>
>   * Can be built from the source package (in isolated environment
>
> without cached hadoop/ozone artifacts)
>
>   * Started the pseudo cluster with `compose/ozone`
>
>   * Executed FULL smoke-test suite (`cd compose && ./test-all.sh`) ALL
>
> passed except some intermittent issues:
>
> * kinit step was failed due to timeout but after that all the
> secure
>
> testss are passed. I think my laptop was too slow... + I had other CPU
>
> sensitive tasks in the mean time
>
>   * Tested to create apache/hadoop-ozone:0.4.1 image
>
>   * Using hadoop-docker-ozone/Dockerfile [1]
>
>   * Started single, one node cluster + tested with AWS cli
>
> (REDUCED_REDUNDANCY) (`docker run elek/ozone:test`)
>
>   * Started pseudo cluster (`docker run elek/ozone:test cat
>
> docker-compose.yaml && docker run elek/ozone:test cat docker-config`)
>
>   * Tested with kubernetes:
>
> * Used the image which is created earlier
>
> * Replaced the images under kubernetes/examples/minikube
>
> * Started with kubectl `kubectl apply -f` to k3s (3!) cluster
>
> * Tested with `ozone sh` commands (put/get keys)
>
>
>
>
>
> Marton
>
>
>
> [1]:
>
> ```
>
> docker build --build-arg
>
> OZONE_URL=
>
> https://home.apache.org/~nanda/ozone/release/0.4.1/RC0/hadoop-ozone-0.4.1-alpha.tar.gz
>
> -t elek/ozone-test .
>
> ```
>
>
>
> On 10/4/19 7:42 PM, Nanda kumar wrote:
>
> > Hi Folks,
>
> >
>
> > I have put together RC0 for Apache Hadoop Ozone 0.4.1-alpha.
>
> >
>
> > The artifacts are at:
>
> > https://home.apache.org/~nanda/ozone/release/0.4.1/RC0/
>
> >
>
> > The maven artifacts are staged at:
>
> >
> https://repository.apache.org/content/repositories/orgapachehadoop-1238/
>
> >
>
> > The RC tag in git is at:
>
> > https://github.com/apache/hadoop/tree/ozone-0.4.1-alpha-RC0
>
> >
>
> > And the public key used for signing the artifacts can be found at:
>
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> >
>
> > This release contains 363 fixes/improvements [1].
>
> > Thanks to everyone who put in the effort to make this happen.
>
> >
>
> > *The vote will run for 7 days, ending on October 11th at 11:59 pm
> IST.*
>
> > Note: This release is alpha quality, it’s not recommended to use in
>
> > production but we believe that it’s stable enough to try out the
> feature
>
> > set and collect feedback.
>
> >
>
> >
>
> > [1] https://s.apache.org/yfudc
>
> >
>
> > Thanks,
>
> > Team Ozone
>
> >
>
>
>
> -
>
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>

Re: [EXTERNAL] Re: [VOTE] Release Apache Hadoop Ozone 0.4.1-alpha

[Matthew Sharp]

+1 (non-binding)


- Started the pseudo docker compose cluster.  Ran ozone freon and looked
through UI metrics and configurations

- Ran the full smoke test suite that Martin mentioned below, noticed a few
warnings but all tests passed






On 10/10/19, 5:36 AM, "Elek, Marton"  wrote:





+1



Thank you Nanda the enormous work to make this release happen.







  * GPG Signatures are fine

  * SHA512 signatures are fine

  * Can be built from the source package (in isolated environment

without cached hadoop/ozone artifacts)

  * Started the pseudo cluster with `compose/ozone`

  * Executed FULL smoke-test suite (`cd compose && ./test-all.sh`) ALL

passed except some intermittent issues:

* kinit step was failed due to timeout but after that all the
secure

testss are passed. I think my laptop was too slow... + I had other CPU

sensitive tasks in the mean time

  * Tested to create apache/hadoop-ozone:0.4.1 image

  * Using hadoop-docker-ozone/Dockerfile [1]

  * Started single, one node cluster + tested with AWS cli

(REDUCED_REDUNDANCY) (`docker run elek/ozone:test`)

  * Started pseudo cluster (`docker run elek/ozone:test cat

docker-compose.yaml && docker run elek/ozone:test cat docker-config`)

  * Tested with kubernetes:

* Used the image which is created earlier

* Replaced the images under kubernetes/examples/minikube

* Started with kubectl `kubectl apply -f` to k3s (3!) cluster

* Tested with `ozone sh` commands (put/get keys)





Marton



[1]:

```

docker build --build-arg

OZONE_URL=
https://home.apache.org/~nanda/ozone/release/0.4.1/RC0/hadoop-ozone-0.4.1-alpha.tar.gz

-t elek/ozone-test .

```



On 10/4/19 7:42 PM, Nanda kumar wrote:

> Hi Folks,

>

> I have put together RC0 for Apache Hadoop Ozone 0.4.1-alpha.

>

> The artifacts are at:

> https://home.apache.org/~nanda/ozone/release/0.4.1/RC0/

>

> The maven artifacts are staged at:

>
https://repository.apache.org/content/repositories/orgapachehadoop-1238/

>

> The RC tag in git is at:

> https://github.com/apache/hadoop/tree/ozone-0.4.1-alpha-RC0

>

> And the public key used for signing the artifacts can be found at:

> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

>

> This release contains 363 fixes/improvements [1].

> Thanks to everyone who put in the effort to make this happen.

>

> *The vote will run for 7 days, ending on October 11th at 11:59 pm
IST.*

> Note: This release is alpha quality, it’s not recommended to use in

> production but we believe that it’s stable enough to try out the
feature

> set and collect feedback.

>

>

> [1] https://s.apache.org/yfudc

>

> Thanks,

> Team Ozone

>



-

To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org

For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Reopened] (HDFS-14305) Serial number in BlockTokenSecretManager could overlap between different namenodes

[Arpit Agarwal (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal reopened HDFS-14305:
--

Reopening because this still needs to be fixed correctly.

> Serial number in BlockTokenSecretManager could overlap between different 
> namenodes
> --
>
> Key: HDFS-14305
> URL: https://issues.apache.org/jira/browse/HDFS-14305
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode, security
>Reporter: Chao Sun
>Assignee: Konstantin Shvachko
>Priority: Major
>  Labels: multi-sbnn, release-blocker
> Fix For: 2.10.0, 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-14305-007.patch, HDFS-14305-008.patch, 
> HDFS-14305.001.patch, HDFS-14305.002.patch, HDFS-14305.003.patch, 
> HDFS-14305.004.patch, HDFS-14305.005.patch, HDFS-14305.006.patch
>
>
> Currently, a {{BlockTokenSecretManager}} starts with a random integer as the 
> initial serial number, and then use this formula to rotate it:
> {code:java}
> this.intRange = Integer.MAX_VALUE / numNNs;
> this.nnRangeStart = intRange * nnIndex;
> this.serialNo = (this.serialNo % intRange) + (nnRangeStart);
>  {code}
> while {{numNNs}} is the total number of NameNodes in the cluster, and 
> {{nnIndex}} is the index of the current NameNode specified in the 
> configuration {{dfs.ha.namenodes.}}.
> However, with this approach, different NameNode could have overlapping ranges 
> for serial number. For simplicity, let's assume {{Integer.MAX_VALUE}} is 100, 
> and we have 2 NameNodes {{nn1}} and {{nn2}} in configuration. Then the ranges 
> for these two are:
> {code}
> nn1 -> [-49, 49]
> nn2 -> [1, 99]
> {code}
> This is because the initial serial number could be any negative integer.
> Moreover, when the keys are updated, the serial number will again be updated 
> with the formula:
> {code}
> this.serialNo = (this.serialNo % intRange) + (nnRangeStart);
> {code}
> which means the new serial number could be updated to a range that belongs to 
> a different NameNode, thus increasing the chance of collision again.
> When the collision happens, DataNodes could overwrite an existing key which 
> will cause clients to fail because of {{InvalidToken}} error.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2285) GetBlock and RadChunk command from the client should be sent to the same datanode to re-use the same connection

[Mukul Kumar Singh (Jira)]

Mukul Kumar Singh created HDDS-2285:
---

 Summary: GetBlock and RadChunk command from the client should be 
sent to the same datanode to re-use the same connection
 Key: HDDS-2285
 URL: https://issues.apache.org/jira/browse/HDDS-2285
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: Ozone Client
Reporter: Mukul Kumar Singh


I can be observed that the GetBlock and ReadChunk command is sent to 2 
different datanodes. It should be sent to the same datanode to re-use the 
connection.

{code}
19/10/10 00:43:42 INFO scm.XceiverClientGrpc: Send command GetBlock to datanode 
172.26.32.224
19/10/10 00:43:42 INFO scm.XceiverClientGrpc: Send command ReadChunk to 
datanode 172.26.32.231
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2284) XceiverClientMetrics should be initialised as part of XceiverClientManager constructor

[Mukul Kumar Singh (Jira)]

Mukul Kumar Singh created HDDS-2284:
---

 Summary: XceiverClientMetrics should be initialised as part of 
XceiverClientManager constructor
 Key: HDDS-2284
 URL: https://issues.apache.org/jira/browse/HDDS-2284
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: Ozone Client
Affects Versions: 0.4.0
Reporter: Mukul Kumar Singh


XceiverClientMetrics is currently initialized in the read write path, the 
metric should be initialized while creating XceiverClientManager



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-2283) Container Creation on datanodes take around 300ms due to rocksdb creation

[Mukul Kumar Singh (Jira)]

Mukul Kumar Singh created HDDS-2283:
---

 Summary: Container Creation on datanodes take around 300ms due to 
rocksdb creation
 Key: HDDS-2283
 URL: https://issues.apache.org/jira/browse/HDDS-2283
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: Ozone Datanode
Reporter: Mukul Kumar Singh


Container Creation on datanodes take around 300ms due to rocksdb creation. 
Rocksdb creation is taking a considerable time and this needs to be optimized.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org