[jira] [Assigned] (HDFS-15304) Infinite loop between DN and NN at rare condition

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-15304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth reassigned HDFS-15304:
---

Assignee: Istvan Fajth

> Infinite loop between DN and NN at rare condition
> -
>
> Key: HDFS-15304
> URL: https://issues.apache.org/jira/browse/HDFS-15304
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Minor
>
> During the investigation lead to HDFS-15303, we have identified the following 
> infinite loop between the DNs affected by the data directory layout problem:
> - for a particular misplaced block, the VolumeScanner finds the block file, 
> and realizes that it is not part of the block map
> - the block is added to the block map
> - at the next FBR the block is reported to the NN
> - the NN finds that the block should have been deleted already, as the 
> corresponding inode was already deleted
> - NN issues the deletion of the block on the DataNode
> - DataNode runs the delete routine, but that fails to delete anything 
> silently as it is trying to delete the block from the wrong internal subdir 
> that is calculated based on the block id with a different algorythm.
> - block is removed from the blockmap
> - VolumeScanner finds the block again, and adds it back to the blockmap
> The problem can happen only when there is a mixed layout on the DataNode due 
> to some issue, and there are blocks in a subdir correct according to Hadoop2 
> format, but the DN is already hadoop3, or vice versa if the problematic 
> layout born during a rollback. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-15304) Infinite loop between DN and NN at rare condition

[Istvan Fajth (Jira)]

Istvan Fajth created HDFS-15304:
---

 Summary: Infinite loop between DN and NN at rare condition
 Key: HDFS-15304
 URL: https://issues.apache.org/jira/browse/HDFS-15304
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Istvan Fajth


During the investigation lead to HDFS-15303, we have identified the following 
infinite loop between the DNs affected by the data directory layout problem:
- for a particular misplaced block, the VolumeScanner finds the block file, and 
realizes that it is not part of the block map
- the block is added to the block map
- at the next FBR the block is reported to the NN
- the NN finds that the block should have been deleted already, as the 
corresponding inode was already deleted
- NN issues the deletion of the block on the DataNode
- DataNode runs the delete routine, but that fails to delete anything silently 
as it is trying to delete the block from the wrong internal subdir that is 
calculated based on the block id with a different algorythm.
- block is removed from the blockmap
- VolumeScanner finds the block again, and adds it back to the blockmap

The problem can happen only when there is a mixed layout on the DataNode due to 
some issue, and there are blocks in a subdir correct according to Hadoop2 
format, but the DN is already hadoop3, or vice versa if the problematic layout 
born during a rollback. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDFS-15303) Provide a tool that can validate/fix the block file placement in DataNode data directories

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-15303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth reassigned HDFS-15303:
---

Assignee: Istvan Fajth

> Provide a tool that can validate/fix the block file placement in DataNode 
> data directories
> --
>
> Key: HDFS-15303
> URL: https://issues.apache.org/jira/browse/HDFS-15303
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Minor
>
> We recently run into an issue where during an upgrade from Hadoop2 to 
> Hadoop3, the filesystem under DataNode data directories was left in an 
> intermediate state, and part of the directories were in Hadoop2 format other 
> parts in Hadoop3 format.
> At first we had to rollback the upgrade, and after the rollback we started to 
> see FileNotFoundExceptions for particular block files.
> The exception was logged on the DataNodes and failed the jobs as well 
> sometimes. HDFS NameNode did not show any missing blocks, and we found the 
> block files and meta files also in the DataNode's data directories, but at a 
> different location.
> This was the point when we realized that something went wrong during the 
> rollback, and some of the data directories had blocks placed according to 
> Hadoop3 rules, while other were placed according to Hadoop2 rules. We suspect 
> a possible premature DataNode shutdown or an unknown failure during the 
> rollback, but at the point when we realized what is the issue and could check 
> into things, we already ran out of the logs that would have been able to show 
> us the cause.
> This JIRA is to suggest two new commands that can help administrators in this 
> situation, to validate the data directory and ensure that blocks are placed 
> correctly according to the rules, and to fix the data directory layout if 
> needed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-15303) Provide a tool that can validate/fix the block file placement in DataNode data directories

[Istvan Fajth (Jira)]

Istvan Fajth created HDFS-15303:
---

 Summary: Provide a tool that can validate/fix the block file 
placement in DataNode data directories
 Key: HDFS-15303
 URL: https://issues.apache.org/jira/browse/HDFS-15303
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Istvan Fajth


We recently run into an issue where during an upgrade from Hadoop2 to Hadoop3, 
the filesystem under DataNode data directories was left in an intermediate 
state, and part of the directories were in Hadoop2 format other parts in 
Hadoop3 format.

At first we had to rollback the upgrade, and after the rollback we started to 
see FileNotFoundExceptions for particular block files.
The exception was logged on the DataNodes and failed the jobs as well 
sometimes. HDFS NameNode did not show any missing blocks, and we found the 
block files and meta files also in the DataNode's data directories, but at a 
different location.

This was the point when we realized that something went wrong during the 
rollback, and some of the data directories had blocks placed according to 
Hadoop3 rules, while other were placed according to Hadoop2 rules. We suspect a 
possible premature DataNode shutdown or an unknown failure during the rollback, 
but at the point when we realized what is the issue and could check into 
things, we already ran out of the logs that would have been able to show us the 
cause.

This JIRA is to suggest two new commands that can help administrators in this 
situation, to validate the data directory and ensure that blocks are placed 
correctly according to the rules, and to fix the data directory layout if 
needed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-15122) Log all balancer related parameters at Balancer startup

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-15122:

Description: 
Currently Balancer just logs the parameters it deals with.
It would be good to emit all the Balancer related parameters into its log, it 
would be easier to see all at once when checking into any problem regarding 
balancing.

The maximum balancing bandwidth is one of the missing configuration values that 
are not there, but other related parameters should be added as well all at once 
in this effort.

  was:
Currently Balancer just logs the parameters it deals with.
It would be good to emit all the Balancer related parameter into its log, it 
would be easier to see all at once when checking into any problem regarding 
balancing.

The maximum balancing bandwidth is one of the missing configuration values that 
are not there, but other related parameters should be added as well all at once 
in this effort.


> Log all balancer related parameters at Balancer startup
> ---
>
> Key: HDFS-15122
> URL: https://issues.apache.org/jira/browse/HDFS-15122
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: balancer  mover
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Minor
>  Labels: balancer
>
> Currently Balancer just logs the parameters it deals with.
> It would be good to emit all the Balancer related parameters into its log, it 
> would be easier to see all at once when checking into any problem regarding 
> balancing.
> The maximum balancing bandwidth is one of the missing configuration values 
> that are not there, but other related parameters should be added as well all 
> at once in this effort.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-15122) Log all balancer related parameters at Balancer startup

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-15122:

Component/s: balancer & mover

> Log all balancer related parameters at Balancer startup
> ---
>
> Key: HDFS-15122
> URL: https://issues.apache.org/jira/browse/HDFS-15122
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: balancer  mover
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Minor
>  Labels: balancer
>
> Currently Balancer just logs the parameters it deals with.
> It would be good to emit all the Balancer related parameter into its log, it 
> would be easier to see all at once when checking into any problem regarding 
> balancing.
> The maximum balancing bandwidth is one of the missing configuration values 
> that are not there, but other related parameters should be added as well all 
> at once in this effort.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-15122) Log all balancer related parameters at Balancer startup

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-15122:

Labels: balancer  (was: )

> Log all balancer related parameters at Balancer startup
> ---
>
> Key: HDFS-15122
> URL: https://issues.apache.org/jira/browse/HDFS-15122
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Minor
>  Labels: balancer
>
> Currently Balancer just logs the parameters it deals with.
> It would be good to emit all the Balancer related parameter into its log, it 
> would be easier to see all at once when checking into any problem regarding 
> balancing.
> The maximum balancing bandwidth is one of the missing configuration values 
> that are not there, but other related parameters should be added as well all 
> at once in this effort.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDFS-15122) Log all balancer related parameters at Balancer startup

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth reassigned HDFS-15122:
---

Assignee: Istvan Fajth

> Log all balancer related parameters at Balancer startup
> ---
>
> Key: HDFS-15122
> URL: https://issues.apache.org/jira/browse/HDFS-15122
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Minor
>
> Currently Balancer just logs the parameters it deals with.
> It would be good to emit all the Balancer related parameter into its log, it 
> would be easier to see all at once when checking into any problem regarding 
> balancing.
> The maximum balancing bandwidth is one of the missing configuration values 
> that are not there, but other related parameters should be added as well all 
> at once in this effort.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-15122) Log all balancer related parameters at Balancer startup

[Istvan Fajth (Jira)]

Istvan Fajth created HDFS-15122:
---

 Summary: Log all balancer related parameters at Balancer startup
 Key: HDFS-15122
 URL: https://issues.apache.org/jira/browse/HDFS-15122
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Istvan Fajth


Currently Balancer just logs the parameters it deals with.
It would be good to emit all the Balancer related parameter into its log, it 
would be easier to see all at once when checking into any problem regarding 
balancing.

The maximum balancing bandwidth is one of the missing configuration values that 
are not there, but other related parameters should be added as well all at once 
in this effort.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-14668) Support Fuse with Users from multiple Security Realms

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-14668:

Description: 
UPDATE:
See 
[this|https://issues.apache.org/jira/browse/HDFS-14668?focusedCommentId=16979466=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16979466]
 comment for the complete description of what is happening here.


Users from non-default  krb5 domain can't use hadoop-fuse.
There are 2 Realms with kdc. 
-one realm is for human users  (USERS.COM.US) 
-the other is for service principals.   (SERVICE.COM.US) 
Cross realm trust is setup.
In krb5.conf  the default domain  is set to SERVICE.COM.US

Users within USERS.COM.US Realm are not able to put any files to Fuse mounted 
location

The client shows:
  cp: cannot create regular file ‘/hdfs_mount/tmp/hello_from_fuse.txt’: 
Input/output error

  was:
UPDATE:
See this comment for the complete description of what is happening here.


Users from non-default  krb5 domain can't use hadoop-fuse.
There are 2 Realms with kdc. 
-one realm is for human users  (USERS.COM.US) 
-the other is for service principals.   (SERVICE.COM.US) 
Cross realm trust is setup.
In krb5.conf  the default domain  is set to SERVICE.COM.US

Users within USERS.COM.US Realm are not able to put any files to Fuse mounted 
location

The client shows:
  cp: cannot create regular file ‘/hdfs_mount/tmp/hello_from_fuse.txt’: 
Input/output error


> Support Fuse with Users from multiple Security Realms
> -
>
> Key: HDFS-14668
> URL: https://issues.apache.org/jira/browse/HDFS-14668
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fuse-dfs
>Affects Versions: 3.1.0, 3.0.3
>Reporter: Sailesh Patel
>Assignee: Istvan Fajth
>Priority: Critical
>  Labels: regression
>
> UPDATE:
> See 
> [this|https://issues.apache.org/jira/browse/HDFS-14668?focusedCommentId=16979466=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16979466]
>  comment for the complete description of what is happening here.
> Users from non-default  krb5 domain can't use hadoop-fuse.
> There are 2 Realms with kdc. 
> -one realm is for human users  (USERS.COM.US) 
> -the other is for service principals.   (SERVICE.COM.US) 
> Cross realm trust is setup.
> In krb5.conf  the default domain  is set to SERVICE.COM.US
> Users within USERS.COM.US Realm are not able to put any files to Fuse mounted 
> location
> The client shows:
>   cp: cannot create regular file ‘/hdfs_mount/tmp/hello_from_fuse.txt’: 
> Input/output error



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-14668) Support Fuse with Users from multiple Security Realms

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-14668:

Description: 
UPDATE:
See this comment for the complete description of what is happening here.


Users from non-default  krb5 domain can't use hadoop-fuse.
There are 2 Realms with kdc. 
-one realm is for human users  (USERS.COM.US) 
-the other is for service principals.   (SERVICE.COM.US) 
Cross realm trust is setup.
In krb5.conf  the default domain  is set to SERVICE.COM.US

Users within USERS.COM.US Realm are not able to put any files to Fuse mounted 
location

The client shows:
  cp: cannot create regular file ‘/hdfs_mount/tmp/hello_from_fuse.txt’: 
Input/output error

  was:
Users from non-default  krb5 domain can't use hadoop-fuse.
There are 2 Realms with kdc. 
-one realm is for human users  (USERS.COM.US) 
-the other is for service principals.   (SERVICE.COM.US) 
Cross realm trust is setup.
In krb5.conf  the default domain  is set to SERVICE.COM.US

Users within USERS.COM.US Realm are not able to put any files to Fuse mounted 
location

The client shows:
  cp: cannot create regular file ‘/hdfs_mount/tmp/hello_from_fuse.txt’: 
Input/output error


> Support Fuse with Users from multiple Security Realms
> -
>
> Key: HDFS-14668
> URL: https://issues.apache.org/jira/browse/HDFS-14668
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fuse-dfs
>Affects Versions: 3.1.0, 3.0.3
>Reporter: Sailesh Patel
>Assignee: Istvan Fajth
>Priority: Critical
>  Labels: regression
>
> UPDATE:
> See this comment for the complete description of what is happening here.
> Users from non-default  krb5 domain can't use hadoop-fuse.
> There are 2 Realms with kdc. 
> -one realm is for human users  (USERS.COM.US) 
> -the other is for service principals.   (SERVICE.COM.US) 
> Cross realm trust is setup.
> In krb5.conf  the default domain  is set to SERVICE.COM.US
> Users within USERS.COM.US Realm are not able to put any files to Fuse mounted 
> location
> The client shows:
>   cp: cannot create regular file ‘/hdfs_mount/tmp/hello_from_fuse.txt’: 
> Input/output error



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-14668) Support Fuse with Users from multiple Security Realms

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-14668:

Labels: regression  (was: )

> Support Fuse with Users from multiple Security Realms
> -
>
> Key: HDFS-14668
> URL: https://issues.apache.org/jira/browse/HDFS-14668
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fuse-dfs
>Affects Versions: 3.1.0, 3.0.3
>Reporter: Sailesh Patel
>Assignee: Istvan Fajth
>Priority: Critical
>  Labels: regression
>
> Users from non-default  krb5 domain can't use hadoop-fuse.
> There are 2 Realms with kdc. 
> -one realm is for human users  (USERS.COM.US) 
> -the other is for service principals.   (SERVICE.COM.US) 
> Cross realm trust is setup.
> In krb5.conf  the default domain  is set to SERVICE.COM.US
> Users within USERS.COM.US Realm are not able to put any files to Fuse mounted 
> location
> The client shows:
>   cp: cannot create regular file ‘/hdfs_mount/tmp/hello_from_fuse.txt’: 
> Input/output error



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-14668) Support Fuse with Users from multiple Security Realms

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-14668:

Priority: Critical  (was: Minor)

> Support Fuse with Users from multiple Security Realms
> -
>
> Key: HDFS-14668
> URL: https://issues.apache.org/jira/browse/HDFS-14668
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fuse-dfs
>Reporter: Sailesh Patel
>Assignee: Istvan Fajth
>Priority: Critical
>
> Users from non-default  krb5 domain can't use hadoop-fuse.
> There are 2 Realms with kdc. 
> -one realm is for human users  (USERS.COM.US) 
> -the other is for service principals.   (SERVICE.COM.US) 
> Cross realm trust is setup.
> In krb5.conf  the default domain  is set to SERVICE.COM.US
> Users within USERS.COM.US Realm are not able to put any files to Fuse mounted 
> location
> The client shows:
>   cp: cannot create regular file ‘/hdfs_mount/tmp/hello_from_fuse.txt’: 
> Input/output error



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-14668) Support Fuse with Users from multiple Security Realms

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-14668:

Affects Version/s: 3.1.0
   3.0.3

> Support Fuse with Users from multiple Security Realms
> -
>
> Key: HDFS-14668
> URL: https://issues.apache.org/jira/browse/HDFS-14668
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fuse-dfs
>Affects Versions: 3.1.0, 3.0.3
>Reporter: Sailesh Patel
>Assignee: Istvan Fajth
>Priority: Critical
>
> Users from non-default  krb5 domain can't use hadoop-fuse.
> There are 2 Realms with kdc. 
> -one realm is for human users  (USERS.COM.US) 
> -the other is for service principals.   (SERVICE.COM.US) 
> Cross realm trust is setup.
> In krb5.conf  the default domain  is set to SERVICE.COM.US
> Users within USERS.COM.US Realm are not able to put any files to Fuse mounted 
> location
> The client shows:
>   cp: cannot create regular file ‘/hdfs_mount/tmp/hello_from_fuse.txt’: 
> Input/output error



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-14668) Support Fuse with Users from multiple Security Realms

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-14668:

Issue Type: Bug  (was: Improvement)

> Support Fuse with Users from multiple Security Realms
> -
>
> Key: HDFS-14668
> URL: https://issues.apache.org/jira/browse/HDFS-14668
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fuse-dfs
>Reporter: Sailesh Patel
>Assignee: Istvan Fajth
>Priority: Minor
>
> Users from non-default  krb5 domain can't use hadoop-fuse.
> There are 2 Realms with kdc. 
> -one realm is for human users  (USERS.COM.US) 
> -the other is for service principals.   (SERVICE.COM.US) 
> Cross realm trust is setup.
> In krb5.conf  the default domain  is set to SERVICE.COM.US
> Users within USERS.COM.US Realm are not able to put any files to Fuse mounted 
> location
> The client shows:
>   cp: cannot create regular file ‘/hdfs_mount/tmp/hello_from_fuse.txt’: 
> Input/output error



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-14668) Support Fuse with Users from multiple Security Realms

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-14668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16989730#comment-16989730
 ] 

Istvan Fajth commented on HDFS-14668:
-

After a couple of days thinking, and few hours of testing I decided to come up 
with the given PR.

The main reasons I chose this solution is the following:
- the affected UGI API calls are public, and may be used in other projects, 
where the necessary tunings might already have happened.
- there does not seem to be a good way of deciding whether the given username 
is a valid principal name, and we can not implement FUSE specific solutions in 
the UGI code
- I am not familiar enough with how other projects are using the UGI, this 
phenomenon might cause problems there as well, and I am not sure why it was 
necessary to add the username as a principal all the time from the UGI, and it 
is not clear if this scenario was considered at that time, but without [~daryn] 
I think we might not get this information ever so removing the newly added 
behaviour does not seem to be a good option and can cause troubles in other 
areas.
- this change has the least effect to any other code that has been written


The solution itself changes the connection builder setup, and in case of a 
kerberized environment FUSE does not set the username, which renders the value 
to null on the Java level properly, so that the Java kerberos layer from inside 
the UGI calls will determine the principal's name from the ticket cache 
provided.
In the non-kerberized environments, we still need to provide the username, as 
in that case we are checking permissions against the OS user name, and we don't 
want to loose this inside the FUSE logic either.

While I have been checking this, I came across the fact that inside FUSE we 
could have check and if set use the value of the HADOOP_USER_NAME environment 
variable, but we currently do not use it anywhere. I filed HDFS-15034 for this 
improvement to track it.

> Support Fuse with Users from multiple Security Realms
> -
>
> Key: HDFS-14668
> URL: https://issues.apache.org/jira/browse/HDFS-14668
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: fuse-dfs
>Reporter: Sailesh Patel
>Assignee: Istvan Fajth
>Priority: Minor
>
> Users from non-default  krb5 domain can't use hadoop-fuse.
> There are 2 Realms with kdc. 
> -one realm is for human users  (USERS.COM.US) 
> -the other is for service principals.   (SERVICE.COM.US) 
> Cross realm trust is setup.
> In krb5.conf  the default domain  is set to SERVICE.COM.US
> Users within USERS.COM.US Realm are not able to put any files to Fuse mounted 
> location
> The client shows:
>   cp: cannot create regular file ‘/hdfs_mount/tmp/hello_from_fuse.txt’: 
> Input/output error



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-15034) fuse-dfs does not respect HADOOP_USER_NAME envvar with simple auth

[Istvan Fajth (Jira)]

Istvan Fajth created HDFS-15034:
---

 Summary: fuse-dfs does not respect HADOOP_USER_NAME envvar with 
simple auth
 Key: HDFS-15034
 URL: https://issues.apache.org/jira/browse/HDFS-15034
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: fuse-dfs
Reporter: Istvan Fajth


In the fuse code, there is an explicit map fro the context uid to the username 
on the OS level with the help of getpwuid() system call.
As we have already a way to access the callers environment, to determine the 
kerberos ticket cache path, we can respect the HADOOP_USER_NAME setting in a 
SIMPLE_AUTH based environment, so that the host where the mount is does not 
need to have all the users that are defined and used on HDFS.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-14668) Support Fuse with Users from multiple Security Realms

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-14668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979466#comment-16979466
 ] 

Istvan Fajth commented on HDFS-14668:
-

After an extensive debugging session, I was able to come up with a Java code 
that reproduces the problem, and was able to nail down the problem as I think, 
and this is broken because of two things, one is the API with which FUSE and 
the native HDFS client connects to the FileSystem API, and how that uses 
UserGroupInformation, and a change in the default behaviour of 
UserGroupInformation in HADOOP-9747.

The Java code reproduces the issue:
{code}
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

import java.io.IOException;
import java.net.URI;

public class TestUGITheFUSEWay {
  public static void main(String[] args)
  throws IOException, InterruptedException {

Configuration conf = new Configuration();

URI ns = URI.create("dfs://ns1");
String krbTicketCachePath = "/tmp/krb5cc_1000";
conf.set("hadoop.security.kerberos.ticket.cache.path", krbTicketCachePath);
String userName = "test";

FileSystem fs = FileSystem.newInstance(ns, conf, userName);
Path p= new Path("/");

for (FileStatus s : fs.listStatus(p)){
  System.out.println(s.toString());
}
  }
}
{code}

The underlying problem is with the fact that in case you specify the userName 
to the FileSystem.newInstance call, then in the old word UserGroupInformation 
did not added the userName to the Kerberos configuration entries as the 
principal, so that as the unerlying Java libraries used the principal in the 
ticket cache as the principal.
After HADOOP-9747 if you specify the userName there, it is added to the 
AppConfigurationEntry given to the HadoopLoginContext in the 
HadoopLoginConfiguration as principal, and then the underlying Java libraries 
will throw an indicative exception (if kerberos debugging is on):
{code}
Acquire TGT from Cache
java.io.IOException: Primary principals don't match.
at 
sun.security.krb5.internal.ccache.FileCredentialsCache.load(FileCredentialsCache.java:179)
at 
sun.security.krb5.internal.ccache.FileCredentialsCache.acquireInstance(FileCredentialsCache.java:82)
at 
sun.security.krb5.internal.ccache.CredentialsCache.getInstance(CredentialsCache.java:83)
at 
sun.security.krb5.Credentials.acquireTGTFromCache(Credentials.java:333)
at 
com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:665)
at 
com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at 
javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at 
javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at 
org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2086)
at 
org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1993)
at 
org.apache.hadoop.security.UserGroupInformation.getUGIFromTicketCache(UserGroupInformation.java:654)
at 
org.apache.hadoop.security.UserGroupInformation.getBestUGI(UserGroupInformation.java:624)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:495)
{code}
And ultimately it will fail with:
{code}
LoginException: Unable to obtain password from user
org.apache.hadoop.security.KerberosAuthException: failure to login: for 
principal: systest using ticket cache file: /tmp/krb5cc_2000 
javax.security.auth.login.LoginException: Unable to obtain password from user

at 
org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2008)
at 
org.apache.hadoop.security.UserGroupInformation.getUGIFromTicketCache(UserGroupInformation.java:654)
at 
org.apache.hadoop.security.UserGroupInformation.getBestUGI(UserGroupInformation.java:624)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:495)
Caused by: javax.security.auth.login.LoginException: Unable to obtain password 
from user

at 

[jira] [Assigned] (HDFS-14668) Support Fuse with Users from multiple Security Realms

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth reassigned HDFS-14668:
---

Assignee: Istvan Fajth

> Support Fuse with Users from multiple Security Realms
> -
>
> Key: HDFS-14668
> URL: https://issues.apache.org/jira/browse/HDFS-14668
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: fuse-dfs
>Reporter: Sailesh Patel
>Assignee: Istvan Fajth
>Priority: Minor
>
> Users from non-default  krb5 domain can't use hadoop-fuse.
> There are 2 Realms with kdc. 
> -one realm is for human users  (USERS.COM.US) 
> -the other is for service principals.   (SERVICE.COM.US) 
> Cross realm trust is setup.
> In krb5.conf  the default domain  is set to SERVICE.COM.US
> Users within USERS.COM.US Realm are not able to put any files to Fuse mounted 
> location
> The client shows:
>   cp: cannot create regular file ‘/hdfs_mount/tmp/hello_from_fuse.txt’: 
> Input/output error



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDDS-2462) Add jq dependency in Contribution guideline

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth reassigned HDDS-2462:
--

Assignee: Istvan Fajth

> Add jq dependency in Contribution guideline
> ---
>
> Key: HDDS-2462
> URL: https://issues.apache.org/jira/browse/HDDS-2462
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Docker based tests are using JQ to parse JMX pages of different processes, 
> but the documentation does not mention it as a dependency.
> Add it to CONTRIBUTION.MD in the "Additional requirements to execute 
> different type of tests" section.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-2462) Add jq dependency in Contribution guideline

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDDS-2462:
---
Summary: Add jq dependency in Contribution guideline  (was: Add jq 
dependency in how to contribute docs)

> Add jq dependency in Contribution guideline
> ---
>
> Key: HDDS-2462
> URL: https://issues.apache.org/jira/browse/HDDS-2462
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Istvan Fajth
>Priority: Major
>
> Docker based tests are using JQ to parse JMX pages of different processes, 
> but the documentation does not mention it as a dependency.
> Add it to CONTRIBUTION.MD in the "Additional requirements to execute 
> different type of tests" section.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDDS-2462) Add jq dependency in how to contribute docs

[Istvan Fajth (Jira)]

Istvan Fajth created HDDS-2462:
--

 Summary: Add jq dependency in how to contribute docs
 Key: HDDS-2462
 URL: https://issues.apache.org/jira/browse/HDDS-2462
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Istvan Fajth


Docker based tests are using JQ to parse JMX pages of different processes, but 
the documentation does not mention it as a dependency.

Add it to CONTRIBUTION.MD in the "Additional requirements to execute different 
type of tests" section.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972334#comment-16972334
 ] 

Istvan Fajth commented on HDFS-14882:
-

Hi [~hexiaoqiao], patch-10 looks good to me, sorry for the slow response, I got 
a bit overwhelmed with other stuff and I couldn't get back here faster.

> Consider DataNode load when #getBlockLocation
> -
>
> Key: HDFS-14882
> URL: https://issues.apache.org/jira/browse/HDFS-14882
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, 
> HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, 
> HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch, 
> HDFS-14882.009.patch, HDFS-14882.010.patch, HDFS-14882.suggestion
>
>
> Currently, we consider load of datanode when #chooseTarget for writer, 
> however not consider it for reader. Thus, the process slot of datanode could 
> be occupied by #BlockSender for reader, and disk/network will be busy 
> workload, then meet some slow node exception. IIRC same case is reported 
> times. Based on the fact, I propose to consider load for reader same as it 
> did #chooseTarget for writer.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-2409) CI - acceptance test results link is broken in GitHub

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDDS-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969004#comment-16969004
 ] 

Istvan Fajth commented on HDDS-2409:


[~elek] can you please check into this?

> CI - acceptance test results link is broken in GitHub
> -
>
> Key: HDDS-2409
> URL: https://issues.apache.org/jira/browse/HDDS-2409
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Istvan Fajth
>Priority: Major
>
> When I click to see the acceptance test results in a Pull Request, I get a 
> 404.
> It seems elek.github.io is not available, and the link points to there.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDDS-2409) CI - acceptance test results link is broken in GitHub

[Istvan Fajth (Jira)]

Istvan Fajth created HDDS-2409:
--

 Summary: CI - acceptance test results link is broken in GitHub
 Key: HDDS-2409
 URL: https://issues.apache.org/jira/browse/HDDS-2409
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Istvan Fajth


When I click to see the acceptance test results in a Pull Request, I get a 404.
It seems elek.github.io is not available, and the link points to there.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDDS-2408) Separate OM Storage initialization from OzoneManager

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth reassigned HDDS-2408:
--

Assignee: Istvan Fajth

> Separate OM Storage initialization from OzoneManager
> 
>
> Key: HDDS-2408
> URL: https://issues.apache.org/jira/browse/HDDS-2408
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> The Ozone Manager Storage initialization is partly using the scm security 
> client protocol.
> As a first step of cleaning up high level protobuf usage from the code, it is 
> beneficial to separate this initialization code, so that it is clearly 
> visible what parts of the SCM it is using, and later it helps to figure out 
> the better design for the SCM security client related calls.
> This JIRA is aiming for the following separation tasks:
> - Clear out the OM Storage initialization related code from OzoneManager class
> - Clear out and move the common code that is used by OzoneManager class 
> initialization and OzoneManager Storage initialization logic to avoid any 
> duplication.
> - Clear unused stuff from OzoneManager after the changes.
> - To be the first step in a longer refactoring activity that clears out the 
> SCM client and security client calls related code in OzoneManager and from 
> things like KeyManager to have these abstracted from protobuf



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDDS-2408) Separate OM Storage initialization from OzoneManager

[Istvan Fajth (Jira)]

Istvan Fajth created HDDS-2408:
--

 Summary: Separate OM Storage initialization from OzoneManager
 Key: HDDS-2408
 URL: https://issues.apache.org/jira/browse/HDDS-2408
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Istvan Fajth


The Ozone Manager Storage initialization is partly using the scm security 
client protocol.

As a first step of cleaning up high level protobuf usage from the code, it is 
beneficial to separate this initialization code, so that it is clearly visible 
what parts of the SCM it is using, and later it helps to figure out the better 
design for the SCM security client related calls.

This JIRA is aiming for the following separation tasks:
- Clear out the OM Storage initialization related code from OzoneManager class
- Clear out and move the common code that is used by OzoneManager class 
initialization and OzoneManager Storage initialization logic to avoid any 
duplication.
- Clear unused stuff from OzoneManager after the changes.
- To be the first step in a longer refactoring activity that clears out the SCM 
client and security client calls related code in OzoneManager and from things 
like KeyManager to have these abstracted from protobuf



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDDS-2391) Consolidate XCeiverClientManager instantiation

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth reassigned HDDS-2391:
--

Assignee: Istvan Fajth

> Consolidate XCeiverClientManager instantiation
> --
>
> Key: HDDS-2391
> URL: https://issues.apache.org/jira/browse/HDDS-2391
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> All the things that are necessary for an XCeiverClientManager instance to be 
> created should be the Configuration object.
> In a secured environment, we have some initialization logic that needs to be 
> done by the clients of this class, this would be better if encapsulated 
> internally by the manager's construction logic.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDDS-2391) Consolidate XCeiverClientManager instantiation

[Istvan Fajth (Jira)]

Istvan Fajth created HDDS-2391:
--

 Summary: Consolidate XCeiverClientManager instantiation
 Key: HDDS-2391
 URL: https://issues.apache.org/jira/browse/HDDS-2391
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Istvan Fajth


All the things that are necessary for an XCeiverClientManager instance to be 
created should be the Configuration object.

In a secured environment, we have some initialization logic that needs to be 
done by the clients of this class, this would be better if encapsulated 
internally by the manager's construction logic.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-2390) SCMSecurityClient and mixing of abstraction layers

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDDS-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964025#comment-16964025
 ] 

Istvan Fajth commented on HDDS-2390:


I am working on a PR for this one and I will publish it hopefully today for 
review.

> SCMSecurityClient and mixing of abstraction layers
> --
>
> Key: HDDS-2390
> URL: https://issues.apache.org/jira/browse/HDDS-2390
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> During the work on HDDS-2362, I ran into a bigger topic with Security and how 
> the code is organized around internal certificate handling.
> What I see is that we have two methods, HDDSUtils.getScmSecurityClient, and 
> HDDSClientUtils.getSCMSecurityClient, which are essentially doing the same, 
> the latter one without infinite retries, but used only from one test class, 
> besides this DefaultCertificateClient replicates the same logic also.
> Further evaluating the first one in HDDSUtils, I see that it is being used at 
> 3 places, ContainerOperationClient, HddsDataNodeService, and OzoneManager, 
> both of the two has its own DefaultCertificateClient descendants as well.
> ContainerOperationClient is using the client to get the CA certificate.
> HddsDataNodeService is using it to have a signed certificate for the DataNode.
> OzoneManager uses it for the same purpose to get a signed certificate.
> In the HddsDataNodeService, and OzoneManager class with this we effectively 
> tinkering with protobuf requests and responses, and protobuf related code, 
> which seems to be pretty high for the communication layer to buble up to.
> Based on the above, I would propose to have this logic encapsulated in a 
> client class that handles all the protobuf related things below 
> DefaultCertificateClient, and let all the code that is depending on 
> SCMSecurityProtocolClientSideTranslatorPB to depend on this new class or the 
> already exsisting DefaultCertificateClient methods and its descendants.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDDS-2390) SCMSecurityClient and mixing of abstraction layers

[Istvan Fajth (Jira)]

Istvan Fajth created HDDS-2390:
--

 Summary: SCMSecurityClient and mixing of abstraction layers
 Key: HDDS-2390
 URL: https://issues.apache.org/jira/browse/HDDS-2390
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Istvan Fajth


During the work on HDDS-2362, I ran into a bigger topic with Security and how 
the code is organized around internal certificate handling.

What I see is that we have two methods, HDDSUtils.getScmSecurityClient, and 
HDDSClientUtils.getSCMSecurityClient, which are essentially doing the same, the 
latter one without infinite retries, but used only from one test class, besides 
this DefaultCertificateClient replicates the same logic also.

Further evaluating the first one in HDDSUtils, I see that it is being used at 3 
places, ContainerOperationClient, HddsDataNodeService, and OzoneManager, both 
of the two has its own DefaultCertificateClient descendants as well.

ContainerOperationClient is using the client to get the CA certificate.
HddsDataNodeService is using it to have a signed certificate for the DataNode.
OzoneManager uses it for the same purpose to get a signed certificate.

In the HddsDataNodeService, and OzoneManager class with this we effectively 
tinkering with protobuf requests and responses, and protobuf related code, 
which seems to be pretty high for the communication layer to buble up to.

Based on the above, I would propose to have this logic encapsulated in a client 
class that handles all the protobuf related things below 
DefaultCertificateClient, and let all the code that is depending on 
SCMSecurityProtocolClientSideTranslatorPB to depend on this new class or the 
already exsisting DefaultCertificateClient methods and its descendants.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDDS-2390) SCMSecurityClient and mixing of abstraction layers

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth reassigned HDDS-2390:
--

Assignee: Istvan Fajth

> SCMSecurityClient and mixing of abstraction layers
> --
>
> Key: HDDS-2390
> URL: https://issues.apache.org/jira/browse/HDDS-2390
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> During the work on HDDS-2362, I ran into a bigger topic with Security and how 
> the code is organized around internal certificate handling.
> What I see is that we have two methods, HDDSUtils.getScmSecurityClient, and 
> HDDSClientUtils.getSCMSecurityClient, which are essentially doing the same, 
> the latter one without infinite retries, but used only from one test class, 
> besides this DefaultCertificateClient replicates the same logic also.
> Further evaluating the first one in HDDSUtils, I see that it is being used at 
> 3 places, ContainerOperationClient, HddsDataNodeService, and OzoneManager, 
> both of the two has its own DefaultCertificateClient descendants as well.
> ContainerOperationClient is using the client to get the CA certificate.
> HddsDataNodeService is using it to have a signed certificate for the DataNode.
> OzoneManager uses it for the same purpose to get a signed certificate.
> In the HddsDataNodeService, and OzoneManager class with this we effectively 
> tinkering with protobuf requests and responses, and protobuf related code, 
> which seems to be pretty high for the communication layer to buble up to.
> Based on the above, I would propose to have this logic encapsulated in a 
> client class that handles all the protobuf related things below 
> DefaultCertificateClient, and let all the code that is depending on 
> SCMSecurityProtocolClientSideTranslatorPB to depend on this new class or the 
> already exsisting DefaultCertificateClient methods and its descendants.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDDS-2376) Fail to read data through XceiverClientGrpc

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDDS-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16963835#comment-16963835
 ] 

Istvan Fajth edited comment on HDDS-2376 at 10/31/19 10:08 AM:
---

Hi [~Sammi],

I have ran into a similar exception in one of our test environments, while I 
was preparing some testing, this appeared after I have updated the Ozone and 
Ratis jars on the cluster, and couldn't get to the bottom of it as there were 
some other minor changes as well, and after rewriting the data everything 
started to work properly, then I couldn't get to a reproduction so far.
Could the same happen on your side? Were there an update on Ozone after which 
you started to see this?


was (Author: pifta):
Hi [~Sammi],

I have run into a similar exception in one of our test environments, while I 
was preparing some testing, this appeared after I have updated the Ozone and 
Ratis jars on the cluster, and couldn't get to the bottom of it as there were 
some other minor changes as well, and after rewriting the data everything 
started to work properly, then I couldn't get to a reproduction so far.
Could the same happen on your side? Were there an update on Ozone after which 
you started to see this?

> Fail to read data through XceiverClientGrpc
> ---
>
> Key: HDDS-2376
> URL: https://issues.apache.org/jira/browse/HDDS-2376
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Sammi Chen
>Assignee: Hanisha Koneru
>Priority: Blocker
>
> Run teragen, application failed with following stack, 
> 19/10/29 14:35:42 INFO mapreduce.Job: Running job: job_1567133159094_0048
> 19/10/29 14:35:59 INFO mapreduce.Job: Job job_1567133159094_0048 running in 
> uber mode : false
> 19/10/29 14:35:59 INFO mapreduce.Job:  map 0% reduce 0%
> 19/10/29 14:35:59 INFO mapreduce.Job: Job job_1567133159094_0048 failed with 
> state FAILED due to: Application application_1567133159094_0048 failed 2 
> times due to AM Container for appattempt_1567133159094_0048_02 exited 
> with  exitCode: -1000
> For more detailed output, check application tracking 
> page:http://host183:8088/cluster/app/application_1567133159094_0048Then, 
> click on links to logs of each attempt.
> Diagnostics: Unexpected OzoneException: 
> org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum mismatch at 
> index 0
> java.io.IOException: Unexpected OzoneException: 
> org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum mismatch at 
> index 0
>   at 
> org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:342)
>   at 
> org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:307)
>   at 
> org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:259)
>   at 
> org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:144)
>   at 
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:239)
>   at 
> org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171)
>   at 
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:52)
>   at java.io.DataInputStream.read(DataInputStream.java:100)
>   at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:86)
>   at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:60)
>   at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:120)
>   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
>   at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:267)
>   at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
>   at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
>   at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
>   at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:359)
>   at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum 
> mismatch at index 0
>   at 
> 

[jira] [Commented] (HDDS-2376) Fail to read data through XceiverClientGrpc

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDDS-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16963835#comment-16963835
 ] 

Istvan Fajth commented on HDDS-2376:


Hi [~Sammi],

I have run into a similar exception in one of our test environments, while I 
was preparing some testing, this appeared after I have updated the Ozone and 
Ratis jars on the cluster, and couldn't get to the bottom of it as there were 
some other minor changes as well, and after rewriting the data everything 
started to work properly, then I couldn't get to a reproduction so far.
Could the same happen on your side? Were there an update on Ozone after which 
you started to see this?

> Fail to read data through XceiverClientGrpc
> ---
>
> Key: HDDS-2376
> URL: https://issues.apache.org/jira/browse/HDDS-2376
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Sammi Chen
>Assignee: Hanisha Koneru
>Priority: Blocker
>
> Run teragen, application failed with following stack, 
> 19/10/29 14:35:42 INFO mapreduce.Job: Running job: job_1567133159094_0048
> 19/10/29 14:35:59 INFO mapreduce.Job: Job job_1567133159094_0048 running in 
> uber mode : false
> 19/10/29 14:35:59 INFO mapreduce.Job:  map 0% reduce 0%
> 19/10/29 14:35:59 INFO mapreduce.Job: Job job_1567133159094_0048 failed with 
> state FAILED due to: Application application_1567133159094_0048 failed 2 
> times due to AM Container for appattempt_1567133159094_0048_02 exited 
> with  exitCode: -1000
> For more detailed output, check application tracking 
> page:http://host183:8088/cluster/app/application_1567133159094_0048Then, 
> click on links to logs of each attempt.
> Diagnostics: Unexpected OzoneException: 
> org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum mismatch at 
> index 0
> java.io.IOException: Unexpected OzoneException: 
> org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum mismatch at 
> index 0
>   at 
> org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:342)
>   at 
> org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:307)
>   at 
> org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:259)
>   at 
> org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:144)
>   at 
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:239)
>   at 
> org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171)
>   at 
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:52)
>   at java.io.DataInputStream.read(DataInputStream.java:100)
>   at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:86)
>   at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:60)
>   at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:120)
>   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
>   at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:267)
>   at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
>   at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
>   at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
>   at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:359)
>   at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.ozone.common.OzoneChecksumException: Checksum 
> mismatch at index 0
>   at 
> org.apache.hadoop.ozone.common.ChecksumData.verifyChecksumDataMatches(ChecksumData.java:148)
>   at 
> org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:275)
>   at 
> org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:238)
>   at 
> org.apache.hadoop.hdds.scm.storage.ChunkInputStream.lambda$new$0(ChunkInputStream.java:375)
>   at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:287)
>   at 
> 

[jira] [Commented] (HDDS-2378) Change "OZONE" as string used in the code where OzoneConsts.OZONE is suitable

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDDS-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962479#comment-16962479
 ] 

Istvan Fajth commented on HDDS-2378:


After going through the code, the following came out of this:
There are some places where we use the string literal "Ozone" which I would not 
change, the places are:
- RocksDBStore, and RDBStore, where it is used to specify the name of jmx keys.
- ServerUtils uses it in a function parameter passed to a log message that 
seems fine.
- StorageContainerManager, and OzoneManager uses the full capital in 
InterfaceAudience annotation, that is fine.
- CommonHeadersContainerResponseFilter is using this in headers I would not 
change the protocol

In TestAuthorizationHeaderV4 we use "ozone" that is ok, as we provide a string 
to parse, and check elements provided, in this case I would not change to a 
constant makes it harder to read.

CreateSubcommand I am unsure, the default is specified with full capital I am 
not brave enough to change it to the lowercase version, though my instinct 
tells that it would be most likely ok.


Also while going through the S3 related tests, that were coming up with the 
query, I changed a few other literals to suitable constants, like:
"OWNER" -> OzoneConsts.OWNER (makes it lowercase)
"VOLUME" -> OzoneConsts.VOLUME (makes it lowercase)
"hdfs" -> OzoneConsts.OZONE_SIMPLE_HDFS_USER
"xxBucket" -> OzoneConsts.BUCKET
"s3Bucket" -> OzoneConsts.S3_BUCKET
"key1" -> OzoneConsts.KEY (in cases where only 1 key is used)

Preparing the PR shortly.

> Change "OZONE" as string used in the code where OzoneConsts.OZONE is suitable
> -
>
> Key: HDDS-2378
> URL: https://issues.apache.org/jira/browse/HDDS-2378
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.4.1
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> Based on a review I have done a quick check, and there are quite a few places 
> where we have hardcoded "ozone" as String literal or a capital version of it 
> into the code.
> Let's check then one by one, and where it is possible replace it with 
> OzoneConsts.OZONE, or if the lower case version is not acceptable at all 
> places, then create an other constant with the uppercase version and use that.
> This is the search, and the results:
> {code:bash}
> find . -name *.java | while read FILE; do NUM=`grep -c -i "\"OZONE\"" $FILE`; 
> if [ $NUM -gt 0 ]; then echo $FILE; fi; done | sort | uniq
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDataYaml.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestBlockManagerImpl.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java
> ./hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ServerUtils.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/SCMContainerManagerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/block/TestBlockManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestCloseContainerEventHandler.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSCMContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestContainerPlacement.java
> ./hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/CreateSubcommand.java
> ./hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/OzoneVersionInfo.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/metrics/TestSCMContainerManagerMetrics.java
> 

[jira] [Updated] (HDDS-2378) Change "OZONE" as string used in the code where OzoneConsts.OZONE is suitable

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDDS-2378:
---
Summary: Change "OZONE" as string used in the code where OzoneConsts.OZONE 
is suitable  (was: Remove "OZONE" as string used in tests)

> Change "OZONE" as string used in the code where OzoneConsts.OZONE is suitable
> -
>
> Key: HDDS-2378
> URL: https://issues.apache.org/jira/browse/HDDS-2378
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.4.1
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> Based on a review I have done a quick check, and there are quite a few places 
> where we have hardcoded "ozone" as String literal or a capital version of it 
> into the code.
> Let's check then one by one, and where it is possible replace it with 
> OzoneConsts.OZONE, or if the lower case version is not acceptable at all 
> places, then create an other constant with the uppercase version and use that.
> This is the search, and the results:
> {code:bash}
> find . -name *.java | while read FILE; do NUM=`grep -c -i "\"OZONE\"" $FILE`; 
> if [ $NUM -gt 0 ]; then echo $FILE; fi; done | sort | uniq
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDataYaml.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestBlockManagerImpl.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java
> ./hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ServerUtils.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/SCMContainerManagerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/block/TestBlockManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestCloseContainerEventHandler.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSCMContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestContainerPlacement.java
> ./hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/CreateSubcommand.java
> ./hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/OzoneVersionInfo.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/metrics/TestSCMContainerManagerMetrics.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerOperations.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerStateMachineIdempotency.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestS3Shell.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestAllocateContainer.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSmallFile.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestGetCommittedBlockLengthAndPutKey.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientManager.java
> 

[jira] [Assigned] (HDDS-2378) Remove "OZONE" as string used in tests

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth reassigned HDDS-2378:
--

Assignee: Istvan Fajth

> Remove "OZONE" as string used in tests
> --
>
> Key: HDDS-2378
> URL: https://issues.apache.org/jira/browse/HDDS-2378
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> Based on a review I have done a quick check, and there are quite a few places 
> where we have hardcoded "ozone" as String literal or a capital version of it 
> into the code.
> Let's check then one by one, and where it is possible replace it with 
> OzoneConsts.OZONE, or if the lower case version is not acceptable at all 
> places, then create an other constant with the uppercase version and use that.
> This is the search, and the results:
> {code:bash}
> find . -name *.java | while read FILE; do NUM=`grep -c -i "\"OZONE\"" $FILE`; 
> if [ $NUM -gt 0 ]; then echo $FILE; fi; done | sort | uniq
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
> ./hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDataYaml.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestBlockManagerImpl.java
> ./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java
> ./hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ServerUtils.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/SCMContainerManagerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java
> ./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/block/TestBlockManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestCloseContainerEventHandler.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSCMContainerManager.java
> ./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestContainerPlacement.java
> ./hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/CreateSubcommand.java
> ./hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/OzoneVersionInfo.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/metrics/TestSCMContainerManagerMetrics.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerOperations.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerStateMachineIdempotency.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestS3Shell.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestAllocateContainer.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSmallFile.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestGetCommittedBlockLengthAndPutKey.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientManager.java
> ./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientMetrics.java
> ./hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
> ./hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestS3BucketManager.java
> 

[jira] [Created] (HDDS-2378) Remove "OZONE" as string used in tests

[Istvan Fajth (Jira)]

Istvan Fajth created HDDS-2378:
--

 Summary: Remove "OZONE" as string used in tests
 Key: HDDS-2378
 URL: https://issues.apache.org/jira/browse/HDDS-2378
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Istvan Fajth


Based on a review I have done a quick check, and there are quite a few places 
where we have hardcoded "ozone" as String literal or a capital version of it 
into the code.

Let's check then one by one, and where it is possible replace it with 
OzoneConsts.OZONE, or if the lower case version is not acceptable at all 
places, then create an other constant with the uppercase version and use that.

This is the search, and the results:
{code:bash}
find . -name *.java | while read FILE; do NUM=`grep -c -i "\"OZONE\"" $FILE`; 
if [ $NUM -gt 0 ]; then echo $FILE; fi; done | sort | uniq
./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java
./hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
./hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDataYaml.java
./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestBlockManagerImpl.java
./hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java
./hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ServerUtils.java
./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/SCMContainerManagerMetrics.java
./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementMetrics.java
./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeMetrics.java
./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineMetrics.java
./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java
./hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/block/TestBlockManager.java
./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestCloseContainerEventHandler.java
./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSCMContainerManager.java
./hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestContainerPlacement.java
./hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/CreateSubcommand.java
./hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/OzoneVersionInfo.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/metrics/TestSCMContainerManagerMetrics.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerOperations.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerStateMachineIdempotency.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManager.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestS3Shell.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestAllocateContainer.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSmallFile.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestGetCommittedBlockLengthAndPutKey.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientManager.java
./hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientMetrics.java
./hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
./hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestS3BucketManager.java
./hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBufferWithOMResponse.java
./hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMAllocateBlockRequest.java
./hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCommitRequest.java

[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961864#comment-16961864
 ] 

Istvan Fajth commented on HDFS-14882:
-

Yes, KeyInputStream is something else I was working on, and forgot there, thank 
you for cleaning up.

The shuffle is there by default at the current code, the patch preserves that 
behaviour when the readConsiderLoad is false, that is there to give some random 
distribution to the DN read load already.

Patch 009 looks good to me, thank you for incorporating the suggestion and 
provide a combined patch. I would suggest two more things to do, we might 
deprecate the old sorter methods, as we most likely won't need them on the long 
run, as their use effectively overrides the new setting, and an update would be 
nice to the APIDoc of these methods.
Other than that if CI does not bring up any relevant issues, I support the 
change, but the only thing I can give is a non-binding +1 :)

> Consider DataNode load when #getBlockLocation
> -
>
> Key: HDFS-14882
> URL: https://issues.apache.org/jira/browse/HDFS-14882
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, 
> HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, 
> HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch, 
> HDFS-14882.009.patch, HDFS-14882.suggestion
>
>
> Currently, we consider load of datanode when #chooseTarget for writer, 
> however not consider it for reader. Thus, the process slot of datanode could 
> be occupied by #BlockSender for reader, and disk/network will be busy 
> workload, then meet some slow node exception. IIRC same case is reported 
> times. Based on the fact, I propose to consider load for reader same as it 
> did #chooseTarget for writer.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDDS-2373) Move isUseRatis getFactor and getType from XCeiverClientManager

[Istvan Fajth (Jira)]

Istvan Fajth created HDDS-2373:
--

 Summary: Move isUseRatis getFactor and getType from 
XCeiverClientManager
 Key: HDDS-2373
 URL: https://issues.apache.org/jira/browse/HDDS-2373
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Istvan Fajth


The given methods in XCeiverClientManager are working based on the 
configuration supplied in the constructor of XCeiverClientManager class.

The only real code usage of this is in ContainerOperationsClient.

Refactor the ContainerOperationsClient constructor to work based on the 
configuration, and then move these values there directly and set the values of 
them at the constructor. Clean up all test references to the methods, and 
remove the methods from the XCeiverClientManager.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDDS-2373) Move isUseRatis getFactor and getType from XCeiverClientManager

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth reassigned HDDS-2373:
--

Assignee: Istvan Fajth

> Move isUseRatis getFactor and getType from XCeiverClientManager
> ---
>
> Key: HDDS-2373
> URL: https://issues.apache.org/jira/browse/HDDS-2373
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> The given methods in XCeiverClientManager are working based on the 
> configuration supplied in the constructor of XCeiverClientManager class.
> The only real code usage of this is in ContainerOperationsClient.
> Refactor the ContainerOperationsClient constructor to work based on the 
> configuration, and then move these values there directly and set the values 
> of them at the constructor. Clean up all test references to the methods, and 
> remove the methods from the XCeiverClientManager.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-14882) Consider DataNode load when #getBlockLocation

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-14882:

Attachment: HDFS-14882.suggestion

> Consider DataNode load when #getBlockLocation
> -
>
> Key: HDFS-14882
> URL: https://issues.apache.org/jira/browse/HDFS-14882
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, 
> HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, 
> HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch, 
> HDFS-14882.suggestion
>
>
> Currently, we consider load of datanode when #chooseTarget for writer, 
> however not consider it for reader. Thus, the process slot of datanode could 
> be occupied by #BlockSender for reader, and disk/network will be busy 
> workload, then meet some slow node exception. IIRC same case is reported 
> times. Based on the fact, I propose to consider load for reader same as it 
> did #chooseTarget for writer.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-14882) Consider DataNode load when #getBlockLocation

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961023#comment-16961023
 ] 

Istvan Fajth commented on HDFS-14882:
-

Hello [~hexiaoqiao],

I was checking into the patch, and into the proposal, and I think even though 
the changes looks cool, and does what it promises as I see, I would have one 
question/suggestion to consider instead of doing this when the 
dfs.namenode.read.considerLoad is set to true:
In NetworkTopology#sortByDistance, we already sort the nodes by network 
distance, and there is a shuffle for the nodes on the same level that thrives 
to ensure some distribution of load. That shuffle can be considered as well as 
a secondary sorting strategy, which we can inject into that point from outside. 
If we inject the secondary sorting from the DataNodeManager, then if the 
read.considerLoad is turned on, we can inject a sorting by transceiver count 
instead of the shuffle.

With this, we can avoid calculating the network distance twice, also we can 
avoid shuffling then sorting by transceiver count. I am posting a proposal, 
just to demonstrate what exactly I am thinking about, the JUnit test in 
patch-008 is passing with it, I haven't tried other tests locally.

Please share what do you think about this approach. Also I am happy to have 
some feedback from you [~ayushtkn] and [~elgoiri] too.

> Consider DataNode load when #getBlockLocation
> -
>
> Key: HDFS-14882
> URL: https://issues.apache.org/jira/browse/HDFS-14882
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, 
> HDFS-14882.003.patch, HDFS-14882.004.patch, HDFS-14882.005.patch, 
> HDFS-14882.006.patch, HDFS-14882.007.patch, HDFS-14882.008.patch
>
>
> Currently, we consider load of datanode when #chooseTarget for writer, 
> however not consider it for reader. Thus, the process slot of datanode could 
> be occupied by #BlockSender for reader, and disk/network will be busy 
> workload, then meet some slow node exception. IIRC same case is reported 
> times. Based on the fact, I propose to consider load for reader same as it 
> did #chooseTarget for writer.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDDS-2362) XCeiverClientManager issues

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth reassigned HDDS-2362:
--

Assignee: Istvan Fajth

> XCeiverClientManager issues
> ---
>
> Key: HDDS-2362
> URL: https://issues.apache.org/jira/browse/HDDS-2362
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> These issues were revealed while reviewing the XCeiverClientManager, and the 
> clients.
> - secure clients are not released properly, so the reference counting does 
> not work with secure clients
> - even though we have reference counting for the clients, the cache can evict 
> and remove client instances with active references, as it is not connected 
> with the reference counts
> - isUseRatis, getFactor, getType is not really belonging to this class
> - acquireClient and acquireClientForRead and release methods of the same 
> kind, seems to be a bad smell, we might separate the two things, especially 
> because reads are using the grpc client while writes are using the ratis 
> client as I know
> - pipelines are leaking from the clients themselves, the pipelines are not 
> modified in these code paths, but it should be better if we can hide the 
> pipeline, and don't serve it for the clients, or if we can serve an immutable 
> version
> - ContainerProtocolCalls seems to be something that is extracted to a utility 
> class but it may be placed into the client itself, as in all the cases, the 
> client is gathered from the XCeiverClientManager, then given to one of 
> ContainerProtocolCalls' method, which calls the sendCommandAsync on the 
> client which does not seem to be necessary, we can encapsulate all the 
> protobuf message creation, and provide response data from the client.
> -ContainerOperationClient acquires the client twice from the 
> XCeiverClientManager in the createContainer call, but releases it only once
> - we can try to get rid of some of the synchronization by trying to eliminate 
> some of the states in the clients, and the manager, and replace them with 
> some polymorphism.
> I will go through this list one by one and will create JIRAs one by one using 
> this one as an umbrella JIRA, so that we can create PRs one by one, or if 
> needed, we can consolidate the whole thing into one PR at the end but review 
> one by one.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDDS-2362) XCeiverClientManager issues

[Istvan Fajth (Jira)]

Istvan Fajth created HDDS-2362:
--

 Summary: XCeiverClientManager issues
 Key: HDDS-2362
 URL: https://issues.apache.org/jira/browse/HDDS-2362
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Istvan Fajth


These issues were revealed while reviewing the XCeiverClientManager, and the 
clients.

- secure clients are not released properly, so the reference counting does not 
work with secure clients
- even though we have reference counting for the clients, the cache can evict 
and remove client instances with active references, as it is not connected with 
the reference counts
- isUseRatis, getFactor, getType is not really belonging to this class
- acquireClient and acquireClientForRead and release methods of the same kind, 
seems to be a bad smell, we might separate the two things, especially because 
reads are using the grpc client while writes are using the ratis client as I 
know
- pipelines are leaking from the clients themselves, the pipelines are not 
modified in these code paths, but it should be better if we can hide the 
pipeline, and don't serve it for the clients, or if we can serve an immutable 
version
- ContainerProtocolCalls seems to be something that is extracted to a utility 
class but it may be placed into the client itself, as in all the cases, the 
client is gathered from the XCeiverClientManager, then given to one of 
ContainerProtocolCalls' method, which calls the sendCommandAsync on the client 
which does not seem to be necessary, we can encapsulate all the protobuf 
message creation, and provide response data from the client.
-ContainerOperationClient acquires the client twice from the 
XCeiverClientManager in the createContainer call, but releases it only once
- we can try to get rid of some of the synchronization by trying to eliminate 
some of the states in the clients, and the manager, and replace them with some 
polymorphism.

I will go through this list one by one and will create JIRAs one by one using 
this one as an umbrella JIRA, so that we can create PRs one by one, or if 
needed, we can consolidate the whole thing into one PR at the end but review 
one by one.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-14932) XCeiverClientManager issues

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth resolved HDFS-14932.
-
Resolution: Invalid

Sorry I meant to create this one in HDDS project, closing this one, as I can 
not move it.

> XCeiverClientManager issues
> ---
>
> Key: HDFS-14932
> URL: https://issues.apache.org/jira/browse/HDFS-14932
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Istvan Fajth
>Priority: Major
>
> These issues were revealed while reviewing the XCeiverClientManager, and the 
> clients.
> - secure clients are not released properly, so the reference counting does 
> not work with secure clients
> - even though we have reference counting for the clients, the cache can evict 
> and remove client instances with active references, as it is not connected 
> with the reference counts
> - isUseRatis, getFactor, getType is not really belonging to this class
> - acquireClient and acquireClientForRead and release methods of the same 
> kind, seems to be a bad smell, we might separate the two things, especially 
> because reads are using the grpc client while writes are using the ratis 
> client as I know
> - pipelines are leaking from the clients themselves, the pipelines are not 
> modified in these code paths, but it should be better if we can hide the 
> pipeline, and don't serve it for the clients, or if we can serve an immutable 
> version
> - ContainerProtocolCalls seems to be something that is extracted to a utility 
> class but it may be placed into the client itself, as in all the cases, the 
> client is gathered from the XCeiverClientManager, then given to one of 
> ContainerProtocolCalls' method, which calls the sendCommandAsync on the 
> client which does not seem to be necessary, we can encapsulate all the 
> protobuf message creation, and provide response data from the client.
> -ContainerOperationClient acquires the client twice from the 
> XCeiverClientManager in the createContainer call, but releases it only once
> - we can try to get rid of some of the synchronization by trying to eliminate 
> some of the states in the clients, and the manager, and replace them with 
> some polymorphism.
> I will go through this list one by one and will create JIRAs one by one using 
> this one as an umbrella JIRA, so that we can create PRs one by one, or if 
> needed, we can consolidate the whole thing into one PR at the end but review 
> one by one.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-14932) XCeiverClientManager issues

[Istvan Fajth (Jira)]

Istvan Fajth created HDFS-14932:
---

 Summary: XCeiverClientManager issues
 Key: HDFS-14932
 URL: https://issues.apache.org/jira/browse/HDFS-14932
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Istvan Fajth


These issues were revealed while reviewing the XCeiverClientManager, and the 
clients.

- secure clients are not released properly, so the reference counting does not 
work with secure clients
- even though we have reference counting for the clients, the cache can evict 
and remove client instances with active references, as it is not connected with 
the reference counts
- isUseRatis, getFactor, getType is not really belonging to this class
- acquireClient and acquireClientForRead and release methods of the same kind, 
seems to be a bad smell, we might separate the two things, especially because 
reads are using the grpc client while writes are using the ratis client as I 
know
- pipelines are leaking from the clients themselves, the pipelines are not 
modified in these code paths, but it should be better if we can hide the 
pipeline, and don't serve it for the clients, or if we can serve an immutable 
version
- ContainerProtocolCalls seems to be something that is extracted to a utility 
class but it may be placed into the client itself, as in all the cases, the 
client is gathered from the XCeiverClientManager, then given to one of 
ContainerProtocolCalls' method, which calls the sendCommandAsync on the client 
which does not seem to be necessary, we can encapsulate all the protobuf 
message creation, and provide response data from the client.
-ContainerOperationClient acquires the client twice from the 
XCeiverClientManager in the createContainer call, but releases it only once
- we can try to get rid of some of the synchronization by trying to eliminate 
some of the states in the clients, and the manager, and replace them with some 
polymorphism.

I will go through this list one by one and will create JIRAs one by one using 
this one as an umbrella JIRA, so that we can create PRs one by one, or if 
needed, we can consolidate the whole thing into one PR at the end but review 
one by one.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-2347) XCeiverClientGrpc's parallel use leads to NPE

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDDS-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16958858#comment-16958858
 ] 

Istvan Fajth commented on HDDS-2347:


I have added a PR to fix this issue in the way Mukul suggested so by just 
adding the necessary synchronization to the XCeiverClientGrpc class and its 
creation. More on the changes can be found in the PR.

I think we need to do some rework on this code to ensure thread safety at 
object creation time, I still need to check a few things, but I think I will 
defer the proposal on this to a new JIRA linked to this one, as this one is 
pretty much needs to be fixed shortly and we can evaluate possible better ways 
later on.

> XCeiverClientGrpc's parallel use leads to NPE
> -
>
> Key: HDDS-2347
> URL: https://issues.apache.org/jira/browse/HDDS-2347
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: Ozone Client
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Critical
>  Labels: pull-request-available
> Attachments: changes.diff, logs.txt
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This issue came up when testing Hive with ORC tables on Ozone storage 
> backend, I so far I could not reproduce it locally within a JUnit test but 
> the issue.
> I am attaching a diff file that shows what logging I have added in 
> XCevierClientGrpc and in KeyInputStream to get the results that made me 
> arrive to the following understanding of the scenario:
> - Hive starts a couple of threads to work on the table data during query 
> execution
> - There is one RPCClient that is being used by these threads
> - The threads are opening different stream to read from the same key in ozone
> - The InputStreams internally are using the same XCeiverClientGrpc
> - XCeiverClientGrpc throws the following NPE intermittently:
> {code}
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:398)
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:295)
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:259)
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:242)
> at 
> org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118)
> at 
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:169)
> at 
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:118)
> at 
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:224)
> at 
> org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:173)
> at 
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:52)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
> at 
> org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
> at 
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:112)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:555)
> at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370)
> at 
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:61)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:105)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:1708)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1596)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2900(OrcInputFormat.java:1383)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1568)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1565)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1565)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1383)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> {code}
> I have two proposals to fix this issue, one is the easy answer to put 
> synchronization to the XCeiverClientGrpc 

[jira] [Updated] (HDDS-2347) XCeiverClientGrpc's parallel use leads to NPE

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDDS-2347:
---
Priority: Critical  (was: Major)

> XCeiverClientGrpc's parallel use leads to NPE
> -
>
> Key: HDDS-2347
> URL: https://issues.apache.org/jira/browse/HDDS-2347
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: Ozone Client
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Critical
>  Labels: pull-request-available
> Attachments: changes.diff, logs.txt
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This issue came up when testing Hive with ORC tables on Ozone storage 
> backend, I so far I could not reproduce it locally within a JUnit test but 
> the issue.
> I am attaching a diff file that shows what logging I have added in 
> XCevierClientGrpc and in KeyInputStream to get the results that made me 
> arrive to the following understanding of the scenario:
> - Hive starts a couple of threads to work on the table data during query 
> execution
> - There is one RPCClient that is being used by these threads
> - The threads are opening different stream to read from the same key in ozone
> - The InputStreams internally are using the same XCeiverClientGrpc
> - XCeiverClientGrpc throws the following NPE intermittently:
> {code}
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:398)
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:295)
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:259)
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:242)
> at 
> org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118)
> at 
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:169)
> at 
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:118)
> at 
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:224)
> at 
> org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:173)
> at 
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:52)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
> at 
> org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
> at 
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:112)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:555)
> at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370)
> at 
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:61)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:105)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:1708)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1596)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2900(OrcInputFormat.java:1383)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1568)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1565)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1565)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1383)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> {code}
> I have two proposals to fix this issue, one is the easy answer to put 
> synchronization to the XCeiverClientGrpc code, the other one is a bit more 
> complicated, let me explain below.
> Naively I would assume that when I get a client SPI instance from 
> XCeiverClientManager, that instance is ready to use. In fact it is not, and 
> when the user of the SPI instance sends the first request that is the point 
> when the client gets essentially ready. Now if we put synchronization to this 
> code, that is the easy solution, but my pragmatic half screams for a better 
> solution, that ensures that the Manager essentially manages the clients 

[jira] [Created] (HDDS-2359) Seeking randomly in a key with more than 2 blocks of data leads to inconsistent reads

[Istvan Fajth (Jira)]

Istvan Fajth created HDDS-2359:
--

 Summary: Seeking randomly in a key with more than 2 blocks of data 
leads to inconsistent reads
 Key: HDDS-2359
 URL: https://issues.apache.org/jira/browse/HDDS-2359
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Istvan Fajth
Assignee: Shashikant Banerjee


During Hive testing we found the following exception:

{code}
TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) : 
attempt_1569246922012_0214_1_03_00_3:java.lang.RuntimeException: 
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: 
java.io.IOException: error iterating
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
java.io.IOException: java.io.IOException: error iterating
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:80)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
... 16 more
Caused by: java.io.IOException: java.io.IOException: error iterating
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:366)
at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
at 
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
... 18 more
Caused by: java.io.IOException: error iterating
at 
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.next(VectorizedOrcAcidRowBatchReader.java:835)
at 
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.next(VectorizedOrcAcidRowBatchReader.java:74)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:361)
... 24 more
Caused by: java.io.IOException: Error reading file: 
o3fs://hive.warehouse.vc0136.halxg.cloudera.com:9862/data/inventory/delta_001_001_/bucket_0
at 
org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1283)
at 
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.nextBatch(RecordReaderImpl.java:156)
at 
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader$1.next(VectorizedOrcAcidRowBatchReader.java:150)
at 
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader$1.next(VectorizedOrcAcidRowBatchReader.java:146)
at 
org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.next(VectorizedOrcAcidRowBatchReader.java:831)
... 26 more
Caused by: java.io.IOException: Inconsistent read for blockID=conID: 2 locID: 

[jira] [Updated] (HDDS-2347) XCeiverClientGrpc's parallel use leads to NPE

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDDS-2347:
---
Attachment: logs.txt

> XCeiverClientGrpc's parallel use leads to NPE
> -
>
> Key: HDDS-2347
> URL: https://issues.apache.org/jira/browse/HDDS-2347
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: Ozone Client
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
> Attachments: changes.diff, logs.txt
>
>
> This issue came up when testing Hive with ORC tables on Ozone storage 
> backend, I so far I could not reproduce it locally within a JUnit test but 
> the issue.
> I am attaching a diff file that shows what logging I have added in 
> XCevierClientGrpc and in KeyInputStream to get the results that made me 
> arrive to the following understanding of the scenario:
> - Hive starts a couple of threads to work on the table data during query 
> execution
> - There is one RPCClient that is being used by these threads
> - The threads are opening different stream to read from the same key in ozone
> - The InputStreams internally are using the same XCeiverClientGrpc
> - XCeiverClientGrpc throws the following NPE intermittently:
> {code}
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:398)
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:295)
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:259)
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:242)
> at 
> org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118)
> at 
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:169)
> at 
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:118)
> at 
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:224)
> at 
> org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:173)
> at 
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:52)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
> at 
> org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
> at 
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:112)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:555)
> at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370)
> at 
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:61)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:105)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:1708)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1596)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2900(OrcInputFormat.java:1383)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1568)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1565)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1565)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1383)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> {code}
> I have two proposals to fix this issue, one is the easy answer to put 
> synchronization to the XCeiverClientGrpc code, the other one is a bit more 
> complicated, let me explain below.
> Naively I would assume that when I get a client SPI instance from 
> XCeiverClientManager, that instance is ready to use. In fact it is not, and 
> when the user of the SPI instance sends the first request that is the point 
> when the client gets essentially ready. Now if we put synchronization to this 
> code, that is the easy solution, but my pragmatic half screams for a better 
> solution, that ensures that the Manager essentially manages the clients that 
> is giving to it's users, and the clients themselves are not getting ready by 
> accident.
> I am working on 

[jira] [Updated] (HDDS-2347) XCeiverClientGrpc's parallel use leads to NPE

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDDS-2347:
---
Attachment: changes.diff

> XCeiverClientGrpc's parallel use leads to NPE
> -
>
> Key: HDDS-2347
> URL: https://issues.apache.org/jira/browse/HDDS-2347
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: Ozone Client
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
> Attachments: changes.diff
>
>
> This issue came up when testing Hive with ORC tables on Ozone storage 
> backend, I so far I could not reproduce it locally within a JUnit test but 
> the issue.
> I am attaching a diff file that shows what logging I have added in 
> XCevierClientGrpc and in KeyInputStream to get the results that made me 
> arrive to the following understanding of the scenario:
> - Hive starts a couple of threads to work on the table data during query 
> execution
> - There is one RPCClient that is being used by these threads
> - The threads are opening different stream to read from the same key in ozone
> - The InputStreams internally are using the same XCeiverClientGrpc
> - XCeiverClientGrpc throws the following NPE intermittently:
> {code}
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:398)
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:295)
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:259)
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:242)
> at 
> org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118)
> at 
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:169)
> at 
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:118)
> at 
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:224)
> at 
> org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:173)
> at 
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:52)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
> at 
> org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
> at 
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:112)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:555)
> at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370)
> at 
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:61)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:105)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:1708)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1596)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2900(OrcInputFormat.java:1383)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1568)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1565)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1565)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1383)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> {code}
> I have two proposals to fix this issue, one is the easy answer to put 
> synchronization to the XCeiverClientGrpc code, the other one is a bit more 
> complicated, let me explain below.
> Naively I would assume that when I get a client SPI instance from 
> XCeiverClientManager, that instance is ready to use. In fact it is not, and 
> when the user of the SPI instance sends the first request that is the point 
> when the client gets essentially ready. Now if we put synchronization to this 
> code, that is the easy solution, but my pragmatic half screams for a better 
> solution, that ensures that the Manager essentially manages the clients that 
> is giving to it's users, and the clients themselves are not getting ready by 
> accident.
> I am working on a 

[jira] [Commented] (HDDS-2347) XCeiverClientGrpc's parallel use leads to NPE

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDDS-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956945#comment-16956945
 ] 

Istvan Fajth commented on HDDS-2347:


CC: [~aengineer], [~elek], [~msingh]

> XCeiverClientGrpc's parallel use leads to NPE
> -
>
> Key: HDDS-2347
> URL: https://issues.apache.org/jira/browse/HDDS-2347
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: Ozone Client
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Major
>
> This issue came up when testing Hive with ORC tables on Ozone storage 
> backend, I so far I could not reproduce it locally within a JUnit test but 
> the issue.
> I am attaching a diff file that shows what logging I have added in 
> XCevierClientGrpc and in KeyInputStream to get the results that made me 
> arrive to the following understanding of the scenario:
> - Hive starts a couple of threads to work on the table data during query 
> execution
> - There is one RPCClient that is being used by these threads
> - The threads are opening different stream to read from the same key in ozone
> - The InputStreams internally are using the same XCeiverClientGrpc
> - XCeiverClientGrpc throws the following NPE intermittently:
> {code}
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:398)
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:295)
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:259)
> at 
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:242)
> at 
> org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118)
> at 
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:169)
> at 
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:118)
> at 
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:224)
> at 
> org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:173)
> at 
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:52)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
> at 
> org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
> at 
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:112)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:555)
> at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370)
> at 
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:61)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:105)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:1708)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1596)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2900(OrcInputFormat.java:1383)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1568)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1565)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1565)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1383)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> {code}
> I have two proposals to fix this issue, one is the easy answer to put 
> synchronization to the XCeiverClientGrpc code, the other one is a bit more 
> complicated, let me explain below.
> Naively I would assume that when I get a client SPI instance from 
> XCeiverClientManager, that instance is ready to use. In fact it is not, and 
> when the user of the SPI instance sends the first request that is the point 
> when the client gets essentially ready. Now if we put synchronization to this 
> code, that is the easy solution, but my pragmatic half screams for a better 
> solution, that ensures that the Manager essentially manages the clients that 
> is giving to it's users, and the clients themselves are not getting ready by 
> accident.
> I am 

[jira] [Created] (HDDS-2347) XCeiverClientGrpc's parallel use leads to NPE

[Istvan Fajth (Jira)]

Istvan Fajth created HDDS-2347:
--

 Summary: XCeiverClientGrpc's parallel use leads to NPE
 Key: HDDS-2347
 URL: https://issues.apache.org/jira/browse/HDDS-2347
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: Ozone Client
Reporter: Istvan Fajth
Assignee: Istvan Fajth


This issue came up when testing Hive with ORC tables on Ozone storage backend, 
I so far I could not reproduce it locally within a JUnit test but the issue.

I am attaching a diff file that shows what logging I have added in 
XCevierClientGrpc and in KeyInputStream to get the results that made me arrive 
to the following understanding of the scenario:
- Hive starts a couple of threads to work on the table data during query 
execution
- There is one RPCClient that is being used by these threads
- The threads are opening different stream to read from the same key in ozone
- The InputStreams internally are using the same XCeiverClientGrpc
- XCeiverClientGrpc throws the following NPE intermittently:

{code}
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:398)
at 
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:295)
at 
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:259)
at 
org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:242)
at 
org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118)
at 
org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:169)
at 
org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:118)
at 
org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:224)
at 
org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:173)
at 
org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:52)
at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
at 
org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:112)
at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:555)
at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:370)
at 
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:61)
at 
org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:105)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.populateAndCacheStripeDetails(OrcInputFormat.java:1708)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1596)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2900(OrcInputFormat.java:1383)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1568)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1565)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1565)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1383)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
{code}

I have two proposals to fix this issue, one is the easy answer to put 
synchronization to the XCeiverClientGrpc code, the other one is a bit more 
complicated, let me explain below.

Naively I would assume that when I get a client SPI instance from 
XCeiverClientManager, that instance is ready to use. In fact it is not, and 
when the user of the SPI instance sends the first request that is the point 
when the client gets essentially ready. Now if we put synchronization to this 
code, that is the easy solution, but my pragmatic half screams for a better 
solution, that ensures that the Manager essentially manages the clients that is 
giving to it's users, and the clients themselves are not getting ready by 
accident.
I am working on a proposal that moves things around a bit, and looking for 
possible other solutions that does not feel hacky as I feel with the easy 
solution.

I am attaching the followings:
- a diff that shows the added extended logging in XCeiverClientGrpc and 
KeyInputStream.
- a job log snippet from a Hive query that shows the relevant output from the 
extensive logging added by the diff in a cluster.
- later a proposal 

[jira] [Updated] (HDDS-2261) Change readChunk methods to return ByteBuffer

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDDS-2261:
---
Labels: pull-request-available  (was: )

> Change readChunk methods to return ByteBuffer
> -
>
> Key: HDDS-2261
> URL: https://issues.apache.org/jira/browse/HDDS-2261
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Istvan Fajth
>Priority: Major
>  Labels: pull-request-available
>
> During refactoring to HDDS-2233 I realized the following:
> KeyValueHandler.handleReadChunk and handleGetSmallFile methods are using 
> ChunkManager.readChunk, which returns a byte[], but then both of them (the 
> only usage points) converts the returning byte[] to a ByteBuffer, and then to 
> a ByteString.
> ChunkManagerImpl on the other hand in readChunk utilizes 
> ChunkUtils.readChunk, which in order to conform the return value converts a 
> ByteBuffer back to a byte[].
> I open this JIRA to change the internal logic to fully rely on ByteBuffers 
> instead of converting from ByteBuffer to byte[] then to ByteBuffer again.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDDS-2261) Change readChunk methods to return ByteBuffer

[Istvan Fajth (Jira)]

Istvan Fajth created HDDS-2261:
--

 Summary: Change readChunk methods to return ByteBuffer
 Key: HDDS-2261
 URL: https://issues.apache.org/jira/browse/HDDS-2261
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Istvan Fajth


During refactoring to HDDS-2233 I realized the following:
KeyValueHandler.handleReadChunk and handleGetSmallFile methods are using 
ChunkManager.readChunk, which returns a byte[], but then both of them (the only 
usage points) converts the returning byte[] to a ByteBuffer, and then to a 
ByteString.
ChunkManagerImpl on the other hand in readChunk utilizes ChunkUtils.readChunk, 
which in order to conform the return value converts a ByteBuffer back to a 
byte[].

I open this JIRA to change the internal logic to fully rely on ByteBuffers 
instead of converting from ByteBuffer to byte[] then to ByteBuffer again.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-2203) Race condition in ByteStringHelper.init()

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDDS-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943452#comment-16943452
 ] 

Istvan Fajth commented on HDDS-2203:


I am already working on the new patch, it is expected to be done later today, 
that is why I just closed out this one ;)

> Race condition in ByteStringHelper.init()
> -
>
> Key: HDDS-2203
> URL: https://issues.apache.org/jira/browse/HDDS-2203
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Ozone Client, SCM
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Critical
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> The current init method:
> {code}
> public static void init(boolean isUnsafeByteOperation) {
>   final boolean set = INITIALIZED.compareAndSet(false, true);
>   if (set) {
> ByteStringHelper.isUnsafeByteOperationsEnabled =
>isUnsafeByteOperation;
>} else {
>  // already initialized, check values
>  Preconditions.checkState(isUnsafeByteOperationsEnabled
>== isUnsafeByteOperation);
>}
> }
> {code}
> In a scenario when two thread accesses this method, and the execution order 
> is the following, then the second thread runs into an exception from 
> PreCondition.checkState() in the else branch.
> In an unitialized state:
> - T1 thread arrives to the method with true as the parameter, the class 
> initialises the isUnsafeByteOperationsEnabled to false
> - T1 sets INITIALIZED true
> - T2 arrives to the method with true as the parameter
> - T2 reads the INITALIZED value and as it is not false goes to else branch
> - T2 tries to check if the internal boolean property is the same true as it 
> wanted to set, and as T1 still to set the value, the checkState throws an 
> IllegalArgumentException.
> This happens in certain Hive query cases, as it came from that testing, the 
> exception we see there is the following:
> {code}
> Error: Error while processing statement: FAILED: Execution Error, return code 
> 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, 
> vertexName=Map 2, vertexId=vertex_1569486223160_0334_1_02, 
> diagnostics=[Vertex vertex_1569486223160_0334_1_02 [Map 2] killed/failed
>  due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: item initializer failed, 
> vertex=vertex_1569486223160_0334_1_02 [Map 2], java.io.IOException: Couldn't 
> create RpcClient protocol
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:203)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:165)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:158)
> at 
> org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:50)
> at 
> org.apache.hadoop.fs.ozone.OzoneFileSystem.createAdapter(OzoneFileSystem.java:102)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:155)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3315)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:136)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3364)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3332)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:491)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1821)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:781)
> at 
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at 
> 

[jira] [Resolved] (HDDS-2203) Race condition in ByteStringHelper.init()

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth resolved HDDS-2203.

Resolution: Duplicate

I am resolving and closing this ticket, and will post a new PR with the 
refactoring suggested and agreed upon.

> Race condition in ByteStringHelper.init()
> -
>
> Key: HDDS-2203
> URL: https://issues.apache.org/jira/browse/HDDS-2203
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Ozone Client, SCM
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Critical
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> The current init method:
> {code}
> public static void init(boolean isUnsafeByteOperation) {
>   final boolean set = INITIALIZED.compareAndSet(false, true);
>   if (set) {
> ByteStringHelper.isUnsafeByteOperationsEnabled =
>isUnsafeByteOperation;
>} else {
>  // already initialized, check values
>  Preconditions.checkState(isUnsafeByteOperationsEnabled
>== isUnsafeByteOperation);
>}
> }
> {code}
> In a scenario when two thread accesses this method, and the execution order 
> is the following, then the second thread runs into an exception from 
> PreCondition.checkState() in the else branch.
> In an unitialized state:
> - T1 thread arrives to the method with true as the parameter, the class 
> initialises the isUnsafeByteOperationsEnabled to false
> - T1 sets INITIALIZED true
> - T2 arrives to the method with true as the parameter
> - T2 reads the INITALIZED value and as it is not false goes to else branch
> - T2 tries to check if the internal boolean property is the same true as it 
> wanted to set, and as T1 still to set the value, the checkState throws an 
> IllegalArgumentException.
> This happens in certain Hive query cases, as it came from that testing, the 
> exception we see there is the following:
> {code}
> Error: Error while processing statement: FAILED: Execution Error, return code 
> 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, 
> vertexName=Map 2, vertexId=vertex_1569486223160_0334_1_02, 
> diagnostics=[Vertex vertex_1569486223160_0334_1_02 [Map 2] killed/failed
>  due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: item initializer failed, 
> vertex=vertex_1569486223160_0334_1_02 [Map 2], java.io.IOException: Couldn't 
> create RpcClient protocol
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:203)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:165)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:158)
> at 
> org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:50)
> at 
> org.apache.hadoop.fs.ozone.OzoneFileSystem.createAdapter(OzoneFileSystem.java:102)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:155)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3315)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:136)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3364)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3332)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:491)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1821)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:781)
> at 
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at 
> 

[jira] [Closed] (HDDS-2203) Race condition in ByteStringHelper.init()

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth closed HDDS-2203.
--

> Race condition in ByteStringHelper.init()
> -
>
> Key: HDDS-2203
> URL: https://issues.apache.org/jira/browse/HDDS-2203
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Ozone Client, SCM
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Critical
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> The current init method:
> {code}
> public static void init(boolean isUnsafeByteOperation) {
>   final boolean set = INITIALIZED.compareAndSet(false, true);
>   if (set) {
> ByteStringHelper.isUnsafeByteOperationsEnabled =
>isUnsafeByteOperation;
>} else {
>  // already initialized, check values
>  Preconditions.checkState(isUnsafeByteOperationsEnabled
>== isUnsafeByteOperation);
>}
> }
> {code}
> In a scenario when two thread accesses this method, and the execution order 
> is the following, then the second thread runs into an exception from 
> PreCondition.checkState() in the else branch.
> In an unitialized state:
> - T1 thread arrives to the method with true as the parameter, the class 
> initialises the isUnsafeByteOperationsEnabled to false
> - T1 sets INITIALIZED true
> - T2 arrives to the method with true as the parameter
> - T2 reads the INITALIZED value and as it is not false goes to else branch
> - T2 tries to check if the internal boolean property is the same true as it 
> wanted to set, and as T1 still to set the value, the checkState throws an 
> IllegalArgumentException.
> This happens in certain Hive query cases, as it came from that testing, the 
> exception we see there is the following:
> {code}
> Error: Error while processing statement: FAILED: Execution Error, return code 
> 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, 
> vertexName=Map 2, vertexId=vertex_1569486223160_0334_1_02, 
> diagnostics=[Vertex vertex_1569486223160_0334_1_02 [Map 2] killed/failed
>  due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: item initializer failed, 
> vertex=vertex_1569486223160_0334_1_02 [Map 2], java.io.IOException: Couldn't 
> create RpcClient protocol
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:203)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:165)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:158)
> at 
> org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:50)
> at 
> org.apache.hadoop.fs.ozone.OzoneFileSystem.createAdapter(OzoneFileSystem.java:102)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:155)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3315)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:136)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3364)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3332)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:491)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1821)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:781)
> at 
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at 
> 

[jira] [Created] (HDDS-2233) Remove ByteStringHelper and refactor the code to the place where it used

[Istvan Fajth (Jira)]

Istvan Fajth created HDDS-2233:
--

 Summary: Remove ByteStringHelper and refactor the code to the 
place where it used
 Key: HDDS-2233
 URL: https://issues.apache.org/jira/browse/HDDS-2233
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Istvan Fajth


See HDDS-2203 where there is a race condition reported by me.

Later in the discussion we agreed that it is better to refactor the code and 
remove the class completely for now, and that would also resolve the race 
condition.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-2203) Race condition in ByteStringHelper.init()

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDDS-2203:
---
Labels:   (was: pull-request-available pull-requests-available)

> Race condition in ByteStringHelper.init()
> -
>
> Key: HDDS-2203
> URL: https://issues.apache.org/jira/browse/HDDS-2203
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Ozone Client, SCM
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Critical
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> The current init method:
> {code}
> public static void init(boolean isUnsafeByteOperation) {
>   final boolean set = INITIALIZED.compareAndSet(false, true);
>   if (set) {
> ByteStringHelper.isUnsafeByteOperationsEnabled =
>isUnsafeByteOperation;
>} else {
>  // already initialized, check values
>  Preconditions.checkState(isUnsafeByteOperationsEnabled
>== isUnsafeByteOperation);
>}
> }
> {code}
> In a scenario when two thread accesses this method, and the execution order 
> is the following, then the second thread runs into an exception from 
> PreCondition.checkState() in the else branch.
> In an unitialized state:
> - T1 thread arrives to the method with true as the parameter, the class 
> initialises the isUnsafeByteOperationsEnabled to false
> - T1 sets INITIALIZED true
> - T2 arrives to the method with true as the parameter
> - T2 reads the INITALIZED value and as it is not false goes to else branch
> - T2 tries to check if the internal boolean property is the same true as it 
> wanted to set, and as T1 still to set the value, the checkState throws an 
> IllegalArgumentException.
> This happens in certain Hive query cases, as it came from that testing, the 
> exception we see there is the following:
> {code}
> Error: Error while processing statement: FAILED: Execution Error, return code 
> 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, 
> vertexName=Map 2, vertexId=vertex_1569486223160_0334_1_02, 
> diagnostics=[Vertex vertex_1569486223160_0334_1_02 [Map 2] killed/failed
>  due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: item initializer failed, 
> vertex=vertex_1569486223160_0334_1_02 [Map 2], java.io.IOException: Couldn't 
> create RpcClient protocol
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:203)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:165)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:158)
> at 
> org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:50)
> at 
> org.apache.hadoop.fs.ozone.OzoneFileSystem.createAdapter(OzoneFileSystem.java:102)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:155)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3315)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:136)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3364)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3332)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:491)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1821)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:781)
> at 
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at 
> 

[jira] [Updated] (HDDS-2203) Race condition in ByteStringHelper.init()

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDDS-2203:
---
Status: Open  (was: Patch Available)

> Race condition in ByteStringHelper.init()
> -
>
> Key: HDDS-2203
> URL: https://issues.apache.org/jira/browse/HDDS-2203
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Ozone Client, SCM
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Critical
>  Labels: pull-request-available, pull-requests-available
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> The current init method:
> {code}
> public static void init(boolean isUnsafeByteOperation) {
>   final boolean set = INITIALIZED.compareAndSet(false, true);
>   if (set) {
> ByteStringHelper.isUnsafeByteOperationsEnabled =
>isUnsafeByteOperation;
>} else {
>  // already initialized, check values
>  Preconditions.checkState(isUnsafeByteOperationsEnabled
>== isUnsafeByteOperation);
>}
> }
> {code}
> In a scenario when two thread accesses this method, and the execution order 
> is the following, then the second thread runs into an exception from 
> PreCondition.checkState() in the else branch.
> In an unitialized state:
> - T1 thread arrives to the method with true as the parameter, the class 
> initialises the isUnsafeByteOperationsEnabled to false
> - T1 sets INITIALIZED true
> - T2 arrives to the method with true as the parameter
> - T2 reads the INITALIZED value and as it is not false goes to else branch
> - T2 tries to check if the internal boolean property is the same true as it 
> wanted to set, and as T1 still to set the value, the checkState throws an 
> IllegalArgumentException.
> This happens in certain Hive query cases, as it came from that testing, the 
> exception we see there is the following:
> {code}
> Error: Error while processing statement: FAILED: Execution Error, return code 
> 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, 
> vertexName=Map 2, vertexId=vertex_1569486223160_0334_1_02, 
> diagnostics=[Vertex vertex_1569486223160_0334_1_02 [Map 2] killed/failed
>  due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: item initializer failed, 
> vertex=vertex_1569486223160_0334_1_02 [Map 2], java.io.IOException: Couldn't 
> create RpcClient protocol
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:203)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:165)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:158)
> at 
> org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:50)
> at 
> org.apache.hadoop.fs.ozone.OzoneFileSystem.createAdapter(OzoneFileSystem.java:102)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:155)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3315)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:136)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3364)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3332)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:491)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1821)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:781)
> at 
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at 
> 

[jira] [Commented] (HDDS-2203) Race condition in ByteStringHelper.init()

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDDS-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940499#comment-16940499
 ] 

Istvan Fajth commented on HDDS-2203:


I have some concerns about the design of this class, and I would like to have 
some insights from others, especially from [~aengineer] as he reviewed the 
change so far.
What do you think about removing this class, and move its parts back to the 
calling code, to
#1 avoid the synchronization at all
#2 eliminate the need of the init call before the conversion methods as this I 
think is an unfortunate design
#3 move this logic where it is used, as it is used from two places, but from 
one place one of the methods are called, from the other, an other method is 
called?

If you agree, I would create a new JIRA for that, and also I am happy to 
provide a PR with the changes.

> Race condition in ByteStringHelper.init()
> -
>
> Key: HDDS-2203
> URL: https://issues.apache.org/jira/browse/HDDS-2203
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Ozone Client, SCM
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Critical
>  Labels: pull-request-available, pull-requests-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The current init method:
> {code}
> public static void init(boolean isUnsafeByteOperation) {
>   final boolean set = INITIALIZED.compareAndSet(false, true);
>   if (set) {
> ByteStringHelper.isUnsafeByteOperationsEnabled =
>isUnsafeByteOperation;
>} else {
>  // already initialized, check values
>  Preconditions.checkState(isUnsafeByteOperationsEnabled
>== isUnsafeByteOperation);
>}
> }
> {code}
> In a scenario when two thread accesses this method, and the execution order 
> is the following, then the second thread runs into an exception from 
> PreCondition.checkState() in the else branch.
> In an unitialized state:
> - T1 thread arrives to the method with true as the parameter, the class 
> initialises the isUnsafeByteOperationsEnabled to false
> - T1 sets INITIALIZED true
> - T2 arrives to the method with true as the parameter
> - T2 reads the INITALIZED value and as it is not false goes to else branch
> - T2 tries to check if the internal boolean property is the same true as it 
> wanted to set, and as T1 still to set the value, the checkState throws an 
> IllegalArgumentException.
> This happens in certain Hive query cases, as it came from that testing, the 
> exception we see there is the following:
> {code}
> Error: Error while processing statement: FAILED: Execution Error, return code 
> 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, 
> vertexName=Map 2, vertexId=vertex_1569486223160_0334_1_02, 
> diagnostics=[Vertex vertex_1569486223160_0334_1_02 [Map 2] killed/failed
>  due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: item initializer failed, 
> vertex=vertex_1569486223160_0334_1_02 [Map 2], java.io.IOException: Couldn't 
> create RpcClient protocol
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:203)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:165)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:158)
> at 
> org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:50)
> at 
> org.apache.hadoop.fs.ozone.OzoneFileSystem.createAdapter(OzoneFileSystem.java:102)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:155)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3315)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:136)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3364)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3332)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:491)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1821)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:781)
> at 
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at 
> 

[jira] [Updated] (HDDS-2203) Race condition in ByteStringHelper.init()

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDDS-2203:
---
Component/s: SCM
 Ozone Client

> Race condition in ByteStringHelper.init()
> -
>
> Key: HDDS-2203
> URL: https://issues.apache.org/jira/browse/HDDS-2203
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Ozone Client, SCM
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Critical
>  Labels: pull-requests-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The current init method:
> {code}
> public static void init(boolean isUnsafeByteOperation) {
>   final boolean set = INITIALIZED.compareAndSet(false, true);
>   if (set) {
> ByteStringHelper.isUnsafeByteOperationsEnabled =
>isUnsafeByteOperation;
>} else {
>  // already initialized, check values
>  Preconditions.checkState(isUnsafeByteOperationsEnabled
>== isUnsafeByteOperation);
>}
> }
> {code}
> In a scenario when two thread accesses this method, and the execution order 
> is the following, then the second thread runs into an exception from 
> PreCondition.checkState() in the else branch.
> In an unitialized state:
> - T1 thread arrives to the method with true as the parameter, the class 
> initialises the isUnsafeByteOperationsEnabled to false
> - T1 sets INITIALIZED true
> - T2 arrives to the method with true as the parameter
> - T2 reads the INITALIZED value and as it is not false goes to else branch
> - T2 tries to check if the internal boolean property is the same true as it 
> wanted to set, and as T1 still to set the value, the checkState throws an 
> IllegalArgumentException.
> This happens in certain Hive query cases, as it came from that testing, the 
> exception we see there is the following:
> {code}
> Error: Error while processing statement: FAILED: Execution Error, return code 
> 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, 
> vertexName=Map 2, vertexId=vertex_1569486223160_0334_1_02, 
> diagnostics=[Vertex vertex_1569486223160_0334_1_02 [Map 2] killed/failed
>  due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: item initializer failed, 
> vertex=vertex_1569486223160_0334_1_02 [Map 2], java.io.IOException: Couldn't 
> create RpcClient protocol
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:203)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:165)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:158)
> at 
> org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:50)
> at 
> org.apache.hadoop.fs.ozone.OzoneFileSystem.createAdapter(OzoneFileSystem.java:102)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:155)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3315)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:136)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3364)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3332)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:491)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1821)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:781)
> at 
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at 
> 

[jira] [Updated] (HDDS-2203) Race condition in ByteStringHelper.init()

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDDS-2203:
---
Labels: pull-requests-available  (was: pull-request-available)

> Race condition in ByteStringHelper.init()
> -
>
> Key: HDDS-2203
> URL: https://issues.apache.org/jira/browse/HDDS-2203
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Critical
>  Labels: pull-requests-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The current init method:
> {code}
> public static void init(boolean isUnsafeByteOperation) {
>   final boolean set = INITIALIZED.compareAndSet(false, true);
>   if (set) {
> ByteStringHelper.isUnsafeByteOperationsEnabled =
>isUnsafeByteOperation;
>} else {
>  // already initialized, check values
>  Preconditions.checkState(isUnsafeByteOperationsEnabled
>== isUnsafeByteOperation);
>}
> }
> {code}
> In a scenario when two thread accesses this method, and the execution order 
> is the following, then the second thread runs into an exception from 
> PreCondition.checkState() in the else branch.
> In an unitialized state:
> - T1 thread arrives to the method with true as the parameter, the class 
> initialises the isUnsafeByteOperationsEnabled to false
> - T1 sets INITIALIZED true
> - T2 arrives to the method with true as the parameter
> - T2 reads the INITALIZED value and as it is not false goes to else branch
> - T2 tries to check if the internal boolean property is the same true as it 
> wanted to set, and as T1 still to set the value, the checkState throws an 
> IllegalArgumentException.
> This happens in certain Hive query cases, as it came from that testing, the 
> exception we see there is the following:
> {code}
> Error: Error while processing statement: FAILED: Execution Error, return code 
> 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, 
> vertexName=Map 2, vertexId=vertex_1569486223160_0334_1_02, 
> diagnostics=[Vertex vertex_1569486223160_0334_1_02 [Map 2] killed/failed
>  due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: item initializer failed, 
> vertex=vertex_1569486223160_0334_1_02 [Map 2], java.io.IOException: Couldn't 
> create RpcClient protocol
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:203)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:165)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:158)
> at 
> org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:50)
> at 
> org.apache.hadoop.fs.ozone.OzoneFileSystem.createAdapter(OzoneFileSystem.java:102)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:155)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3315)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:136)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3364)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3332)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:491)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1821)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:781)
> at 
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at 
> 

[jira] [Assigned] (HDDS-2203) Race condition in ByteStringHelper.init()

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth reassigned HDDS-2203:
--

Assignee: Istvan Fajth

> Race condition in ByteStringHelper.init()
> -
>
> Key: HDDS-2203
> URL: https://issues.apache.org/jira/browse/HDDS-2203
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Istvan Fajth
>Priority: Critical
>
> The current init method:
> {code}
> public static void init(boolean isUnsafeByteOperation) {
>   final boolean set = INITIALIZED.compareAndSet(false, true);
>   if (set) {
> ByteStringHelper.isUnsafeByteOperationsEnabled =
>isUnsafeByteOperation;
>} else {
>  // already initialized, check values
>  Preconditions.checkState(isUnsafeByteOperationsEnabled
>== isUnsafeByteOperation);
>}
> }
> {code}
> In a scenario when two thread accesses this method, and the execution order 
> is the following, then the second thread runs into an exception from 
> PreCondition.checkState() in the else branch.
> In an unitialized state:
> - T1 thread arrives to the method with true as the parameter, the class 
> initialises the isUnsafeByteOperationsEnabled to false
> - T1 sets INITIALIZED true
> - T2 arrives to the method with true as the parameter
> - T2 reads the INITALIZED value and as it is not false goes to else branch
> - T2 tries to check if the internal boolean property is the same true as it 
> wanted to set, and as T1 still to set the value, the checkState throws an 
> IllegalArgumentException.
> This happens in certain Hive query cases, as it came from that testing, the 
> exception we see there is the following:
> {code}
> Error: Error while processing statement: FAILED: Execution Error, return code 
> 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, 
> vertexName=Map 2, vertexId=vertex_1569486223160_0334_1_02, 
> diagnostics=[Vertex vertex_1569486223160_0334_1_02 [Map 2] killed/failed
>  due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: item initializer failed, 
> vertex=vertex_1569486223160_0334_1_02 [Map 2], java.io.IOException: Couldn't 
> create RpcClient protocol
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:203)
> at 
> org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:165)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:158)
> at 
> org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:50)
> at 
> org.apache.hadoop.fs.ozone.OzoneFileSystem.createAdapter(OzoneFileSystem.java:102)
> at 
> org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:155)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3315)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:136)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3364)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3332)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:491)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1821)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:781)
> at 
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at 
> 

[jira] [Created] (HDDS-2203) Race condition in ByteStringHelper.init()

[Istvan Fajth (Jira)]

Istvan Fajth created HDDS-2203:
--

 Summary: Race condition in ByteStringHelper.init()
 Key: HDDS-2203
 URL: https://issues.apache.org/jira/browse/HDDS-2203
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Istvan Fajth


The current init method:

{code}
public static void init(boolean isUnsafeByteOperation) {
  final boolean set = INITIALIZED.compareAndSet(false, true);
  if (set) {
ByteStringHelper.isUnsafeByteOperationsEnabled =
   isUnsafeByteOperation;
   } else {
 // already initialized, check values
 Preconditions.checkState(isUnsafeByteOperationsEnabled
   == isUnsafeByteOperation);
   }
}
{code}

In a scenario when two thread accesses this method, and the execution order is 
the following, then the second thread runs into an exception from 
PreCondition.checkState() in the else branch.

In an unitialized state:
- T1 thread arrives to the method with true as the parameter, the class 
initialises the isUnsafeByteOperationsEnabled to false
- T1 sets INITIALIZED true
- T2 arrives to the method with true as the parameter
- T2 reads the INITALIZED value and as it is not false goes to else branch
- T2 tries to check if the internal boolean property is the same true as it 
wanted to set, and as T1 still to set the value, the checkState throws an 
IllegalArgumentException.

This happens in certain Hive query cases, as it came from that testing, the 
exception we see there is the following:
{code}
Error: Error while processing statement: FAILED: Execution Error, return code 2 
from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 
2, vertexId=vertex_1569486223160_0334_1_02, diagnostics=[Vertex 
vertex_1569486223160_0334_1_02 [Map 2] killed/failed
 due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: item initializer failed, 
vertex=vertex_1569486223160_0334_1_02 [Map 2], java.io.IOException: Couldn't 
create RpcClient protocol
at 
org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263)
at 
org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239)
at 
org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:203)
at 
org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:165)
at 
org.apache.hadoop.fs.ozone.BasicOzoneClientAdapterImpl.(BasicOzoneClientAdapterImpl.java:158)
at 
org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:50)
at 
org.apache.hadoop.fs.ozone.OzoneFileSystem.createAdapter(OzoneFileSystem.java:102)
at 
org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:155)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3315)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:136)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3364)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3332)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:491)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1821)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:781)
at 
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 

[jira] [Commented] (HDDS-2114) Rename does not preserve non-explicitly created interim directories

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDDS-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931246#comment-16931246
 ] 

Istvan Fajth commented on HDDS-2114:


I have tested the fix on a real cluster with Hive and the workflow that have 
failed and got us to this issue is working properly after the fix was applied. 
Thank you [~ljain] for working on it, thanks [~msingh] for the review and 
commit!

> Rename does not preserve non-explicitly created interim directories
> ---
>
> Key: HDDS-2114
> URL: https://issues.apache.org/jira/browse/HDDS-2114
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Lokesh Jain
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 0.5.0
>
> Attachments: demonstrative_test.patch
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> I am attaching a patch that adds a test that demonstrates the problem.
> The scenario is coming from the way how Hive implements acid transactions 
> with the ORC table format, but the test is redacted to the simplest possible 
> code that reproduces the issue.
> The scenario:
>  * Given a 3 level directory structure, where the top level directory was 
> explicitly created, and the interim directory is implicitly created (for 
> example either by creating a file with create("/top/interim/file") or by 
> creating a directory with mkdirs("top/interim/dir"))
>  * When the leaf is moved out from the implicitly created directory making 
> this directory an empty directory
>  * Then a FileNotFoundException is thrown when getFileStatus or listStatus is 
> called on the interim directory.
> The expected behaviour:
> after the directory is becoming empty, the directory should still be part of 
> the file system, moreover an empty FileStatus array should be returned when 
> listStatus is called on it, and also a valid FileStatus object should be 
> returned when getFileStatus is called on it.
>  
>  
> As this issue is present with Hive, and as this is how a FileSystem is 
> expected to work this seems to be an at least critical issue as I see, please 
> feel free to change the priority if needed.
> Also please note that, if the interim directory is explicitly created with 
> mkdirs("top/interim") before creating the leaf, then the issue does not 
> appear.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDDS-2114) Rename does not preserve non-explicitly created interim directories

[Istvan Fajth (Jira)]

Istvan Fajth created HDDS-2114:
--

 Summary: Rename does not preserve non-explicitly created interim 
directories
 Key: HDDS-2114
 URL: https://issues.apache.org/jira/browse/HDDS-2114
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Istvan Fajth
 Attachments: demonstrative_test.patch

I am attaching a patch that adds a test that demonstrates the problem.

The scenario is coming from the way how Hive implements acid transactions with 
the ORC table format, but the test is redacted to the simplest possible code 
that reproduces the issue.

The scenario:
 * Given a 3 level directory structure, where the top level directory was 
explicitly created, and the interim directory is implicitly created (for 
example either by creating a file with create("/top/interim/file") or by 
creating a directory with mkdirs("top/interim/dir"))
 * When the leaf is moved out from the implicitly created directory making this 
directory an empty directory
 * Then a FileNotFoundException is thrown when getFileStatus or listStatus is 
called on the interim directory.

The expected behaviour:

after the directory is becoming empty, the directory should still be part of 
the file system, moreover an empty FileStatus array should be returned when 
listStatus is called on it, and also a valid FileStatus object should be 
returned when getFileStatus is called on it.

 

 

As this issue is present with Hive, and as this is how a FileSystem is expected 
to work this seems to be an at least critical issue as I see, please feel free 
to change the priority if needed.

Also please note that, if the interim directory is explicitly created with 
mkdirs("top/interim") before creating the leaf, then the issue does not appear.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-2114) Rename does not preserve non-explicitly created interim directories

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDDS-2114:
---
Attachment: demonstrative_test.patch

> Rename does not preserve non-explicitly created interim directories
> ---
>
> Key: HDDS-2114
> URL: https://issues.apache.org/jira/browse/HDDS-2114
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Istvan Fajth
>Priority: Critical
> Attachments: demonstrative_test.patch
>
>
> I am attaching a patch that adds a test that demonstrates the problem.
> The scenario is coming from the way how Hive implements acid transactions 
> with the ORC table format, but the test is redacted to the simplest possible 
> code that reproduces the issue.
> The scenario:
>  * Given a 3 level directory structure, where the top level directory was 
> explicitly created, and the interim directory is implicitly created (for 
> example either by creating a file with create("/top/interim/file") or by 
> creating a directory with mkdirs("top/interim/dir"))
>  * When the leaf is moved out from the implicitly created directory making 
> this directory an empty directory
>  * Then a FileNotFoundException is thrown when getFileStatus or listStatus is 
> called on the interim directory.
> The expected behaviour:
> after the directory is becoming empty, the directory should still be part of 
> the file system, moreover an empty FileStatus array should be returned when 
> listStatus is called on it, and also a valid FileStatus object should be 
> returned when getFileStatus is called on it.
>  
>  
> As this issue is present with Hive, and as this is how a FileSystem is 
> expected to work this seems to be an at least critical issue as I see, please 
> feel free to change the priority if needed.
> Also please note that, if the interim directory is explicitly created with 
> mkdirs("top/interim") before creating the leaf, then the issue does not 
> appear.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-2112) Ozone rename is behaving different compared with HDFS

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDDS-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928007#comment-16928007
 ] 

Istvan Fajth commented on HDDS-2112:


Great [~ste...@apache.org], I was as well unsure, and I did not checked the 
contract tests to see, but I completely agree that it is tedious, also it would 
spare me a couple of hours if rename threw an exception in these situations.

Thank you for the clarification, and closure of the ticket!

> Ozone rename is behaving different compared with HDFS
> -
>
> Key: HDDS-2112
> URL: https://issues.apache.org/jira/browse/HDDS-2112
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Ozone Filesystem
>Reporter: Istvan Fajth
>Priority: Major
> Attachments: demonstrative_test.patch
>
>
> I am attaching a patch file, that introduces two new tests for the 
> OzoneFileSystem implementation which demonstrates the expected behaviour.
> Case 1:
> Given a directory a file "/source/subdir/file", and a directory /target
> When fs.rename("/source/subdir/file", "/target/subdir/file") is called
> Then DistributedFileSystem (HDFS), is returning false from the method, while 
> OzoneFileSystem throws a FileNotFoundException as "/target/subdir" is not 
> existing.
> The expected behaviour would be to return false in this case instead of 
> throwing an exception with that behave the same as DistributedFileSystem does.
>  
> Case 2:
> Given a directory "/source" and a file "/targetFile"
> When fs.rename("/source", "/targetFile") is called
> Then DistributedFileSystem (HDFS), is returning false from the method, while 
> OzoneFileSystem throws a FileAlreadyExistsException as "/targetFile" does 
> exist.
> The expected behaviour would be to return false in this case instead of 
> throwing an exception with that behave the same as DistributedFileSystem does.
>  
> It may be considered as well a bug in HDFS, however it is not clear from the 
> FileSystem interface's documentation on the two rename methods that it 
> defines in which cases an exception should be thrown and in which cases a 
> return false is the expected behaviour.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-2112) rename is behaving different compared with HDFS

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDDS-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDDS-2112:
---
Attachment: demonstrative_test.patch

> rename is behaving different compared with HDFS
> ---
>
> Key: HDDS-2112
> URL: https://issues.apache.org/jira/browse/HDDS-2112
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Ozone Filesystem
>Reporter: Istvan Fajth
>Priority: Major
> Attachments: demonstrative_test.patch
>
>
> I am attaching a patch file, that introduces two new tests for the 
> OzoneFileSystem implementation which demonstrates the expected behaviour.
> Case 1:
> Given a directory a file "/source/subdir/file", and a directory /target
> When fs.rename("/source/subdir/file", "/target/subdir/file") is called
> Then DistributedFileSystem (HDFS), is returning false from the method, while 
> OzoneFileSystem throws a FileNotFoundException as "/target/subdir" is not 
> existing.
> The expected behaviour would be to return false in this case instead of 
> throwing an exception with that behave the same as DistributedFileSystem does.
>  
> Case 2:
> Given a directory "/source" and a file "/targetFile"
> When fs.rename("/source", "/targetFile") is called
> Then DistributedFileSystem (HDFS), is returning false from the method, while 
> OzoneFileSystem throws a FileAlreadyExistsException as "/targetFile" does 
> exist.
> The expected behaviour would be to return false in this case instead of 
> throwing an exception with that behave the same as DistributedFileSystem does.
>  
> It may be considered as well a bug in HDFS, however it is not clear from the 
> FileSystem interface's documentation on the two rename methods that it 
> defines in which cases an exception should be thrown and in which cases a 
> return false is the expected behaviour.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDDS-2112) rename is behaving different compared with HDFS

[Istvan Fajth (Jira)]

Istvan Fajth created HDDS-2112:
--

 Summary: rename is behaving different compared with HDFS
 Key: HDDS-2112
 URL: https://issues.apache.org/jira/browse/HDDS-2112
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: Ozone Filesystem
Reporter: Istvan Fajth
 Attachments: demonstrative_test.patch

I am attaching a patch file, that introduces two new tests for the 
OzoneFileSystem implementation which demonstrates the expected behaviour.

Case 1:
Given a directory a file "/source/subdir/file", and a directory /target
When fs.rename("/source/subdir/file", "/target/subdir/file") is called
Then DistributedFileSystem (HDFS), is returning false from the method, while 
OzoneFileSystem throws a FileNotFoundException as "/target/subdir" is not 
existing.

The expected behaviour would be to return false in this case instead of 
throwing an exception with that behave the same as DistributedFileSystem does.

 

Case 2:
Given a directory "/source" and a file "/targetFile"
When fs.rename("/source", "/targetFile") is called
Then DistributedFileSystem (HDFS), is returning false from the method, while 
OzoneFileSystem throws a FileAlreadyExistsException as "/targetFile" does exist.

The expected behaviour would be to return false in this case instead of 
throwing an exception with that behave the same as DistributedFileSystem does.

 

It may be considered as well a bug in HDFS, however it is not clear from the 
FileSystem interface's documentation on the two rename methods that it defines 
in which cases an exception should be thrown and in which cases a return false 
is the expected behaviour.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919636#comment-16919636
 ] 

Istvan Fajth commented on HDFS-8178:


Added the 3.1 patch, thank you Wei-Chiu for the reviews and commit!

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Fix For: 3.3.0, 3.2.1
>
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch, HDFS-8178.004.patch, HDFS-8178.005.patch, 
> HDFS-8178.006.patch, HDFS-8178.007.patch, HDFS-8178.008.addendum, 
> HDFS-8178.008.merged, HDFS-8178.008.patch, HDFS-8178.branch-3.1.patch, 
> HDFS-8178.branch-3.2.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-8178:
---
Attachment: HDFS-8178.branch-3.1.patch

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Fix For: 3.3.0, 3.2.1
>
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch, HDFS-8178.004.patch, HDFS-8178.005.patch, 
> HDFS-8178.006.patch, HDFS-8178.007.patch, HDFS-8178.008.addendum, 
> HDFS-8178.008.merged, HDFS-8178.008.patch, HDFS-8178.branch-3.1.patch, 
> HDFS-8178.branch-3.2.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918685#comment-16918685
 ] 

Istvan Fajth commented on HDFS-8178:


Test failures does not seem to be related, the patch does not add any 
allocation on the heap, and does not create any new threads.

TestDFSAdminWithHA tests are running fine in my environment.

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch, HDFS-8178.004.patch, HDFS-8178.005.patch, 
> HDFS-8178.006.patch, HDFS-8178.007.patch, HDFS-8178.008.addendum, 
> HDFS-8178.008.merged, HDFS-8178.008.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918088#comment-16918088
 ] 

Istvan Fajth edited comment on HDFS-8178 at 8/28/19 8:30 PM:
-

After reviewing the test results, it seems that these test failures are 
unrelated to the changes, the first one failed on a timeout due to a slower 
startup of the NameNode than the test expected.

The second one seems to be a timing issue in the test, and the JournalNode 
seems to be starting too quickly and the socket did not got freed up until that 
point.

 

I am attaching 3 files:
 * HDFS-8178.008.patch : addressing checkstyle issues in the patch, and review 
comments by Wei-Chiu, and a refactoring in the test code.
 * HDFS-8178.008.addendum : addressing all checkstyle issues in the files that 
are in the patch with a refactoring in the code
 * HDFS-8178.008.merged: the patch and the addendum merged together


was (Author: pifta):
After reviewing the test results, it seems that these test failures are 
unrelated to the changes, the first one failed on a timeout due to a slower 
startup of the NameNode than the test expected.

The second one seems to be a timing issue in the test, and the JournalNode 
seems to be starting too quickly and the socket did not got freed up until that 
point.

 

I am attaching 3 files:
 * HDFS-8178.008.patch : addressing checkstyle issues in the patch, and review 
comments by Wei-Chiu, and a refactoring in the test code.
 * HDFS-8178.008.addendum : addressing all checkstyle issues in the files that 
are in the patch
 * HDFS-8178.008.merged: the patch and the addendum merged together

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch, HDFS-8178.004.patch, HDFS-8178.005.patch, 
> HDFS-8178.006.patch, HDFS-8178.007.patch, HDFS-8178.008.addendum, 
> HDFS-8178.008.merged, HDFS-8178.008.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-8178:
---
Attachment: HDFS-8178.008.patch
HDFS-8178.008.merged
HDFS-8178.008.addendum

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch, HDFS-8178.004.patch, HDFS-8178.005.patch, 
> HDFS-8178.006.patch, HDFS-8178.007.patch, HDFS-8178.008.addendum, 
> HDFS-8178.008.merged, HDFS-8178.008.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918088#comment-16918088
 ] 

Istvan Fajth commented on HDFS-8178:


After reviewing the test results, it seems that these test failures are 
unrelated to the changes, the first one failed on a timeout due to a slower 
startup of the NameNode than the test expected.

The second one seems to be a timing issue in the test, and the JournalNode 
seems to be starting too quickly and the socket did not got freed up until that 
point.

 

I am attaching 3 files:
 * HDFS-8178.008.patch : addressing checkstyle issues in the patch, and review 
comments by Wei-Chiu, and a refactoring in the test code.
 * HDFS-8178.008.addendum : addressing all checkstyle issues in the files that 
are in the patch
 * HDFS-8178.008.merged: the patch and the addendum merged together

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch, HDFS-8178.004.patch, HDFS-8178.005.patch, 
> HDFS-8178.006.patch, HDFS-8178.007.patch, HDFS-8178.008.addendum, 
> HDFS-8178.008.merged, HDFS-8178.008.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914208#comment-16914208
 ] 

Istvan Fajth commented on HDFS-8178:


Re-attached a new path it seems that I have accidentally changed something in 
the previous patch file

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch, HDFS-8178.004.patch, HDFS-8178.005.patch, 
> HDFS-8178.006.patch, HDFS-8178.007.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-8178:
---
Attachment: HDFS-8178.007.patch

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch, HDFS-8178.004.patch, HDFS-8178.005.patch, 
> HDFS-8178.006.patch, HDFS-8178.007.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914112#comment-16914112
 ] 

Istvan Fajth commented on HDFS-8178:


Added a new patch that addresses the synchronized access issue (findbugs), and 
the whitespace issue, I have checked the test, it seem to be failing due to a 
timeout, I haven't find the source of it yet, and locally I need significant 
increases in the timeouts in order this test to pass, so let's wait for a 
second run and see if the problem is still there. If it is then it will need 
further investigation to be sure what is causing the delay. For the first sight 
for me it is not obvious how the changes can cause a significant delay, but the 
timeout happens in the test when it waits for a NameNode to become Active, so 
it might be related.

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch, HDFS-8178.004.patch, HDFS-8178.005.patch, 
> HDFS-8178.006.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-8178:
---
Attachment: HDFS-8178.006.patch

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch, HDFS-8178.004.patch, HDFS-8178.005.patch, 
> HDFS-8178.006.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913528#comment-16913528
 ] 

Istvan Fajth edited comment on HDFS-8178 at 8/22/19 6:02 PM:
-

I have attached a new patch, that tries to incorporate concerns from [~atm].

The current solution does the following:
 * in JNStorage it uses the FileJournalManager's purgeLogsOlderThan(long) 
method to handle old edits removal
 * the purge logic has extended to do the following:
 ** purge edits file segments that are ending before the given transaction ID
 ** purge edits in progress files of any kind (normal leftover, marked as 
trash, empty, stale, and corrupted that is starting before the given 
transaction ID
 ** edits in progress files that are normal leftovers but starting after the 
given transaction ID are marked as stale

Added a test that tests the removal of the new files, and actually as I thought 
it through now, I will file a new patch shortly with a new test, that tests the 
removal of marked files.

The suggested patch incorporates HDFS-8304 as well.


was (Author: pifta):
I have attached a new patch, that tries to incorporate concerns from [~atm].

The current solution does the following:
 * in JNStorage it uses the FileJournalManager's purgeLogsOlderThan(long) 
method to handle old edits removal
 * the purge logic has extended to do the following:
 ** purge edits file segments that are ending before the given transaction ID
 ** purge edits in progress files of any kind (normal leftover, marked as 
trash, empty, stale, and corrupted that is starting before the given 
transaction ID
 ** edits in progress files that are normal leftovers but starting after the 
given transaction ID are marked as stale

Added a test that tests the removal of the new files, and actually as I thought 
it through now, I will file a new patch shortly with a new test, that tests the 
removal of marked files.

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch, HDFS-8178.004.patch, HDFS-8178.005.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-8178:
---
Status: Patch Available  (was: Open)

Added the new patch with extended tests.

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch, HDFS-8178.004.patch, HDFS-8178.005.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-8178:
---
Attachment: HDFS-8178.005.patch

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch, HDFS-8178.004.patch, HDFS-8178.005.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-8178:
---
Status: Open  (was: Patch Available)

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch, HDFS-8178.004.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913528#comment-16913528
 ] 

Istvan Fajth commented on HDFS-8178:


I have attached a new patch, that tries to incorporate concerns from [~atm].

The current solution does the following:
 * in JNStorage it uses the FileJournalManager's purgeLogsOlderThan(long) 
method to handle old edits removal
 * the purge logic has extended to do the following:
 ** purge edits file segments that are ending before the given transaction ID
 ** purge edits in progress files of any kind (normal leftover, marked as 
trash, empty, stale, and corrupted that is starting before the given 
transaction ID
 ** edits in progress files that are normal leftovers but starting after the 
given transaction ID are marked as stale

Added a test that tests the removal of the new files, and actually as I thought 
it through now, I will file a new patch shortly with a new test, that tests the 
removal of marked files.

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch, HDFS-8178.004.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-8178:
---
Attachment: HDFS-8178.004.patch

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch, HDFS-8178.004.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDFS-8303) QJM should purge old logs in the current directory through FJM

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth reassigned HDFS-8303:
--

Assignee: Istvan Fajth  (was: Zhe Zhang)

> QJM should purge old logs in the current directory through FJM
> --
>
> Key: HDFS-8303
> URL: https://issues.apache.org/jira/browse/HDFS-8303
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8303.0.patch, HDFS-8303.1.patch, HDFS-8303.2.patch
>
>
> As the first step of the consolidation effort, QJM should call its FJM to 
> purge the current directory. 
> The current QJM logic of purging current dir is very similar to FJM purging 
> logic.
> QJM:
> {code}
>  private static final List CURRENT_DIR_PURGE_REGEXES =
>   ImmutableList.of(
> Pattern.compile("edits_\\d+-(\\d+)"),
> Pattern.compile("edits_inprogress_(\\d+)(?:\\..*)?"));
> ...
>   long txid = Long.parseLong(matcher.group(1));
>   if (txid < minTxIdToKeep) {
> LOG.info("Purging no-longer needed file " + txid);
> if (!f.delete()) {
> ...
> {code}
> FJM:
> {code}
>   private static final Pattern EDITS_REGEX = Pattern.compile(
> NameNodeFile.EDITS.getName() + "_(\\d+)-(\\d+)");
>   private static final Pattern EDITS_INPROGRESS_REGEX = Pattern.compile(
> NameNodeFile.EDITS_INPROGRESS.getName() + "_(\\d+)");
>   private static final Pattern EDITS_INPROGRESS_STALE_REGEX = Pattern.compile(
>   NameNodeFile.EDITS_INPROGRESS.getName() + "_(\\d+).*(\\S+)");
> ...
> List editLogs = matchEditLogs(files, true);
> for (EditLogFile log : editLogs) {
>   if (log.getFirstTxId() < minTxIdToKeep &&
>   log.getLastTxId() < minTxIdToKeep) {
> purger.purgeLog(log);
>   }
> }
> {code}
> I can see 2 differences:
> # Different regex in matching for empty/corrupt in-progress files. The FJM 
> pattern makes more sense to me.
> # FJM verifies that both start and end txID of a finalized edit file to be 
> old enough. This doesn't make sense because end txID is always larger than 
> start txID



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-8303) QJM should purge old logs in the current directory through FJM

[Istvan Fajth (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDFS-8303:
---
Attachment: HDFS-8303.2.patch

> QJM should purge old logs in the current directory through FJM
> --
>
> Key: HDFS-8303
> URL: https://issues.apache.org/jira/browse/HDFS-8303
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8303.0.patch, HDFS-8303.1.patch, HDFS-8303.2.patch
>
>
> As the first step of the consolidation effort, QJM should call its FJM to 
> purge the current directory. 
> The current QJM logic of purging current dir is very similar to FJM purging 
> logic.
> QJM:
> {code}
>  private static final List CURRENT_DIR_PURGE_REGEXES =
>   ImmutableList.of(
> Pattern.compile("edits_\\d+-(\\d+)"),
> Pattern.compile("edits_inprogress_(\\d+)(?:\\..*)?"));
> ...
>   long txid = Long.parseLong(matcher.group(1));
>   if (txid < minTxIdToKeep) {
> LOG.info("Purging no-longer needed file " + txid);
> if (!f.delete()) {
> ...
> {code}
> FJM:
> {code}
>   private static final Pattern EDITS_REGEX = Pattern.compile(
> NameNodeFile.EDITS.getName() + "_(\\d+)-(\\d+)");
>   private static final Pattern EDITS_INPROGRESS_REGEX = Pattern.compile(
> NameNodeFile.EDITS_INPROGRESS.getName() + "_(\\d+)");
>   private static final Pattern EDITS_INPROGRESS_STALE_REGEX = Pattern.compile(
>   NameNodeFile.EDITS_INPROGRESS.getName() + "_(\\d+).*(\\S+)");
> ...
> List editLogs = matchEditLogs(files, true);
> for (EditLogFile log : editLogs) {
>   if (log.getFirstTxId() < minTxIdToKeep &&
>   log.getLastTxId() < minTxIdToKeep) {
> purger.purgeLog(log);
>   }
> }
> {code}
> I can see 2 differences:
> # Different regex in matching for empty/corrupt in-progress files. The FJM 
> pattern makes more sense to me.
> # FJM verifies that both start and end txID of a finalized edit file to be 
> old enough. This doesn't make sense because end txID is always larger than 
> start txID



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8303) QJM should purge old logs in the current directory through FJM

[Istvan Fajth (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912236#comment-16912236
 ] 

Istvan Fajth commented on HDFS-8303:


As this one is a dependency of HDFS-8178 I started to work on, I am assigning 
this one as well to me. I am also attaching a patch based on the original 
changeset that applies.

> QJM should purge old logs in the current directory through FJM
> --
>
> Key: HDFS-8303
> URL: https://issues.apache.org/jira/browse/HDFS-8303
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8303.0.patch, HDFS-8303.1.patch
>
>
> As the first step of the consolidation effort, QJM should call its FJM to 
> purge the current directory. 
> The current QJM logic of purging current dir is very similar to FJM purging 
> logic.
> QJM:
> {code}
>  private static final List CURRENT_DIR_PURGE_REGEXES =
>   ImmutableList.of(
> Pattern.compile("edits_\\d+-(\\d+)"),
> Pattern.compile("edits_inprogress_(\\d+)(?:\\..*)?"));
> ...
>   long txid = Long.parseLong(matcher.group(1));
>   if (txid < minTxIdToKeep) {
> LOG.info("Purging no-longer needed file " + txid);
> if (!f.delete()) {
> ...
> {code}
> FJM:
> {code}
>   private static final Pattern EDITS_REGEX = Pattern.compile(
> NameNodeFile.EDITS.getName() + "_(\\d+)-(\\d+)");
>   private static final Pattern EDITS_INPROGRESS_REGEX = Pattern.compile(
> NameNodeFile.EDITS_INPROGRESS.getName() + "_(\\d+)");
>   private static final Pattern EDITS_INPROGRESS_STALE_REGEX = Pattern.compile(
>   NameNodeFile.EDITS_INPROGRESS.getName() + "_(\\d+).*(\\S+)");
> ...
> List editLogs = matchEditLogs(files, true);
> for (EditLogFile log : editLogs) {
>   if (log.getFirstTxId() < minTxIdToKeep &&
>   log.getLastTxId() < minTxIdToKeep) {
> purger.purgeLog(log);
>   }
> }
> {code}
> I can see 2 differences:
> # Different regex in matching for empty/corrupt in-progress files. The FJM 
> pattern makes more sense to me.
> # FJM verifies that both start and end txID of a finalized edit file to be 
> old enough. This doesn't make sense because end txID is always larger than 
> start txID



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth reassigned HDFS-8178:
--

Assignee: Istvan Fajth  (was: Zhe Zhang)

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Istvan Fajth
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8178) QJM doesn't move aside stale inprogress edits files

[Istvan Fajth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908156#comment-16908156
 ] 

Istvan Fajth commented on HDFS-8178:


I would be interested in picking up this thing, and push it through after 
actualizing the patch if needed, I am assigning the issue to me based on 
offline discussions with [~jojochuang]. If you have any concerns, please let me 
know.

> QJM doesn't move aside stale inprogress edits files
> ---
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: qjm
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>Priority: Major
>  Labels: BB2015-05-TBR
> Attachments: HDFS-8178.000.patch, HDFS-8178.002.patch, 
> HDFS-8178.003.patch
>
>
> When a QJM crashes, the in-progress edit log file at that time remains in the 
> file system. When the node comes back, it will accept new edit logs and those 
> stale in-progress files are never cleaned up. QJM treats them as regular 
> in-progress edit log files and tries to finalize them, which potentially 
> causes high memory usage. This JIRA aims to move aside those stale edit log 
> files to avoid this scenario.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-13322) fuse dfs - uid persists when switching between ticket caches

[Istvan Fajth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-13322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907005#comment-16907005
 ] 

Istvan Fajth commented on HDFS-13322:
-

Further information, that we discovered and discussed with [~abukor] and 
[~wolfosis] after dig into this together a bit more:

The FUSE context struct does not expose anything from the callers environment, 
mostly as it is not possible from the FUSE code's perspective, the struct we 
get in the calls coming in to fuse-dfs code contains the uid, gid, pid, umask 
of the caller, the fuse struct (which contains implementation details, and 
mount args mostly), and private data that the FS itself exposes. See the [code 
here|https://github.com/libfuse/libfuse/blob/master/include/fuse.h#L786-L804].

The limitation comes from deeper levels in the OS and process handling in 
POSIX, which is summarized pretty well in [this StackExchange 
question|https://unix.stackexchange.com/questions/29128/how-to-read-environment-variables-of-a-process]
 (See the answer from Jonathan Ben-Avraham edited by Toby Speight, currently 
the second answer).
 In a nutshell: the kernel, when executes and starts a process, puts the 
initial environment onto the stack of the process into a fixed length 
structure, this area of the stack is exposed in the /proc/[pid]/environ system 
path. After the start of the process, a POSIX process has a global __environ 
variable that is allocated and updated in the heap of the process, by libc 
routines every time the environment changes. This area is not accessible for 
other processes or for the kernel, at least it is not easy to access and the 
access is restricted, as you need ptrace, and the symbol table of the caller 
process, and also access permissions to the memory of the other process to do 
so.

Based on the research, this limitation is not something we can resolve from the 
fuse-dfs code, so if you need to use this feature, you need to apply the 
workaround and ensure that all dfs access via the fuse mount has the proper 
initial environment (i.e: it is forked from a process that already has the 
environment variable set so the forked process can inherit it in its initial 
environment.)

> fuse dfs - uid persists when switching between ticket caches
> 
>
> Key: HDFS-13322
> URL: https://issues.apache.org/jira/browse/HDFS-13322
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fuse-dfs
>Affects Versions: 2.6.0
> Environment: Linux xx.xx.xx.xxx 3.10.0-514.el7.x86_64 #1 SMP Wed 
> Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
>  
>Reporter: Shoeb Sheyx
>Assignee: Istvan Fajth
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HDFS-13322.001.patch, HDFS-13322.002.patch, 
> HDFS-13322.003.patch, TestFuse.java, TestFuse2.java, catter.sh, catter2.sh, 
> perftest_new_behaviour_10k_different_1KB.txt, perftest_new_behaviour_1B.txt, 
> perftest_new_behaviour_1KB.txt, perftest_new_behaviour_1MB.txt, 
> perftest_old_behaviour_10k_different_1KB.txt, perftest_old_behaviour_1B.txt, 
> perftest_old_behaviour_1KB.txt, perftest_old_behaviour_1MB.txt, 
> testHDFS-13322.sh, test_after_patch.out, test_before_patch.out
>
>
> The symptoms of this issue are the same as described in HDFS-3608 except the 
> workaround that was applied (detect changes in UID ticket cache) doesn't 
> resolve the issue when multiple ticket caches are in use by the same user.
> Our use case requires that a job scheduler running as a specific uid obtain 
> separate kerberos sessions per job and that each of these sessions use a 
> separate cache. When switching sessions this way, no change is made to the 
> original ticket cache so the cached filesystem instance doesn't get 
> regenerated.
>  
> {{$ export KRB5CCNAME=/tmp/krb5cc_session1}}
> {{$ kinit user_a@domain}}
> {{$ touch /fuse_mount/tmp/testfile1}}
> {{$ ls -l /fuse_mount/tmp/testfile1}}
> {{ *-rwxrwxr-x 1 user_a user_a 0 Mar 21 13:37 /fuse_mount/tmp/testfile1*}}
> {{$ export KRB5CCNAME=/tmp/krb5cc_session2}}
> {{$ kinit user_b@domain}}
> {{$ touch /fuse_mount/tmp/testfile2}}
> {{$ ls -l /fuse_mount/tmp/testfile2}}
> {{ *-rwxrwxr-x 1 user_a user_a 0 Mar 21 13:37 /fuse_mount/tmp/testfile2*}}
> {{   }}{color:#d04437}*{{** expected owner to be user_b **}}*{color}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-13322) fuse dfs - uid persists when switching between ticket caches

[Istvan Fajth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-13322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893236#comment-16893236
 ] 

Istvan Fajth commented on HDFS-13322:
-

To document a new learning point regarding this change, I would like to add the 
following information:

FUSE is figuring out the environment of the caller based on its pid that is 
passed on to the FUSE code from the kernel in the FuseContext structure. With 
the pid fuse-dfs is turning to /proc/(context->pid)/environ file to find the 
KRB5CCNAME environment variable.

After the change the connection cache keys are consists of a (username, 
kerberos ticket cache path) pair if authentication is set to KERBEROS, so for 
every pair we hold a different connection build with the ticket cache path, and 
we use that later on. With SIMPLE authentication, the ticket cache path is 
presented in the pair as the \0 character always.

For this the ticket cache pathin case of KERBEROS authentication is being read 
from /proc/(context->pid)/environ on every access.

In the Linux [proc file system man 
page|http://man7.org/linux/man-pages/man5/proc.5.html] the following is written 
for /proc/[pid]/environ:
{quote}This file contains the *initial* environment that was set when the 
currently executing program was started via execve(2).{quote}

This can lead to odd behaviors in case the access is not happening in a new 
process but it is part of a process that exported the KRB5CCNAME environment 
variable. So for example in a shell when executing the following commands, FUSE 
will not be able to read the KRB5CCNAME variable from the 
/proc/(context->pid)/environ file:
{code:java}
$ export KRB5CCNAME=/tmp/myticketcache
$ echo "foo" > /mnt/hdfs/tmp/foo.txt{code}
This is because in this case echo is happening in the shell, and the shell's 
process id will be there in context->pid, and the /proc/(context->pid)/environ 
file will not contain the environment variable KRB5CCNAME as it is not part of 
the initial environment.

In the meantime the following will work because cp will be a new process which 
inherits the environment from the current shell:
{code:java}
$ export KRB5CCNAME=/tmp/myticketcache
$ echo "foo" > /tmp/foo.txt
$ cp /tmp/foo.txt /mnt/hdfs/tmp/foo.txt{code}
 

To workaround this behaviour, the caller has to ensure that the initial 
environment of every accessing process has the correct KRB5CCNAME set. So for 
example the echo example would work correctly the following way:
{code:java}
$ export KRB5CCNAME=/tmp/myticketcache
$ /bin/sh
$ echo "foo" > /mnt/hdfs/tmp/foo.txt
$ exit{code}

> fuse dfs - uid persists when switching between ticket caches
> 
>
> Key: HDFS-13322
> URL: https://issues.apache.org/jira/browse/HDFS-13322
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fuse-dfs
>Affects Versions: 2.6.0
> Environment: Linux xx.xx.xx.xxx 3.10.0-514.el7.x86_64 #1 SMP Wed 
> Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
>  
>Reporter: Shoeb Sheyx
>Assignee: Istvan Fajth
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HDFS-13322.001.patch, HDFS-13322.002.patch, 
> HDFS-13322.003.patch, TestFuse.java, TestFuse2.java, catter.sh, catter2.sh, 
> perftest_new_behaviour_10k_different_1KB.txt, perftest_new_behaviour_1B.txt, 
> perftest_new_behaviour_1KB.txt, perftest_new_behaviour_1MB.txt, 
> perftest_old_behaviour_10k_different_1KB.txt, perftest_old_behaviour_1B.txt, 
> perftest_old_behaviour_1KB.txt, perftest_old_behaviour_1MB.txt, 
> testHDFS-13322.sh, test_after_patch.out, test_before_patch.out
>
>
> The symptoms of this issue are the same as described in HDFS-3608 except the 
> workaround that was applied (detect changes in UID ticket cache) doesn't 
> resolve the issue when multiple ticket caches are in use by the same user.
> Our use case requires that a job scheduler running as a specific uid obtain 
> separate kerberos sessions per job and that each of these sessions use a 
> separate cache. When switching sessions this way, no change is made to the 
> original ticket cache so the cached filesystem instance doesn't get 
> regenerated.
>  
> {{$ export KRB5CCNAME=/tmp/krb5cc_session1}}
> {{$ kinit user_a@domain}}
> {{$ touch /fuse_mount/tmp/testfile1}}
> {{$ ls -l /fuse_mount/tmp/testfile1}}
> {{ *-rwxrwxr-x 1 user_a user_a 0 Mar 21 13:37 /fuse_mount/tmp/testfile1*}}
> {{$ export KRB5CCNAME=/tmp/krb5cc_session2}}
> {{$ kinit user_b@domain}}
> {{$ touch /fuse_mount/tmp/testfile2}}
> {{$ ls -l /fuse_mount/tmp/testfile2}}
> {{ *-rwxrwxr-x 1 user_a user_a 0 Mar 21 13:37 /fuse_mount/tmp/testfile2*}}
> {{   }}{color:#d04437}*{{** expected owner to be user_b **}}*{color}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (HDDS-1762) Reorganize docker-compose examples

[Istvan Fajth (JIRA)]

Istvan Fajth created HDDS-1762:
--

 Summary: Reorganize docker-compose examples
 Key: HDDS-1762
 URL: https://issues.apache.org/jira/browse/HDDS-1762
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: docker
Affects Versions: 0.4.0
Reporter: Istvan Fajth
Assignee: Istvan Fajth


The current docker example directories are all containing their own 
docker-compose.yaml, docker-config, .env, and sometimes test.sh files.

There are a lot of duplicates, and if there is something new that needs to be 
set, it is error prone to set it in all the directories.

The aim of this ticket is to organize the common parts into one place, and use 
the inheritance and dynamic configuration possibilities of docker to have all 
things written in one place, so that maintainability get easier. On the 
contrary, it would be a bit harder to understand the things forr the first 
sight, so documentation needs to be improved as well to guide the reader more 
on what is happening where.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1742) Merge ozone-perf and ozonetrace example clusters

[Istvan Fajth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Fajth updated HDDS-1742:
---
Status: Patch Available  (was: Open)

> Merge ozone-perf and ozonetrace example clusters
> 
>
> Key: HDDS-1742
> URL: https://issues.apache.org/jira/browse/HDDS-1742
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: docker
>Reporter: Elek, Marton
>Assignee: Istvan Fajth
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We have multiple example clusters in hadoop-ozone/dist/src/main/compose to 
> demonstrate how different type of configuration can be set with ozone.
> But some of them can be consolidated. I propose to combine ozonetrace to 
> ozoneperf to one ozoneperf which includes all the required components for a 
> local performance testing:
>  # opentracing (jaeger component in docker-compose + environment variables)
>  # monitoring (grafana + prometheus)
>  # perf profile (as of now it's enabled only in the ozone cluster[1])
>  
> [1]
> {code:java}
> cat compose/ozone/docker-config | grep prof
> OZONE-SITE.XML_hdds.profiler.endpoint.enabled=true
> ASYNC_PROFILER_HOME=/opt/profiler
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1741) Fix prometheus configuration in ozoneperf example cluster

[Istvan Fajth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876550#comment-16876550
 ] 

Istvan Fajth commented on HDDS-1741:


Hi [~elek],

I have added the PR to fix this config problem, based on a quick check it is 
working properly in my env after a clean build.

> Fix prometheus configuration in ozoneperf example cluster
> -
>
> Key: HDDS-1741
> URL: https://issues.apache.org/jira/browse/HDDS-1741
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: docker
>Affects Versions: 0.4.0
>Reporter: Elek, Marton
>Assignee: Istvan Fajth
>Priority: Trivial
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> HDDS-1216 renamed the ozoneManager components to om in the docker-compose 
> file. But the prometheus configuration of the compose/ozoneperf environment 
> is not updated.
> We need to updated it to get meaningful metrics from om.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-14619) chmod changes the mask when ACL is enabled

[Istvan Fajth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-14619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875262#comment-16875262
 ] 

Istvan Fajth commented on HDFS-14619:
-

Based on the description and the explanation and also further reviewing the 
things, I think this is not a problem, [~smeng] if you think so as well, then 
we may close this one as well as I have closed the related issue already.

> chmod changes the mask when ACL is enabled
> --
>
> Key: HDFS-14619
> URL: https://issues.apache.org/jira/browse/HDFS-14619
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 3.1.2
>Reporter: Siyao Meng
>Priority: Major
>
> When setting a directory's permission with HDFS shell chmod, it changes the 
> ACL mask instead of the permission bits:
> {code:bash}
> $ sudo -u impala hdfs dfs -getfacl /user/hive/warehouse/exttablename/key=1/
> # file: /user/hive/warehouse/exttablename/key=1
> # owner: hive
> # group: hive
> user::rwx
> user:impala:rwx   #effective:r-x
> group::rwx#effective:r-x
> mask::r-x
> other::r-x
> default:user::rwx
> default:user:impala:rwx
> default:group::rwx
> default:mask::rwx
> default:other::rwx
> $ sudo -u hdfs hdfs dfs -chmod 777 /user/hive/warehouse/exttablename/key=1/
> $ sudo -u impala hdfs dfs -getfacl /user/hive/warehouse/exttablename/key=1/
> # file: /user/hive/warehouse/exttablename/key=1
> # owner: hive
> # group: hive
> user::rwx
> user:impala:rwx
> group::rwx
> mask::rwx
> other::rwx
> default:user::rwx
> default:user:impala:rwx
> default:group::rwx
> default:mask::rwx
> default:other::rwx
> $ sudo -u hdfs hdfs dfs -chmod 755 /user/hive/warehouse/exttablename/key=1/
> $ sudo -u impala hdfs dfs -getfacl /user/hive/warehouse/exttablename/key=1/
> # file: /user/hive/warehouse/exttablename/key=1
> # owner: hive
> # group: hive
> user::rwx
> user:impala:rwx   #effective:r-x
> group::rwx#effective:r-x
> mask::r-x
> other::r-x
> default:user::rwx
> default:user:impala:rwx
> default:group::rwx
> default:mask::rwx
> default:other::rwx
> $ sudo -u impala hdfs dfs -touch /user/hive/warehouse/exttablename/key=1/file
> touch: Permission denied: user=impala, access=WRITE, 
> inode="/user/hive/warehouse/exttablename/key=1/file":hive:hive:drwxr-xr-x
> {code}
> The cluster has dfs.namenode.acls.enabled=true and 
> dfs.namenode.posix.acl.inheritance.enabled=true.
> As far as I understand, the chmod should change the permission bits instead 
> of the ACL mask. CMIIW
> Might be related to HDFS-14517. [~pifta]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org