[jira] [Updated] (AMBARI-21070) Race condition: webhdfs call mkdir /tmp/druid-indexing before /tmp making tmp not writable.
[ https://issues.apache.org/jira/browse/AMBARI-21070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Onischuk updated AMBARI-21070: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk and branch-2.5 > Race condition: webhdfs call mkdir /tmp/druid-indexing before /tmp making > tmp not writable. > -- > > Key: AMBARI-21070 > URL: https://issues.apache.org/jira/browse/AMBARI-21070 > Project: Ambari > Issue Type: Bug >Reporter: Andrew Onischuk >Assignee: Andrew Onischuk > Fix For: 2.5.2 > > Attachments: AMBARI-21070.patch > > > Race condition: webhdfs call mkdir /tmp/druid-indexing before /tmp making tmp > not writable. > @HDP install through ambari , just at the step start components on host< > we > have some webhdfs operations in background which is creating HDFS directory > structures required for specific components like (/tmp, /tmp/hive /user/druid > /tmp/druid-indexing ...) > generally the expected order is getfileInfo : /tmp --> mkdir: /tmp > changePermission: /tmp to 777 (hdfs:hdfs) so that /tmp is accessible to all , > hence hivemetastore able to create /tmp/hive(hive scratch directory) > But here in this case specific to druid install , most of the times mkdir of > /tmp/druid-indexing called before(actual /tmp creation) and thus /tmp is > having just default directory permission(755). > ->So next call of getfileInfo : /tmp says already exist it will not further > create and change permission > This made /tmp not accessible to write, So HiveServer process gets shutdown as > it unable to create/access /tmp/hive. > hdfs-audit log: > > > > 2017-05-12 06:39:51,067 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.26.3 cmd=getfileinfo src=/tmp/druid-indexing > dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,120 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.22.81cmd=contentSummary > src=/user/druid dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,133 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setPermission > src=/ats/active dst=nullperm=hdfs:hadoop:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,155 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.26.3 cmd=mkdirs src=/tmp/druid-indexing > dst=nullperm=hdfs:hdfs:rwxr-xr-xproto=webhdfs > 2017-05-12 06:39:51,206 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.22.81cmd=listStatus src=/user/druid > dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,235 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setPermission src=/ats/ > dst=nullperm=yarn:hadoop:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,249 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.26.3 cmd=setPermission > src=/tmp/druid-indexing dst=nullperm=hdfs:hdfs:rwxr-xr-x > proto=webhdfs > 2017-05-12 06:39:51,290 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.22.81cmd=listStatus src=/user/druid/data > dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,339 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setPermission > src=/ats/active/dst=nullperm=hdfs:hadoop:rwxr-xr-x > proto=webhdfs > 2017-05-12 06:39:51,341 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.26.3 cmd=setOwnersrc=/tmp/druid-indexing > dst=nullperm=druid:hdfs:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,380 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.22.81cmd=setOwnersrc=/user/druid/data > dst=nullperm=druid:hdfs:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,431 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setOwnersrc=/ats/active > dst=nullperm=yarn:hadoop:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,526 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setOwnersrc=/ats/ > dst=nullperm=yarn:hadoop:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,580 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.32.12cmd=getfileinfo > src=/apps/hbase/staging dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,620 INFO FSNamesystem.audit:
[jira] [Updated] (AMBARI-21070) Race condition: webhdfs call mkdir /tmp/druid-indexing before /tmp making tmp not writable.
[ https://issues.apache.org/jira/browse/AMBARI-21070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated AMBARI-21070: --- Fix Version/s: (was: 2.5.1) 2.5.2 Moving out of 2.5.1 since release is done. > Race condition: webhdfs call mkdir /tmp/druid-indexing before /tmp making > tmp not writable. > -- > > Key: AMBARI-21070 > URL: https://issues.apache.org/jira/browse/AMBARI-21070 > Project: Ambari > Issue Type: Bug >Reporter: Andrew Onischuk >Assignee: Andrew Onischuk > Fix For: 2.5.2 > > Attachments: AMBARI-21070.patch > > > Race condition: webhdfs call mkdir /tmp/druid-indexing before /tmp making tmp > not writable. > @HDP install through ambari , just at the step start components on host< > we > have some webhdfs operations in background which is creating HDFS directory > structures required for specific components like (/tmp, /tmp/hive /user/druid > /tmp/druid-indexing ...) > generally the expected order is getfileInfo : /tmp --> mkdir: /tmp > changePermission: /tmp to 777 (hdfs:hdfs) so that /tmp is accessible to all , > hence hivemetastore able to create /tmp/hive(hive scratch directory) > But here in this case specific to druid install , most of the times mkdir of > /tmp/druid-indexing called before(actual /tmp creation) and thus /tmp is > having just default directory permission(755). > ->So next call of getfileInfo : /tmp says already exist it will not further > create and change permission > This made /tmp not accessible to write, So HiveServer process gets shutdown as > it unable to create/access /tmp/hive. > hdfs-audit log: > > > > 2017-05-12 06:39:51,067 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.26.3 cmd=getfileinfo src=/tmp/druid-indexing > dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,120 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.22.81cmd=contentSummary > src=/user/druid dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,133 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setPermission > src=/ats/active dst=nullperm=hdfs:hadoop:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,155 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.26.3 cmd=mkdirs src=/tmp/druid-indexing > dst=nullperm=hdfs:hdfs:rwxr-xr-xproto=webhdfs > 2017-05-12 06:39:51,206 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.22.81cmd=listStatus src=/user/druid > dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,235 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setPermission src=/ats/ > dst=nullperm=yarn:hadoop:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,249 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.26.3 cmd=setPermission > src=/tmp/druid-indexing dst=nullperm=hdfs:hdfs:rwxr-xr-x > proto=webhdfs > 2017-05-12 06:39:51,290 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.22.81cmd=listStatus src=/user/druid/data > dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,339 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setPermission > src=/ats/active/dst=nullperm=hdfs:hadoop:rwxr-xr-x > proto=webhdfs > 2017-05-12 06:39:51,341 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.26.3 cmd=setOwnersrc=/tmp/druid-indexing > dst=nullperm=druid:hdfs:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,380 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.22.81cmd=setOwnersrc=/user/druid/data > dst=nullperm=druid:hdfs:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,431 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setOwnersrc=/ats/active > dst=nullperm=yarn:hadoop:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,526 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setOwnersrc=/ats/ > dst=nullperm=yarn:hadoop:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,580 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.32.12cmd=getfileinfo > src=/apps/hbase/staging dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,620 INFO FSNamesystem.audit:
[jira] [Updated] (AMBARI-21070) Race condition: webhdfs call mkdir /tmp/druid-indexing before /tmp making tmp not writable.
[ https://issues.apache.org/jira/browse/AMBARI-21070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Onischuk updated AMBARI-21070: - Attachment: AMBARI-21070.patch > Race condition: webhdfs call mkdir /tmp/druid-indexing before /tmp making > tmp not writable. > -- > > Key: AMBARI-21070 > URL: https://issues.apache.org/jira/browse/AMBARI-21070 > Project: Ambari > Issue Type: Bug >Reporter: Andrew Onischuk >Assignee: Andrew Onischuk > Fix For: 2.5.1 > > Attachments: AMBARI-21070.patch > > > Race condition: webhdfs call mkdir /tmp/druid-indexing before /tmp making tmp > not writable. > @HDP install through ambari , just at the step start components on host< > we > have some webhdfs operations in background which is creating HDFS directory > structures required for specific components like (/tmp, /tmp/hive /user/druid > /tmp/druid-indexing ...) > generally the expected order is getfileInfo : /tmp --> mkdir: /tmp > changePermission: /tmp to 777 (hdfs:hdfs) so that /tmp is accessible to all , > hence hivemetastore able to create /tmp/hive(hive scratch directory) > But here in this case specific to druid install , most of the times mkdir of > /tmp/druid-indexing called before(actual /tmp creation) and thus /tmp is > having just default directory permission(755). > ->So next call of getfileInfo : /tmp says already exist it will not further > create and change permission > This made /tmp not accessible to write, So HiveServer process gets shutdown as > it unable to create/access /tmp/hive. > hdfs-audit log: > > > > 2017-05-12 06:39:51,067 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.26.3 cmd=getfileinfo src=/tmp/druid-indexing > dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,120 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.22.81cmd=contentSummary > src=/user/druid dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,133 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setPermission > src=/ats/active dst=nullperm=hdfs:hadoop:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,155 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.26.3 cmd=mkdirs src=/tmp/druid-indexing > dst=nullperm=hdfs:hdfs:rwxr-xr-xproto=webhdfs > 2017-05-12 06:39:51,206 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.22.81cmd=listStatus src=/user/druid > dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,235 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setPermission src=/ats/ > dst=nullperm=yarn:hadoop:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,249 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.26.3 cmd=setPermission > src=/tmp/druid-indexing dst=nullperm=hdfs:hdfs:rwxr-xr-x > proto=webhdfs > 2017-05-12 06:39:51,290 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.22.81cmd=listStatus src=/user/druid/data > dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,339 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setPermission > src=/ats/active/dst=nullperm=hdfs:hadoop:rwxr-xr-x > proto=webhdfs > 2017-05-12 06:39:51,341 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.26.3 cmd=setOwnersrc=/tmp/druid-indexing > dst=nullperm=druid:hdfs:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,380 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.22.81cmd=setOwnersrc=/user/druid/data > dst=nullperm=druid:hdfs:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,431 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setOwnersrc=/ats/active > dst=nullperm=yarn:hadoop:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,526 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setOwnersrc=/ats/ > dst=nullperm=yarn:hadoop:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,580 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.32.12cmd=getfileinfo > src=/apps/hbase/staging dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,620 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200
[jira] [Updated] (AMBARI-21070) Race condition: webhdfs call mkdir /tmp/druid-indexing before /tmp making tmp not writable.
[ https://issues.apache.org/jira/browse/AMBARI-21070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Onischuk updated AMBARI-21070: - Status: Patch Available (was: Open) > Race condition: webhdfs call mkdir /tmp/druid-indexing before /tmp making > tmp not writable. > -- > > Key: AMBARI-21070 > URL: https://issues.apache.org/jira/browse/AMBARI-21070 > Project: Ambari > Issue Type: Bug >Reporter: Andrew Onischuk >Assignee: Andrew Onischuk > Fix For: 2.5.1 > > Attachments: AMBARI-21070.patch > > > Race condition: webhdfs call mkdir /tmp/druid-indexing before /tmp making tmp > not writable. > @HDP install through ambari , just at the step start components on host< > we > have some webhdfs operations in background which is creating HDFS directory > structures required for specific components like (/tmp, /tmp/hive /user/druid > /tmp/druid-indexing ...) > generally the expected order is getfileInfo : /tmp --> mkdir: /tmp > changePermission: /tmp to 777 (hdfs:hdfs) so that /tmp is accessible to all , > hence hivemetastore able to create /tmp/hive(hive scratch directory) > But here in this case specific to druid install , most of the times mkdir of > /tmp/druid-indexing called before(actual /tmp creation) and thus /tmp is > having just default directory permission(755). > ->So next call of getfileInfo : /tmp says already exist it will not further > create and change permission > This made /tmp not accessible to write, So HiveServer process gets shutdown as > it unable to create/access /tmp/hive. > hdfs-audit log: > > > > 2017-05-12 06:39:51,067 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.26.3 cmd=getfileinfo src=/tmp/druid-indexing > dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,120 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.22.81cmd=contentSummary > src=/user/druid dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,133 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setPermission > src=/ats/active dst=nullperm=hdfs:hadoop:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,155 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.26.3 cmd=mkdirs src=/tmp/druid-indexing > dst=nullperm=hdfs:hdfs:rwxr-xr-xproto=webhdfs > 2017-05-12 06:39:51,206 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.22.81cmd=listStatus src=/user/druid > dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,235 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setPermission src=/ats/ > dst=nullperm=yarn:hadoop:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,249 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.26.3 cmd=setPermission > src=/tmp/druid-indexing dst=nullperm=hdfs:hdfs:rwxr-xr-x > proto=webhdfs > 2017-05-12 06:39:51,290 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.22.81cmd=listStatus src=/user/druid/data > dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,339 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setPermission > src=/ats/active/dst=nullperm=hdfs:hadoop:rwxr-xr-x > proto=webhdfs > 2017-05-12 06:39:51,341 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.26.3 cmd=setOwnersrc=/tmp/druid-indexing > dst=nullperm=druid:hdfs:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,380 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.22.81cmd=setOwnersrc=/user/druid/data > dst=nullperm=druid:hdfs:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,431 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setOwnersrc=/ats/active > dst=nullperm=yarn:hadoop:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,526 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200 cmd=setOwnersrc=/ats/ > dst=nullperm=yarn:hadoop:rwxr-xr-x proto=webhdfs > 2017-05-12 06:39:51,580 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.32.12cmd=getfileinfo > src=/apps/hbase/staging dst=nullperm=null proto=webhdfs > 2017-05-12 06:39:51,620 INFO FSNamesystem.audit: allowed=true ugi=hdfs > (auth:SIMPLE) ip=/172.27.37.200