[jira] [Commented] (KUDU-1855) Kudu file UNIX permissions are inconsistent
[
https://issues.apache.org/jira/browse/KUDU-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866703#comment-15866703
]
Adar Dembo commented on KUDU-1855:
--
bq. Do you think that's OK, or should we try to make the temporary files also
conform to the configured umask?
I think they should conform too, if only to better adhere to the principle of
least surprise. Could probably just fchmod() before closing the temp file, or
chmod() after closing it.
> Kudu file UNIX permissions are inconsistent
> ---
>
> Key: KUDU-1855
> URL: https://issues.apache.org/jira/browse/KUDU-1855
> Project: Kudu
> Issue Type: Bug
> Components: security
>Affects Versions: 1.3.0
>Reporter: Adar Dembo
>Priority: Blocker
>
> Right now the access modes of Kudu files on disk are quite inconsistent. For
> example:
> {noformat}
> 6942960 drwxr-xr-x 4 kudu kudu 27 Jun 21 2016
> /data/1/kudu/tablet
> 21501102670 drwxr-xr-x 5 kudu kudu 71 Jan 17 17:42
> /data/1/kudu/tablet/data
> 21501241274 -rw--- 1 kudu kudu 665 Jun 21 2016
> /data/1/kudu/tablet/data/instance
> 702851 16 drwxr-xr-x 2 kudu kudu12288 Jan 31 14:02
> /data/1/kudu/tablet/data/tablet-meta
> 702894 12 -rw--- 1 kudu kudu 9501 Jan 22 13:38
> /data/1/kudu/tablet/data/tablet-meta/33cc61001d1442048a588f930d973464
> ...
> 21538687298 -rw--- 1 kudu kudu 8109 Jan 12 22:26
> /data/1/kudu/tablet/data/consensus-meta/83a2f75a88bd48f5a8ec28f8328af481
> 702854 152 drwxr-xr-x 2 kudu kudu 110592 Jan 23 16:59
> /data/1/kudu/tablet/data/data
> 15456410 -rw-r--r-- 1 kudu kudu 10743091200 Jan 9 13:06
> /data/1/kudu/tablet/data/data/bdf6d87c2ba34d598327b0b9e159a5ea.data
> 7056904 -rw--- 1 kudu kudu 990 Jun 21 2016
> /data/1/kudu/tablet/data/data/block_manager_instance
> 278147 652 -rw-r--r-- 1 kudu kudu 660084 Aug 15 00:09
> /data/1/kudu/tablet/data/data/3917e1185471419786f59ad72bb7ba63.metadata
> 2905870 -rw-r--r-- 1 kudu kudu 10746855424 Jan 9 13:05
> /data/1/kudu/tablet/data/data/3917e1185471419786f59ad72bb7ba63.data
> 291129 672 -rw-r--r-- 1 kudu kudu 681669 Aug 15 00:09
> /data/1/kudu/tablet/data/data/756bd793f0c24188853e096f835ba7b4.metadata
> {noformat}
> I suspect it's due to the difference of "files opened as temp files and
> renamed into place" and "files opened directly". The former have permissions
> that are even more restrictive than the umask (022), while the latter adhere
> to the umask.
> I don't know whether more restrictive or less restrictive modes should be
> used, but I don't see any reason why all files in Kudu's filesystem layout
> shouldn't have the same permissions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KUDU-1855) Kudu file UNIX permissions are inconsistent
[
https://issues.apache.org/jira/browse/KUDU-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866522#comment-15866522
]
Todd Lipcon commented on KUDU-1855:
---
I'm working on a patch that does the following:
- add a new --umask flag which can be set to the octal-formatted umask. The
default is 077 (so files would be only readable/writable by the kudu user
itself)
- when the fs manager starts up, it checks that the existing directories
conform to the configured umask and chmods them if not (eg 0755 -> 0700)
One question I'd like another opinion on: even with this setting, files created
using pb_util will end up with 0600 permissions (even if the user had set the
umask to 077) This means it's impossible to configure Kudu in such a way that
the kudu *group* can read the data on disk. Do you think that's OK, or should
we try to make the temporary files also conform to the configured umask?
The other places we create files either use a umask 0666 (log_index.cc) or 0664
(env_posix.cc). So, we should probably try to make them all consistent.
> Kudu file UNIX permissions are inconsistent
> ---
>
> Key: KUDU-1855
> URL: https://issues.apache.org/jira/browse/KUDU-1855
> Project: Kudu
> Issue Type: Bug
> Components: security
>Affects Versions: 1.3.0
>Reporter: Adar Dembo
>Priority: Blocker
>
> Right now the access modes of Kudu files on disk are quite inconsistent. For
> example:
> {noformat}
> 6942960 drwxr-xr-x 4 kudu kudu 27 Jun 21 2016
> /data/1/kudu/tablet
> 21501102670 drwxr-xr-x 5 kudu kudu 71 Jan 17 17:42
> /data/1/kudu/tablet/data
> 21501241274 -rw--- 1 kudu kudu 665 Jun 21 2016
> /data/1/kudu/tablet/data/instance
> 702851 16 drwxr-xr-x 2 kudu kudu12288 Jan 31 14:02
> /data/1/kudu/tablet/data/tablet-meta
> 702894 12 -rw--- 1 kudu kudu 9501 Jan 22 13:38
> /data/1/kudu/tablet/data/tablet-meta/33cc61001d1442048a588f930d973464
> ...
> 21538687298 -rw--- 1 kudu kudu 8109 Jan 12 22:26
> /data/1/kudu/tablet/data/consensus-meta/83a2f75a88bd48f5a8ec28f8328af481
> 702854 152 drwxr-xr-x 2 kudu kudu 110592 Jan 23 16:59
> /data/1/kudu/tablet/data/data
> 15456410 -rw-r--r-- 1 kudu kudu 10743091200 Jan 9 13:06
> /data/1/kudu/tablet/data/data/bdf6d87c2ba34d598327b0b9e159a5ea.data
> 7056904 -rw--- 1 kudu kudu 990 Jun 21 2016
> /data/1/kudu/tablet/data/data/block_manager_instance
> 278147 652 -rw-r--r-- 1 kudu kudu 660084 Aug 15 00:09
> /data/1/kudu/tablet/data/data/3917e1185471419786f59ad72bb7ba63.metadata
> 2905870 -rw-r--r-- 1 kudu kudu 10746855424 Jan 9 13:05
> /data/1/kudu/tablet/data/data/3917e1185471419786f59ad72bb7ba63.data
> 291129 672 -rw-r--r-- 1 kudu kudu 681669 Aug 15 00:09
> /data/1/kudu/tablet/data/data/756bd793f0c24188853e096f835ba7b4.metadata
> {noformat}
> I suspect it's due to the difference of "files opened as temp files and
> renamed into place" and "files opened directly". The former have permissions
> that are even more restrictive than the umask (022), while the latter adhere
> to the umask.
> I don't know whether more restrictive or less restrictive modes should be
> used, but I don't see any reason why all files in Kudu's filesystem layout
> shouldn't have the same permissions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KUDU-1855) Kudu file UNIX permissions are inconsistent
[
https://issues.apache.org/jira/browse/KUDU-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865066#comment-15865066
]
Adar Dembo commented on KUDU-1855:
--
My two cents:
bq. 1) should we explicitly set our own umask on process startup to something
restrictive like 077?
I don't know. I would assume that the system administrator has configured umask
(explicitly or implicitly) to some specific value, and we should honor whatever
it is.
That said, we should probably encourage vendors to default to restrictive
umasks like 077 in their Kudu service scripts and configuration tools.
bq. 2) should we add code to chmod 700 our data/wal dirs on startup, so that
people running previous versions would get their permissions fixed on upgrade?
Maybe not explicitly 700, but we could examine the umask, calculate the
appropriate chmod, and fix up any file/directory permissions that do not
comply. That could be expensive with the file block manager though.
bq. 3) does any of this need to be configurable? are there valid use cases
where people want g+rx or a+rx permissions? (eg maybe so admins in the 'kudu'
group could run offline tools?)
Yes, but see above; I think the umask _is_ that configuration mechanism. Might
be interesting to take a look at how other database projects manage on-disk
file permissions.
> Kudu file UNIX permissions are inconsistent
> ---
>
> Key: KUDU-1855
> URL: https://issues.apache.org/jira/browse/KUDU-1855
> Project: Kudu
> Issue Type: Bug
> Components: security
>Affects Versions: 1.3.0
>Reporter: Adar Dembo
>Priority: Blocker
>
> Right now the access modes of Kudu files on disk are quite inconsistent. For
> example:
> {noformat}
> 6942960 drwxr-xr-x 4 kudu kudu 27 Jun 21 2016
> /data/1/kudu/tablet
> 21501102670 drwxr-xr-x 5 kudu kudu 71 Jan 17 17:42
> /data/1/kudu/tablet/data
> 21501241274 -rw--- 1 kudu kudu 665 Jun 21 2016
> /data/1/kudu/tablet/data/instance
> 702851 16 drwxr-xr-x 2 kudu kudu12288 Jan 31 14:02
> /data/1/kudu/tablet/data/tablet-meta
> 702894 12 -rw--- 1 kudu kudu 9501 Jan 22 13:38
> /data/1/kudu/tablet/data/tablet-meta/33cc61001d1442048a588f930d973464
> ...
> 21538687298 -rw--- 1 kudu kudu 8109 Jan 12 22:26
> /data/1/kudu/tablet/data/consensus-meta/83a2f75a88bd48f5a8ec28f8328af481
> 702854 152 drwxr-xr-x 2 kudu kudu 110592 Jan 23 16:59
> /data/1/kudu/tablet/data/data
> 15456410 -rw-r--r-- 1 kudu kudu 10743091200 Jan 9 13:06
> /data/1/kudu/tablet/data/data/bdf6d87c2ba34d598327b0b9e159a5ea.data
> 7056904 -rw--- 1 kudu kudu 990 Jun 21 2016
> /data/1/kudu/tablet/data/data/block_manager_instance
> 278147 652 -rw-r--r-- 1 kudu kudu 660084 Aug 15 00:09
> /data/1/kudu/tablet/data/data/3917e1185471419786f59ad72bb7ba63.metadata
> 2905870 -rw-r--r-- 1 kudu kudu 10746855424 Jan 9 13:05
> /data/1/kudu/tablet/data/data/3917e1185471419786f59ad72bb7ba63.data
> 291129 672 -rw-r--r-- 1 kudu kudu 681669 Aug 15 00:09
> /data/1/kudu/tablet/data/data/756bd793f0c24188853e096f835ba7b4.metadata
> {noformat}
> I suspect it's due to the difference of "files opened as temp files and
> renamed into place" and "files opened directly". The former have permissions
> that are even more restrictive than the umask (022), while the latter adhere
> to the umask.
> I don't know whether more restrictive or less restrictive modes should be
> used, but I don't see any reason why all files in Kudu's filesystem layout
> shouldn't have the same permissions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KUDU-1855) Kudu file UNIX permissions are inconsistent
[
https://issues.apache.org/jira/browse/KUDU-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864960#comment-15864960
]
Todd Lipcon commented on KUDU-1855:
---
BTW, you're right -- files that are created as temp and moved into place get
permissions 0600 because mkstemp() always uses those permissions regardless of
umask.
> Kudu file UNIX permissions are inconsistent
> ---
>
> Key: KUDU-1855
> URL: https://issues.apache.org/jira/browse/KUDU-1855
> Project: Kudu
> Issue Type: Bug
> Components: security
>Affects Versions: 1.3.0
>Reporter: Adar Dembo
>Priority: Blocker
>
> Right now the access modes of Kudu files on disk are quite inconsistent. For
> example:
> {noformat}
> 6942960 drwxr-xr-x 4 kudu kudu 27 Jun 21 2016
> /data/1/kudu/tablet
> 21501102670 drwxr-xr-x 5 kudu kudu 71 Jan 17 17:42
> /data/1/kudu/tablet/data
> 21501241274 -rw--- 1 kudu kudu 665 Jun 21 2016
> /data/1/kudu/tablet/data/instance
> 702851 16 drwxr-xr-x 2 kudu kudu12288 Jan 31 14:02
> /data/1/kudu/tablet/data/tablet-meta
> 702894 12 -rw--- 1 kudu kudu 9501 Jan 22 13:38
> /data/1/kudu/tablet/data/tablet-meta/33cc61001d1442048a588f930d973464
> ...
> 21538687298 -rw--- 1 kudu kudu 8109 Jan 12 22:26
> /data/1/kudu/tablet/data/consensus-meta/83a2f75a88bd48f5a8ec28f8328af481
> 702854 152 drwxr-xr-x 2 kudu kudu 110592 Jan 23 16:59
> /data/1/kudu/tablet/data/data
> 15456410 -rw-r--r-- 1 kudu kudu 10743091200 Jan 9 13:06
> /data/1/kudu/tablet/data/data/bdf6d87c2ba34d598327b0b9e159a5ea.data
> 7056904 -rw--- 1 kudu kudu 990 Jun 21 2016
> /data/1/kudu/tablet/data/data/block_manager_instance
> 278147 652 -rw-r--r-- 1 kudu kudu 660084 Aug 15 00:09
> /data/1/kudu/tablet/data/data/3917e1185471419786f59ad72bb7ba63.metadata
> 2905870 -rw-r--r-- 1 kudu kudu 10746855424 Jan 9 13:05
> /data/1/kudu/tablet/data/data/3917e1185471419786f59ad72bb7ba63.data
> 291129 672 -rw-r--r-- 1 kudu kudu 681669 Aug 15 00:09
> /data/1/kudu/tablet/data/data/756bd793f0c24188853e096f835ba7b4.metadata
> {noformat}
> I suspect it's due to the difference of "files opened as temp files and
> renamed into place" and "files opened directly". The former have permissions
> that are even more restrictive than the umask (022), while the latter adhere
> to the umask.
> I don't know whether more restrictive or less restrictive modes should be
> used, but I don't see any reason why all files in Kudu's filesystem layout
> shouldn't have the same permissions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KUDU-1855) Kudu file UNIX permissions are inconsistent
[
https://issues.apache.org/jira/browse/KUDU-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848772#comment-15848772
]
Todd Lipcon commented on KUDU-1855:
---
Ah, OK. Let's consider this a blocker for 1.3, then, since we're adding
authorization.
> Kudu file UNIX permissions are inconsistent
> ---
>
> Key: KUDU-1855
> URL: https://issues.apache.org/jira/browse/KUDU-1855
> Project: Kudu
> Issue Type: Bug
> Components: security
>Affects Versions: 1.3.0
>Reporter: Adar Dembo
>
> Right now the access modes of Kudu files on disk are quite inconsistent. For
> example:
> {noformat}
> 6942960 drwxr-xr-x 4 kudu kudu 27 Jun 21 2016
> /data/1/kudu/tablet
> 21501102670 drwxr-xr-x 5 kudu kudu 71 Jan 17 17:42
> /data/1/kudu/tablet/data
> 21501241274 -rw--- 1 kudu kudu 665 Jun 21 2016
> /data/1/kudu/tablet/data/instance
> 702851 16 drwxr-xr-x 2 kudu kudu12288 Jan 31 14:02
> /data/1/kudu/tablet/data/tablet-meta
> 702894 12 -rw--- 1 kudu kudu 9501 Jan 22 13:38
> /data/1/kudu/tablet/data/tablet-meta/33cc61001d1442048a588f930d973464
> ...
> 21538687298 -rw--- 1 kudu kudu 8109 Jan 12 22:26
> /data/1/kudu/tablet/data/consensus-meta/83a2f75a88bd48f5a8ec28f8328af481
> 702854 152 drwxr-xr-x 2 kudu kudu 110592 Jan 23 16:59
> /data/1/kudu/tablet/data/data
> 15456410 -rw-r--r-- 1 kudu kudu 10743091200 Jan 9 13:06
> /data/1/kudu/tablet/data/data/bdf6d87c2ba34d598327b0b9e159a5ea.data
> 7056904 -rw--- 1 kudu kudu 990 Jun 21 2016
> /data/1/kudu/tablet/data/data/block_manager_instance
> 278147 652 -rw-r--r-- 1 kudu kudu 660084 Aug 15 00:09
> /data/1/kudu/tablet/data/data/3917e1185471419786f59ad72bb7ba63.metadata
> 2905870 -rw-r--r-- 1 kudu kudu 10746855424 Jan 9 13:05
> /data/1/kudu/tablet/data/data/3917e1185471419786f59ad72bb7ba63.data
> 291129 672 -rw-r--r-- 1 kudu kudu 681669 Aug 15 00:09
> /data/1/kudu/tablet/data/data/756bd793f0c24188853e096f835ba7b4.metadata
> {noformat}
> I suspect it's due to the difference of "files opened as temp files and
> renamed into place" and "files opened directly". The former have permissions
> that are even more restrictive than the umask (022), while the latter adhere
> to the umask.
> I don't know whether more restrictive or less restrictive modes should be
> used, but I don't see any reason why all files in Kudu's filesystem layout
> shouldn't have the same permissions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KUDU-1855) Kudu file UNIX permissions are inconsistent
[
https://issues.apache.org/jira/browse/KUDU-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847672#comment-15847672
]
Adar Dembo commented on KUDU-1855:
--
I don't think it's safe; this is from an internal cluster where I'm a user
outside the kudu group and I can read the .data and .metadata files myself.
> Kudu file UNIX permissions are inconsistent
> ---
>
> Key: KUDU-1855
> URL: https://issues.apache.org/jira/browse/KUDU-1855
> Project: Kudu
> Issue Type: Bug
> Components: security
>Affects Versions: 1.3.0
>Reporter: Adar Dembo
>
> Right now the access modes of Kudu files on disk are quite inconsistent. For
> example:
> {noformat}
> 6942960 drwxr-xr-x 4 kudu kudu 27 Jun 21 2016
> /data/1/kudu/tablet
> 21501102670 drwxr-xr-x 5 kudu kudu 71 Jan 17 17:42
> /data/1/kudu/tablet/data
> 21501241274 -rw--- 1 kudu kudu 665 Jun 21 2016
> /data/1/kudu/tablet/data/instance
> 702851 16 drwxr-xr-x 2 kudu kudu12288 Jan 31 14:02
> /data/1/kudu/tablet/data/tablet-meta
> 702894 12 -rw--- 1 kudu kudu 9501 Jan 22 13:38
> /data/1/kudu/tablet/data/tablet-meta/33cc61001d1442048a588f930d973464
> ...
> 21538687298 -rw--- 1 kudu kudu 8109 Jan 12 22:26
> /data/1/kudu/tablet/data/consensus-meta/83a2f75a88bd48f5a8ec28f8328af481
> 702854 152 drwxr-xr-x 2 kudu kudu 110592 Jan 23 16:59
> /data/1/kudu/tablet/data/data
> 15456410 -rw-r--r-- 1 kudu kudu 10743091200 Jan 9 13:06
> /data/1/kudu/tablet/data/data/bdf6d87c2ba34d598327b0b9e159a5ea.data
> 7056904 -rw--- 1 kudu kudu 990 Jun 21 2016
> /data/1/kudu/tablet/data/data/block_manager_instance
> 278147 652 -rw-r--r-- 1 kudu kudu 660084 Aug 15 00:09
> /data/1/kudu/tablet/data/data/3917e1185471419786f59ad72bb7ba63.metadata
> 2905870 -rw-r--r-- 1 kudu kudu 10746855424 Jan 9 13:05
> /data/1/kudu/tablet/data/data/3917e1185471419786f59ad72bb7ba63.data
> 291129 672 -rw-r--r-- 1 kudu kudu 681669 Aug 15 00:09
> /data/1/kudu/tablet/data/data/756bd793f0c24188853e096f835ba7b4.metadata
> {noformat}
> I suspect it's due to the difference of "files opened as temp files and
> renamed into place" and "files opened directly". The former have permissions
> that are even more restrictive than the umask (022), while the latter adhere
> to the umask.
> I don't know whether more restrictive or less restrictive modes should be
> used, but I don't see any reason why all files in Kudu's filesystem layout
> shouldn't have the same permissions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KUDU-1855) Kudu file UNIX permissions are inconsistent
[
https://issues.apache.org/jira/browse/KUDU-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847665#comment-15847665
]
Todd Lipcon commented on KUDU-1855:
---
Seems like we should probably be making our data 0600 even if the user's umask
is a bit too open... Surprising to see the data files a+r (even if probably the
root of the data dir is a-x and thus safe?)
> Kudu file UNIX permissions are inconsistent
> ---
>
> Key: KUDU-1855
> URL: https://issues.apache.org/jira/browse/KUDU-1855
> Project: Kudu
> Issue Type: Bug
> Components: security
>Affects Versions: 1.3.0
>Reporter: Adar Dembo
>
> Right now the access modes of Kudu files on disk are quite inconsistent. For
> example:
> {noformat}
> 6942960 drwxr-xr-x 4 kudu kudu 27 Jun 21 2016
> /data/1/kudu/tablet
> 21501102670 drwxr-xr-x 5 kudu kudu 71 Jan 17 17:42
> /data/1/kudu/tablet/data
> 21501241274 -rw--- 1 kudu kudu 665 Jun 21 2016
> /data/1/kudu/tablet/data/instance
> 702851 16 drwxr-xr-x 2 kudu kudu12288 Jan 31 14:02
> /data/1/kudu/tablet/data/tablet-meta
> 702894 12 -rw--- 1 kudu kudu 9501 Jan 22 13:38
> /data/1/kudu/tablet/data/tablet-meta/33cc61001d1442048a588f930d973464
> ...
> 21538687298 -rw--- 1 kudu kudu 8109 Jan 12 22:26
> /data/1/kudu/tablet/data/consensus-meta/83a2f75a88bd48f5a8ec28f8328af481
> 702854 152 drwxr-xr-x 2 kudu kudu 110592 Jan 23 16:59
> /data/1/kudu/tablet/data/data
> 15456410 -rw-r--r-- 1 kudu kudu 10743091200 Jan 9 13:06
> /data/1/kudu/tablet/data/data/bdf6d87c2ba34d598327b0b9e159a5ea.data
> 7056904 -rw--- 1 kudu kudu 990 Jun 21 2016
> /data/1/kudu/tablet/data/data/block_manager_instance
> 278147 652 -rw-r--r-- 1 kudu kudu 660084 Aug 15 00:09
> /data/1/kudu/tablet/data/data/3917e1185471419786f59ad72bb7ba63.metadata
> 2905870 -rw-r--r-- 1 kudu kudu 10746855424 Jan 9 13:05
> /data/1/kudu/tablet/data/data/3917e1185471419786f59ad72bb7ba63.data
> 291129 672 -rw-r--r-- 1 kudu kudu 681669 Aug 15 00:09
> /data/1/kudu/tablet/data/data/756bd793f0c24188853e096f835ba7b4.metadata
> {noformat}
> I suspect it's due to the difference of "files opened as temp files and
> renamed into place" and "files opened directly". The former have permissions
> that are even more restrictive than the umask (022), while the latter adhere
> to the umask.
> I don't know whether more restrictive or less restrictive modes should be
> used, but I don't see any reason why all files in Kudu's filesystem layout
> shouldn't have the same permissions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
