yes, that's the difference between a (network service which exposes ) block storage like iSCSI and a network file system like NFS ( or glusterFS or Lustre... ).
I don't see why on a local device (iSCSI) you'd have any issue with numeric uid not matching the 'name' of the user - unless, of course, you were to detach the volume from host/vm-1 and attached it to another system which had a different user table. centralised authentication systems like ldap would solve this for you too :) (though it seems overkill when all you'd have to do is plan your uid space properly across client + nfs servers ...). configuration systems like puppet / chef / ansible / salt should also help you ensure uids across multiple system... Anyway, Magnus' suggestion is on the money for the '1 server problem' On Wed, Mar 18, 2015 at 3:00 AM, David Reagan <[email protected]> wrote: > @Mark Walkom, So, I'm looking into iscsi. From what I have learned so far, > you actually format the LUN with whatever file system you want. So, > wouldn't the gid/uid issue show up there as well, if I formatted to ext3 or > ext4? Since Ubuntu would treat it like a normal partition and use typical > linux file perms on it. > > --David Reagan > > On Mon, Mar 16, 2015 at 5:37 PM, David Reagan <[email protected]> wrote: > >> If I were manually creating the elasticsearch user, that'd be easy. But >> I'm relying on apt to do the job for me. So, yeah... >> >> Hmm... I suppose I could manually create an elasticsearch2 user, then >> modify the defaults files to use it when running ES. Still seems clunky... >> >> --David Reagan >> >> On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden <[email protected]> wrote: >> >>> I’m not that familiar with iSCSI so I hesitate to say for sure, but >>> anytime you are cross-mounting filesystems on Linux you have to take >>> uid/gid consistency into account. >>> >>> - Andrew >>> >>> On Mar 16, 2015, at 4:46 PM, David Reagan <[email protected]> wrote: >>> >>> Would an iSCSI mount have the same issue? I believe our SAN supports >>> both. >>> >>> --David Reagan >>> >>> On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden <[email protected]> >>> wrote: >>> >>>> Hi David, >>>> >>>> This is a common problem with NFS. Unfortunately the protocol assumes >>>> identical uid/gid mappings across all machines. It’s just one of those >>>> annoying sys-admin tasks that one has to take into account when using NFS. >>>> To get your permissions back to less permissive settings you will have to >>>> edit the /etc/passwd and /etc/group files to keep them in sync. >>>> >>>> See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4 >>>> for more context. >>>> >>>> - Andrew >>>> >>>> >>>> On Mar 16, 2015, at 4:04 PM, David Reagan <[email protected]> wrote: >>>> >>>> First, it is a file permissions issue. I did get snapshots to run when >>>> I chmoded to 777. As you can see from the ls output, /mounts/prod_backup is >>>> 777. Prior to that it was 775 or 755. So, I could revise my question to >>>> "How can I get snapshots working without using insecure file permissions?" >>>> >>>> root@log-elasticsearch-01:~# mount >>>> /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro) >>>> proc on /proc type proc (rw,noexec,nosuid,nodev) >>>> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) >>>> none on /sys/fs/fuse/connections type fusectl (rw) >>>> none on /sys/kernel/debug type debugfs (rw) >>>> none on /sys/kernel/security type securityfs (rw) >>>> udev on /dev type devtmpfs (rw,mode=0755) >>>> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) >>>> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) >>>> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) >>>> none on /run/shm type tmpfs (rw,nosuid,nodev) >>>> /dev/sda1 on /boot type ext2 (rw) >>>> rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw) >>>> nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs >>>> (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) >>>> nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 >>>> type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) >>>> >>>> root@log-elasticsearch-01:~# ls -ld /mounts >>>> drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts >>>> >>>> root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/ >>>> drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 >>>> /mounts/prod_backup/ >>>> >>>> --David Reagan >>>> >>>> On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom <[email protected]> >>>> wrote: >>>> >>>>> Can you post the output from *mount* and *ls -ld /mounts >>>>> /mounts/prod_backup*? >>>>> >>>>> On 16 March 2015 at 13:33, David Reagan <[email protected]> wrote: >>>>> >>>>>> Why does this happen? >>>>>> >>>>>> >>>>>> curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' >>>>>>> -d '{ >>>>>>> > "type": "fs", >>>>>>> > "settings": { >>>>>>> > "location": "/mounts/prod_backup/my_backup", >>>>>>> > "compress": true >>>>>>> > } >>>>>>> > }' >>>>>>> { >>>>>>> "error" : >>>>>>> "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; >>>>>>> nested: RepositoryVerificationException[[my_backup] >>>>>>> [vxUQwUTCQwOaLyCy0eMK8A, >>>>>>> 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; >>>>>>> nested: RepositoryVerificationException[[my_backup] store location >>>>>>> [/mounts/prod_backup/my_backup] is not accessible on the node >>>>>>> [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; >>>>>>> nested: >>>>>>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A >>>>>>> (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, >>>>>>> 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; >>>>>>> nested: RepositoryVerificationException[[my_backup] store location >>>>>>> [/mounts/prod_backup/my_backup] is not accessible on the node >>>>>>> [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; >>>>>>> nested: >>>>>>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ >>>>>>> (Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw, >>>>>>> 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; >>>>>>> nested: RepositoryVerificationException[[my_backup] store location >>>>>>> [/mounts/prod_backup/my_backup] is not accessible on the node >>>>>>> [[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; >>>>>>> nested: >>>>>>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw >>>>>>> (Permission denied)]; ']]]; ", >>>>>>> "status" : 500 >>>>>>> } >>>>>> >>>>>> >>>>>> >>>>>> /mounts/prod_backup is an NFS share mounted on all my ES nodes. >>>>>> >>>>>> The only thing I can think of is that file permissions are wrong. But >>>>>> I'm not sure how to fix that. >>>>>> >>>>>> On the two nodes not listed in the error, the elasticsearch gid and >>>>>> uid are different that on the other three nodes. So I can't just *chown >>>>>> -R elasticsearch:elasticsearch*. >>>>>> >>>>>> Do I have to make sure that the uid and gid are the same on all >>>>>> nodes? How am I supposed to do that when I install from the apt >>>>>> repositories? >>>>>> >>>>>> Also, log-estools-01 has *data.node = false*. Why does it still need >>>>>> the prod_backup mount? >>>>>> >>>>>> log-estools-01 is on Ubuntu 14.04. >>>>>> >>>>>> Everything else is Ubuntu 12.04. >>>>>> >>>>>> Running ES 1.4.4. >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "elasticsearch" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "elasticsearch" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co >>>> <https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co >>> <https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Norberto 'Beto' Meijome -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACj2-4KkXbBG9S36ViJwkOzGg%3DyLjLAgT0uPEnfRcy2%3DQvWVJA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
