> unless, of course, you were to detach the volume from host/vm-1 and attached it to another system which had a different user table. This is exactly the problem. For snapshot repositories, your repository directory has to be accessible by all your data nodes. Thus, I mount the same folder, in the same location, on all my data nodes. Thus, my uid/gid issues. Which would also apply on iSCSI.
Anyway, I modified my Ansible elasticsearch role to support creating a custom user and group. Tested it on my tiny test cluster, and it seems to work ok. Now I just need to run it on my prod cluster and deal with a full cluster restart. *sigh* --David Reagan On Wed, Mar 18, 2015 at 5:21 AM, Norberto Meijome <[email protected]> wrote: > yes, that's the difference between a (network service which exposes ) > block storage like iSCSI and a network file system like NFS ( or glusterFS > or Lustre... ). > > I don't see why on a local device (iSCSI) you'd have any issue with > numeric uid not matching the 'name' of the user - unless, of course, you > were to detach the volume from host/vm-1 and attached it to another system > which had a different user table. > > centralised authentication systems like ldap would solve this for you too > :) (though it seems overkill when all you'd have to do is plan your uid > space properly across client + nfs servers ...). configuration systems like > puppet / chef / ansible / salt should also help you ensure uids across > multiple system... > > Anyway, Magnus' suggestion is on the money for the '1 server problem' > > On Wed, Mar 18, 2015 at 3:00 AM, David Reagan <[email protected]> wrote: > >> @Mark Walkom, So, I'm looking into iscsi. From what I have learned so >> far, you actually format the LUN with whatever file system you want. So, >> wouldn't the gid/uid issue show up there as well, if I formatted to ext3 or >> ext4? Since Ubuntu would treat it like a normal partition and use typical >> linux file perms on it. >> >> --David Reagan >> >> On Mon, Mar 16, 2015 at 5:37 PM, David Reagan <[email protected]> wrote: >> >>> If I were manually creating the elasticsearch user, that'd be easy. But >>> I'm relying on apt to do the job for me. So, yeah... >>> >>> Hmm... I suppose I could manually create an elasticsearch2 user, then >>> modify the defaults files to use it when running ES. Still seems clunky... >>> >>> --David Reagan >>> >>> On Mon, Mar 16, 2015 at 5:20 PM, Andrew Selden <[email protected]> >>> wrote: >>> >>>> I’m not that familiar with iSCSI so I hesitate to say for sure, but >>>> anytime you are cross-mounting filesystems on Linux you have to take >>>> uid/gid consistency into account. >>>> >>>> - Andrew >>>> >>>> On Mar 16, 2015, at 4:46 PM, David Reagan <[email protected]> wrote: >>>> >>>> Would an iSCSI mount have the same issue? I believe our SAN supports >>>> both. >>>> >>>> --David Reagan >>>> >>>> On Mon, Mar 16, 2015 at 4:40 PM, Andrew Selden <[email protected]> >>>> wrote: >>>> >>>>> Hi David, >>>>> >>>>> This is a common problem with NFS. Unfortunately the protocol assumes >>>>> identical uid/gid mappings across all machines. It’s just one of those >>>>> annoying sys-admin tasks that one has to take into account when using NFS. >>>>> To get your permissions back to less permissive settings you will have to >>>>> edit the /etc/passwd and /etc/group files to keep them in sync. >>>>> >>>>> See http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM4 >>>>> for more context. >>>>> >>>>> - Andrew >>>>> >>>>> >>>>> On Mar 16, 2015, at 4:04 PM, David Reagan <[email protected]> wrote: >>>>> >>>>> First, it is a file permissions issue. I did get snapshots to run when >>>>> I chmoded to 777. As you can see from the ls output, /mounts/prod_backup >>>>> is >>>>> 777. Prior to that it was 775 or 755. So, I could revise my question to >>>>> "How can I get snapshots working without using insecure file permissions?" >>>>> >>>>> root@log-elasticsearch-01:~# mount >>>>> /dev/mapper/ws--template--01-root on / type ext4 (rw,errors=remount-ro) >>>>> proc on /proc type proc (rw,noexec,nosuid,nodev) >>>>> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) >>>>> none on /sys/fs/fuse/connections type fusectl (rw) >>>>> none on /sys/kernel/debug type debugfs (rw) >>>>> none on /sys/kernel/security type securityfs (rw) >>>>> udev on /dev type devtmpfs (rw,mode=0755) >>>>> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) >>>>> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) >>>>> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) >>>>> none on /run/shm type tmpfs (rw,nosuid,nodev) >>>>> /dev/sda1 on /boot type ext2 (rw) >>>>> rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw) >>>>> nfsip:/vol/Logs/prod_backup on /mounts/prod_backup type nfs >>>>> (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) >>>>> nfsip:/vol/Logs/log-elasticsearch-01 on /mounts/log-elasticsearch-01 >>>>> type nfs (rw,nfsvers=3,hard,intr,tcp,actimeo=3,addr=nfsip) >>>>> >>>>> root@log-elasticsearch-01:~# ls -ld /mounts >>>>> drwxr-xr-x 6 root root 4096 Oct 1 13:43 /mounts >>>>> >>>>> root@log-elasticsearch-01:~# ls -ld /mounts/prod_backup/ >>>>> drwxrwxrwx 4 elasticsearch elasticsearch 4096 Mar 16 13:41 >>>>> /mounts/prod_backup/ >>>>> >>>>> --David Reagan >>>>> >>>>> On Mon, Mar 16, 2015 at 3:47 PM, Mark Walkom <[email protected]> >>>>> wrote: >>>>> >>>>>> Can you post the output from *mount* and *ls -ld /mounts >>>>>> /mounts/prod_backup*? >>>>>> >>>>>> On 16 March 2015 at 13:33, David Reagan <[email protected]> wrote: >>>>>> >>>>>>> Why does this happen? >>>>>>> >>>>>>> >>>>>>> curl -XPUT 'http://localhost:9200/_snapshot/my_backup?pretty=true' >>>>>>>> -d '{ >>>>>>>> > "type": "fs", >>>>>>>> > "settings": { >>>>>>>> > "location": "/mounts/prod_backup/my_backup", >>>>>>>> > "compress": true >>>>>>>> > } >>>>>>>> > }' >>>>>>>> { >>>>>>>> "error" : >>>>>>>> "RemoteTransportException[[log-elasticsearch-02][inet[/10.x.x.83:9300]][cluster:admin/repository/put]]; >>>>>>>> nested: RepositoryVerificationException[[my_backup] >>>>>>>> [vxUQwUTCQwOaLyCy0eMK8A, >>>>>>>> 'RemoteTransportException[[log-elasticsearch-04][inet[/10.x.x.80:9300]][internal:admin/repository/verify]]; >>>>>>>> nested: RepositoryVerificationException[[my_backup] store location >>>>>>>> [/mounts/prod_backup/my_backup] is not accessible on the node >>>>>>>> [[log-elasticsearch-04][vxUQwUTCQwOaLyCy0eMK8A][log-elasticsearch-04][inet[/10.x.x.80:9300]]]]; >>>>>>>> nested: >>>>>>>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-vxUQwUTCQwOaLyCy0eMK8A >>>>>>>> (Permission denied)]; '], [GMTt6Y-3Qle1Fm3SGl-LTQ, >>>>>>>> 'RemoteTransportException[[log-estools-01][inet[/10.x.x.8:9300]][internal:admin/repository/verify]]; >>>>>>>> nested: RepositoryVerificationException[[my_backup] store location >>>>>>>> [/mounts/prod_backup/my_backup] is not accessible on the node >>>>>>>> [[log-estools-01][GMTt6Y-3Qle1Fm3SGl-LTQ][log-estools-01][inet[/10.x.x.8:9300]]{data=false}]]; >>>>>>>> nested: >>>>>>>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-GMTt6Y-3Qle1Fm3SGl-LTQ >>>>>>>> (Permission denied)]; '], [ffpuQF_zRZGGPRkZRgq1mw, >>>>>>>> 'RemoteTransportException[[log-elasticsearch-03][inet[/10.x.x.92:9300]][internal:admin/repository/verify]]; >>>>>>>> nested: RepositoryVerificationException[[my_backup] store location >>>>>>>> [/mounts/prod_backup/my_backup] is not accessible on the node >>>>>>>> [[log-elasticsearch-03][ffpuQF_zRZGGPRkZRgq1mw][log-elasticsearch-03][inet[/10.x.x.92:9300]]]]; >>>>>>>> nested: >>>>>>>> FileNotFoundException[/mounts/prod_backup/my_backup/tests-yZ57gviiQUGS55tr_ULhhg-ffpuQF_zRZGGPRkZRgq1mw >>>>>>>> (Permission denied)]; ']]]; ", >>>>>>>> "status" : 500 >>>>>>>> } >>>>>>> >>>>>>> >>>>>>> >>>>>>> /mounts/prod_backup is an NFS share mounted on all my ES nodes. >>>>>>> >>>>>>> The only thing I can think of is that file permissions are wrong. >>>>>>> But I'm not sure how to fix that. >>>>>>> >>>>>>> On the two nodes not listed in the error, the elasticsearch gid and >>>>>>> uid are different that on the other three nodes. So I can't just *chown >>>>>>> -R elasticsearch:elasticsearch*. >>>>>>> >>>>>>> Do I have to make sure that the uid and gid are the same on all >>>>>>> nodes? How am I supposed to do that when I install from the apt >>>>>>> repositories? >>>>>>> >>>>>>> Also, log-estools-01 has *data.node = false*. Why does it still >>>>>>> need the prod_backup mount? >>>>>>> >>>>>>> log-estools-01 is on Ubuntu 14.04. >>>>>>> >>>>>>> Everything else is Ubuntu 12.04. >>>>>>> >>>>>>> Running ES 1.4.4. >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "elasticsearch" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/elasticsearch/8bf960fd-c629-4e1a-92c1-9cc92f43487b%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "elasticsearch" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe >>>>>> . >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9YtDD2H_0uuaoXJ%3DGCSAoDYeBuQpc-wQfe8Yd1m%3D7Q_w%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AeD-Fpkg0Z8UNoQVSguQo7rPHBSKca1iAOt7_xjV4eAcg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "elasticsearch" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co >>>>> <https://groups.google.com/d/msgid/elasticsearch/BF10597A-F9C6-4A2B-B26C-334D9E33BC49%40elastic.co?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AdniWYRrWW13WhnMOx8Ps3KFZQ%2BxrHfpL8zeN%3DuVM-iVw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "elasticsearch" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co >>>> <https://groups.google.com/d/msgid/elasticsearch/F361E03C-41F5-491F-B8E6-7C9DBA199047%40elastic.co?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CANo%2B_Ac-YQAcZ9f8e5afrymYmbNs%3DQZZNhYCFEd4Hu6atqZcNw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Norberto 'Beto' Meijome > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/_1ClJTJOEcU/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CACj2-4KkXbBG9S36ViJwkOzGg%3DyLjLAgT0uPEnfRcy2%3DQvWVJA%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CACj2-4KkXbBG9S36ViJwkOzGg%3DyLjLAgT0uPEnfRcy2%3DQvWVJA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANo%2B_AcPMQ2W5tffd3cMCfuvjH%2BAfuk9U9GiPeLMAGvN2G_gGQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
