Re: [OmniOS-discuss] Big Data
You might want to read this thread (especially the comments). http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-January/015599.html On Sat, Jun 16, 2018 at 10:46 AM Michael Talbott wrote: > We've been using OmniOS happily for years now for our storage server > needs. But we're rapidly increasing our data footprint and growing so much > (multiple PBs per year) that ideally I'd like to move to a cluster based > object store based system ontop of OmniOS. I successfully use BeeGFS inside > lxzones in OmniOS which seems to work nicely for our HPC scratch volume, > but, it doesn't sound like it scales to hundreds of millions of files very > well. > > I am hoping that someone has some ideas for me. Ideally I'd like something > that's cluster capable and has erasure coding like Ceph and have cluster > aware snapshots (not local zfs snaps) and an s3 compatibility/access layer. > > Any thoughts on the topic are greatly appreciated. > > Thanks, > > Michael > Sent from my iPhone > ___ > OmniOS-discuss mailing list > OmniOS-discuss@lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] questions
What about the attached vnics? Can you do: dladm show-linkprop vnic# for the vnics connected to the etherstub? There may be a maxbw setting ... On Thu, Sep 14, 2017 at 9:41 AM, Dirk Willems <dirk.will...@exitas.be> wrote: > just execute => dladm create-etherstub Backend_Switch0 > > > and having => Backend_Switch0 etherstub 9000 up > > On 14-09-17 18:26, Ian Kaufman wrote: > > Networking has always used *bps - that's been the standard for many years. > Megabits, Gigabits ... > > Disk tools have always measured in bytes since that is how the capacity is > defined. > > How did you create your etherstub? I know you can set a maxbw (maximum > bandiwdth), but I don't know what the default behavior is. > > Ian > > > > On Thu, Sep 14, 2017 at 9:13 AM, Dirk Willems <dirk.will...@exitas.be> > wrote: > >> Thank you all, the water is already clearing up :) >> >> >> So infiniband is 40 Gbps an not 40GB/s, very confusing GB/s Gbps why they >> not take a standaard and set everything in GB/s or MB/s ? >> >> A lot of people make a lot of mistakes between them, me too ... >> >> If it is 40 Gbps a factor of 8 then we theoretical have max 5 GB/s >> throughput. >> >> Little difference 40 or 5 :) >> >> So Ian you have the full blow with 36Gbps very cool looks more like it :) >> >> Did I play with the frame size, not really sure what you mean by that >> sorry but I think its default on 9000 >> >> Backend_Switch0 etherstub 9000 up >> >> >> Do understand that if we use UDP streams from process to process it will >> be much quicker over the etherstub gonna need more test to do. >> >> We used for a customer Mbuffer with zfs send over Lan that is also very >> quick sometimes I also use it at my home very good prog. >> >> But still do not understand how it is that I copy from 1 NGZ with >> 100MB/s, I receive on the other NGZ 250MB/s very strange ? >> >> >> the command dlstat difference between OmniOSce and Solaris ? >> >> RBYTES => receiving >> >> OBYTES => sending >> >> root@test2:~# dlstat -i 2 >> > >> > LINKIPKTS RBYTESOPKTS OBYTES >> >net1 25.76K 185.14M 10.08K2.62M >> >net1 27.04K 187.16M 11.23K3.22M >> >> >> >> BYTES => receiving and sending ? >> >> But then still if the copy is not running I have 0 so doesn't explain why >> I see 216 MB where come the rest of the 116 MB from is it compression ? >> >> root@NGINX:/root# dlstat show-link NGINX1 -i 2 >> > >> > LINK TYPE ID INDEX PKTSBYTES >> > NGINX1rx bcast --00 >> > NGINX1rx sw --00 >> > NGINX1tx bcast --00 >> > NGINX1tx sw --9.26K 692.00K >> > NGINX1rx local -- 26.00K 216.32M >> >> >> Thank you all for your feedback much appreciations ! >> >> >> Kind Regards, >> >> >> Dirk >> >> >> >> On 14-09-17 17:07, Ian Kaufman wrote: >> >> Some other things you need to take into account: >> >> QDR Infiniband is 40Gbps, not 40GB/s. That is a factor of 8 difference. >> That is also a theoretical maximum throughput, there is some overhead. In >> reality, you will never see 40Gbps. >> >> My system tested out at 6Gbps - 8Gbps using NFS over IPoIB, with DDR >> (20Gbps) nodes and a QDR (40Gbps) storage server. IPoIB drops the >> theoretical max rates to 18Gbps and 36Gbps respectively. >> >> If you are getting 185MB/s, you are seeing 1.48Gbps. >> >> Keep your B's and b's straight. Did you play with your frame size at all? >> >> Ian >> >> On Thu, Sep 14, 2017 at 7:10 AM, Jim Klimov <jimkli...@cos.ru> wrote: >> >>> On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems < >>> dirk.will...@exitas.be> wrote: >>> >Hello, >>> > >>> > >>> >I'm trying to understand something let me explain. >>> > >>> > >>> >Oracle always told to me that if you create a etherstub switch it has >>> >infiniband speed 40GB/s. >>> > >>> >But I have a customer running on Solaris (Yeah I know but let me >>> >explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan >>> >(I know told him to to use etherstub). >>> > >>> >The copy witch
Re: [OmniOS-discuss] questions
Networking has always used *bps - that's been the standard for many years. Megabits, Gigabits ... Disk tools have always measured in bytes since that is how the capacity is defined. How did you create your etherstub? I know you can set a maxbw (maximum bandiwdth), but I don't know what the default behavior is. Ian On Thu, Sep 14, 2017 at 9:13 AM, Dirk Willems <dirk.will...@exitas.be> wrote: > Thank you all, the water is already clearing up :) > > > So infiniband is 40 Gbps an not 40GB/s, very confusing GB/s Gbps why they > not take a standaard and set everything in GB/s or MB/s ? > > A lot of people make a lot of mistakes between them, me too ... > > If it is 40 Gbps a factor of 8 then we theoretical have max 5 GB/s > throughput. > > Little difference 40 or 5 :) > > So Ian you have the full blow with 36Gbps very cool looks more like it :) > > Did I play with the frame size, not really sure what you mean by that > sorry but I think its default on 9000 > > Backend_Switch0 etherstub 9000 up > > > Do understand that if we use UDP streams from process to process it will > be much quicker over the etherstub gonna need more test to do. > > We used for a customer Mbuffer with zfs send over Lan that is also very > quick sometimes I also use it at my home very good prog. > > But still do not understand how it is that I copy from 1 NGZ with 100MB/s, > I receive on the other NGZ 250MB/s very strange ? > > > the command dlstat difference between OmniOSce and Solaris ? > > RBYTES => receiving > > OBYTES => sending > > root@test2:~# dlstat -i 2 > > > > LINKIPKTS RBYTESOPKTS OBYTES > >net1 25.76K 185.14M 10.08K2.62M > >net1 27.04K 187.16M 11.23K3.22M > > > > BYTES => receiving and sending ? > > But then still if the copy is not running I have 0 so doesn't explain why > I see 216 MB where come the rest of the 116 MB from is it compression ? > > root@NGINX:/root# dlstat show-link NGINX1 -i 2 > > > > LINK TYPE ID INDEX PKTSBYTES > > NGINX1rx bcast --00 > > NGINX1rx sw --00 > > NGINX1tx bcast --00 > > NGINX1 tx sw --9.26K 692.00K > > NGINX1rx local -- 26.00K 216.32M > > > Thank you all for your feedback much appreciations ! > > > Kind Regards, > > > Dirk > > > > On 14-09-17 17:07, Ian Kaufman wrote: > > Some other things you need to take into account: > > QDR Infiniband is 40Gbps, not 40GB/s. That is a factor of 8 difference. > That is also a theoretical maximum throughput, there is some overhead. In > reality, you will never see 40Gbps. > > My system tested out at 6Gbps - 8Gbps using NFS over IPoIB, with DDR > (20Gbps) nodes and a QDR (40Gbps) storage server. IPoIB drops the > theoretical max rates to 18Gbps and 36Gbps respectively. > > If you are getting 185MB/s, you are seeing 1.48Gbps. > > Keep your B's and b's straight. Did you play with your frame size at all? > > Ian > > On Thu, Sep 14, 2017 at 7:10 AM, Jim Klimov <jimkli...@cos.ru> wrote: > >> On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems < >> dirk.will...@exitas.be> wrote: >> >Hello, >> > >> > >> >I'm trying to understand something let me explain. >> > >> > >> >Oracle always told to me that if you create a etherstub switch it has >> >infiniband speed 40GB/s. >> > >> >But I have a customer running on Solaris (Yeah I know but let me >> >explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan >> >(I know told him to to use etherstub). >> > >> >The copy witch is performed for a Oracle database with sql command, the >> > >> >DBA witch have 5 streams say it's waiting on the disk, the disk are 50 >> >- >> >60 % busy the speed is 30 mb/s. >> > >> > >> >So I did some test just to see and understand if it's the database or >> >the system, but with doing my tests I get very confused ??? >> > >> > >> >On another Solaris at my work copy over etherstub switch => copy speed >> >is 185MB/s expected much more of infiniband speed ??? >> > >> > >> >root@test1:/export/home/Admin# scp test10G >> >Admin@192.168.1.2:/export/home/Admin/ >> >Password: >> >test10G 100% >> >|| >> >10240 >> >MB00:59 >>
Re: [OmniOS-discuss] questions
NGINX1rx local -- 30.65K 253.73M > > NGINX1rx bcast --00 > > NGINX1rx sw --00 > > NGINX1tx bcast --00 > > NGINX1tx sw --8.95K 669.32K > > NGINX1rx local -- 29.10K 241.15M > > > > > >On the other NGZ I receive 250MB/s > > > > > >- So my question is how comes that the speed is equal to Lan 100MB/s on > > > >OmniOSce but i receive 250MB/s ? > > > >- Why is etherstub so slow if infiniband speed is 40GB/s ??? > > > > > >I'm very confused right now ... > > > > > >And want to know for sure how to understand and see this in the right > >way, because this customer will be the first customer from my who gonna > > > >switch complety over to OmniOSce on production and because this > >customer > >is one or the biggest company's in Belgium I really don't want to mess > >up !!! > > > > > >So any help and clarification will be highly appreciate !!! > > > > > >Thank you very much. > > > > > >Kind Regards, > > > > > >Dirk > > I am not sure where the infiniband claim comes from, but copying data disk > to disk, you involve the slow layers like disk, skewed by faster layers > like cache of already-read data and delayed writes :) > > If you have a wide pipe that you may fill, it doesn't mean you do have the > means to fill it with a few disks. > > To estimate the speeds, try pure UDP streams from process to process (no > disk), large-packet floodping, etc. > > I believe etherstub is not constrained artificially, and defaults to jumbo > frames. Going to LAN and back can in fact use external hardware (IIRC there > may be a system option to disable that, not sure) and so is constrained by > that. > > Jim > -- > Typos courtesy of K-9 Mail on my Android > ___ > OmniOS-discuss mailing list > OmniOS-discuss@lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] NFS mounts seems as nobody on Centos6
Are you using NFSv4? Are all machines using the same idmap domain? Ian On Thu, Jun 15, 2017 at 11:18 AM, Andries Annema <an3s.ann...@gmail.com> wrote: > Hi Özkan, > > The first thing that comes to mind is "no_root_squash" and, based on your > example setting with options like "anon=root", I assume this option can be > set as well on OmniOS with the "sharenfs" setting. > > Maybe these will help: > https://serverfault.com/questions/364613/centos-6- > ldap-nfs-file-ownership-is-stuck-on-nobody > http://www.linuxtopia.org/online_books/rhel6/rhel_6_ > storage_admin/rhel_6_storage_s2-nfs-security-files.html > > > Disclaimer: I am not an expert on NFS! Far from it. The above suggestion > is based on some personal experience where I needed that "no_root_squash" > option (although that was with some Linux distro's), and some Google-fu. > With that said, I am not sure if it adds a security threat or something. > > Anyway, my two cents. > > Regards, > Andries > > > On 2017-06-14 17:44, Özkan Göksu wrote: > > Hello. > > I have a nfs share and zfs settings like= "sharenfs anon=root,rw=* " > When i mount it from Centos, owner changes as nobody. (yes im root on > Centos) > But in omnios or ubuntu i see as root when i mount it. > > What is the cause of this problem? > > > > > ___ > OmniOS-discuss mailing > listOmniOS-discuss@lists.omniti.comhttp://lists.omniti.com/mailman/listinfo/omnios-discuss > > > > ___ > OmniOS-discuss mailing list > OmniOS-discuss@lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] OmniOS r151022 is now out!
Hmmm, having trouble going from 14 to 22. I cannot migrate to OpenSSH -
root@hiperq-data:/root# /usr/bin/pkg install --no-backup-be --reject
pkg:/network/ssh --reject pkg:/network/ssh/ssh-key --reject
pkg:/service/network/ssh --reject pkg:/service/network/ssh-common
pkg:/network/openssh pkg:/network/openssh-server
Creating Plan (Solver setup): \
pkg install: No matching version of network/openssh can be installed:
Reject: pkg://omnios/network/openssh@7.4.1,5.11-0.151022:20170511T001844Z
Reason: All versions matching 'require' dependency pkg:/library/security/
openssl@1.0.2.11,5.11-0.151022 are rejected
Reject: pkg://omnios/library/security/openssl@1.0.2.11
,5.11-0.151022:20170510T232316Z
Reason: This version is excluded by installed incorporation
pkg://omnios/incorporation/jeos/omnios-userland@11
,5.11-0.151014:20161221T194806Z
No matching version of network/openssh-server can be installed:
Reject: pkg://omnios/network/openssh-server@7.4.1
,5.11-0.151022:20170511T001853Z
Reason: All versions matching 'require' dependency
pkg:/SUNWcs@0.5.11,5.11-0.151022
are rejected
Reject: pkg://omnios/SUNWcs@0.5.11,5.11-0.151022:20170510T212740Z
Reason: This version is excluded by installed incorporation
pkg://omnios/consolidation/osnet/osnet-incorporation@0.5.11
,5.11-0.151014:20170301T162824Z
This version is excluded by installed incorporation
pkg://omnios/incorporation/jeos/illumos-gate@11
,5.11-0.151014:20160804T060038Z
On Fri, May 12, 2017 at 7:50 AM, Dan McDonald <dan...@omniti.com> wrote:
> As always, please start with the release notes:
>
> https://omnios.omniti.com/wiki.php/ReleaseNotes/r151022
>
> And for this release, there are upgrade notes:
>
> https://omnios.omniti.com/wiki.php/Upgrade_to_r151022
>
> This is an LTS and Stable update. Highlights include:
>
> - BSD Loader is now available. This has its own wiki page, including how
> to stick with GRUB if you think you need to.
>
> - Kayak Interactive Installer --> No more Caiman, no more F-keys & curses,
> and no-more hacking timezone names. Also, one less repo to maintain for
> OmniOS. Kayak does it all now.
>
> - Python is now 2.7. This means an IPS/pkg(5) update too. There's a
> subtle, but noteworthy, update to linked-image semantics ("pkg update -r")
> AND upgrading non-linked-image "ipkg" zones is a bit more difficult due to
> the transition (chicken & egg problem with ipkg brand "attach").
>
> - Perl is now 5.24.1.
>
> - USB 3.0 support.
>
> With OmniTI withdrawing funding for OmniOS-as-product, this will be the
> last OmniTI-pushed "release" of OmniOS. It contains all of the most recent
> (as of May) illumos-gate changes, AND illumos-joyent LX brand changes.
>
> This release broke the 6-month cadence, due to Loader, Python2.7, and the
> need for Kayak to take over ALL installation duties. I'm sorry for it
> being 1-2 months late, but I wanted to make sure these changes were
> comfortably in place during the 021 bloody.
>
> Because of the sheer volume of change in this release, I had to update
> many OmniOS wiki pages. Here's a list of all of those I changed
> nontrivially due to r151022. Please check them out to make sure I'm not
> missing anything...
>
> https://omnios.omniti.com/wiki.php/ReleaseNotes/r151022
>
> https://omnios.omniti.com/wiki.php/Installation
>
> https://omnios.omniti.com/wiki.php/LXZones
>
> https://omnios.omniti.com/wiki.php/Upgrade_to_r151022
>
> https://omnios.omniti.com/wiki.php/NewLinkedImages
>
> https://omnios.omniti.com/wiki.php/linked_images
>
> https://omnios.omniti.com/wiki.php/KayakInteractive
>
> https://omnios.omniti.com/wiki.php/BSDLoader
>
> https://omnios.omniti.com/wiki.php/ReleaseCycle
>
> https://omnios.omniti.com/wiki.php/Packaging
>
> https://omnios.omniti.com/wiki.php/ReleaseNotes/
>
> https://omnios.omniti.com/wiki.php/WikiStart
>
> https://omnios.omniti.com/wiki.php/BuildInstructions
>
> https://omnios.omniti.com/wiki.php/Maintainers
>
> https://omnios.omniti.com/wiki.php/illumos-tools
>
> https://omnios.omniti.com/wiki.php/buildctl
>
> https://omnios.omniti.com/wiki.php/ReleaseMedia
>
> I'll have more to say about my departure from OmniTI in a distinct e-mail.
>
> Thanks, and happy upgrading!
> Dan
>
> ___
> OmniOS-discuss mailing list
> OmniOS-discuss@lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
--
Ian Kaufman
Research Systems Administrator
UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Network >10Gb/s
Hmm, as far as I know, Omni-Path is not an ethernet replacement, but an IB replacement. While Intel is investing heavily in Omni-Path, it should not impact their ethernet offerings as they really are two different markets. Ian On Tue, Jan 10, 2017 at 8:10 AM, Schweiss, Chip <c...@innovates.com> wrote: > > On Tue, Jan 10, 2017 at 9:58 AM, Dan McDonald <dan...@omniti.com> wrote: > >> >> > On Jan 10, 2017, at 8:41 AM, Schweiss, Chip <c...@innovates.com> wrote: >> > >> > It appears that my options for 40Gb/s Ethernet are Intel, Chelsio and >> SolarFlare. >> > >> > Can anyone comment on which of these is the most stable solution when >> running under OmniOS? What's the fastest NFS throughput you've been able >> to achieve? >> >> The Intel i40e driver is nascent, but it will receive more attention as >> time passes. Doug's point about SolarFlare is a good one. >> >> > I'm a bit concerned on the Intel because of posts like this: > https://news.ycombinator.com/item?id=11373848 and the fact that they > seem to have shifted their focus to Omni-Path which from my understanding > is incompatible with the existing 40G gear. > > SolarFlare seems promising, but I'd like to know of at least on success > story. > > -Chip > > > >> You may wish to ping the larger illumos community about this as well. >> > > >> Dan >> >> > > ___ > OmniOS-discuss mailing list > OmniOS-discuss@lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Increase default maximum NFS server threads?
I agree, and, it really has been a long time coming. Those defaults were set ages ago, systems and networking have vastly improved since then. Might as well set defaults accordingly. Ian On Tue, Dec 6, 2016 at 7:51 AM, Richard Elling < richard.ell...@richardelling.com> wrote: > > > On Dec 6, 2016, at 7:30 AM, Dan McDonald <dan...@omniti.com> wrote: > > > > I got a link to this commit from the Delphix illumos repo a while back: > > > > https://github.com/openzfs/openzfs/pull/186/ > > > > I was curious if NFS-using people in the audience here would like to see > this one Just Land (TM) in illumos-omnios or not? > > just land it, no real downside > — richard > > > ___ > OmniOS-discuss mailing list > OmniOS-discuss@lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] LDAP external auth for CIFS service
In all honesty, the native Solaris LDAP client sucks. I would investigate installing an OpenLDAP client, or make the system an OpenLDAP slave to your 389 DS, and have the local client talk to the local OpenLDAP slave via loopback. That's how we were able to successfully set things up on our Thors so that they could talk to our OpenLDAP servers. Ian On Thu, Aug 18, 2016 at 11:28 AM, Andries Annema <an3s.ann...@gmail.com> wrote: > *raises hand* > Here's another one interested in this matter. > > Researched the possibilities about two years ago myself, but eventually > gave > up; it didn't seem to be possible. > Would be awesome if it would be one day, though. > > Regards, > Andries > > > -Original Message- > From: OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] On > Behalf Of Tobias Oetiker > Sent: donderdag 18 augustus 2016 17:33 > To: Dan McDonald > Cc: omnios-discuss > Subject: Re: [OmniOS-discuss] LDAP external auth for CIFS service > > - On 18 Aug, 2016, at 17:15, Dan McDonald dan...@omniti.com wrote: > > >> On Aug 18, 2016, at 11:04 AM, Mick Burns <bmx1...@gmail.com> wrote: > >> > >> *bump* > >> anyone ? > > > > I'm going to forward your note to someone I know who works on CIFS. He's > not on > > this list. > > looking forward to the answer ... :) I have always used an AD for this but > openldap would be so much cooler. > > cheers > tobi > ___ > OmniOS-discuss mailing list > OmniOS-discuss@lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > > _______ > OmniOS-discuss mailing list > OmniOS-discuss@lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] LDAP and Active Directory via rfc2307
Can you pull an complete user object via LDAP query? There might be secondary attributes that include a POSIX compliant short name. Ian On Fri, Apr 22, 2016 at 2:37 PM, Michael Talbott <mtalb...@lji.org> wrote: > It does have the unix extensions on it which is how I was able to get this > far (set uids/gids/etc in AD). But I don't have the old windows NIS service > running though, so I don't use the SFU30 or whatever attributes since I > believe those are all obsoleted and will soon likely disappear. > > > Michael Talbott > Systems Administrator > La Jolla Institute > > On Apr 22, 2016, at 1:18 PM, Ian Kaufman <ikauf...@eng.ucsd.edu> wrote: > > Does your AD have SFU (or whatever it is called these days) set up? > > Ian > > On Fri, Apr 22, 2016 at 12:58 PM, Michael Talbott <mtalb...@lji.org> > wrote: > >> You're exactly right. The DN in ad is the full name and if I create a >> user where the DN and shortname match, then everything works great. >> Unfortunately, I'm not sure if updating all the DNs to match the short name >> will break other dependancies of it deployed in existing software >> elsewhere. One day when I'm feeling brave and have a little downtime >> scheduled, I'll batch update all the entries and see if anything breaks. >> But, I suppose I'm stuck with winbind for the time being. But thank you for >> all the help. >> >> >> >> > On Apr 22, 2016, at 11:27 AM, Paul B. Henson <hen...@acm.org> wrote: >> > >> > On Thu, Apr 21, 2016 at 11:35:56PM -0700, Michael Talbott wrote: >> > >> >> all the group members are listed as "John Doe" rather than jdoe which >> >> means that when jdoe logs in, he can't access his groups due to the >> >> naming disconnect. Any ideas of how to fix that? Somehow map the group >> >> members to samAccountName rather than the DN? >> > >> > How is your AD structured? It sounds like it's using full names for DN's >> > rather than usernames? If so, that's not going to work. >> > >> > Our AD uses usernames for DN's; for example, I'm: >> > >> > dn: CN=henson,OU=user,DC=ad,DC=cpp,DC=edu >> > cn: henson >> > sn: Henson >> > givenName: Paul >> > initials: B. >> > distinguishedName: CN=henson,OU=user,DC=ad,DC=cpp,DC=edu >> > displayName: Paul B. Henson >> > sAMAccountName: henson >> > >> > and if you look at a group I'm in: >> > >> > dn: CN=netadmin,OU=group,DC=ad,DC=cpp,DC=edu >> > cn: netadmin >> > description: Network admins >> > member: CN=henson,OU=user,DC=ad,DC=cpp,DC=edu >> > distinguishedName: CN=netadmin,OU=group,DC=ad,DC=cpp,DC=edu >> > sAMAccountName: netadmin >> > >> > So the RDN for both users and groups is the short name that a unix box >> > expects to see, and the long name is in the displayName or description. >> > I'm guessing you're using the full name as the CN and your users look >> > like: >> > >> > dn: CN=Paul B. Henson,OU=user,DC=ad,DC=cpp,DC=edu >> > >> > so your group members look like: >> > >> > member: CN=Paul B. Henson,OU=user,DC=ad,DC=cpp,DC=edu >> > >> > If that's the case, I don't think there's any way you can get it to >> > work. The rfc2307bis group support expects the RDN to be the username, >> > there's no way to get it to look up some other attribute of the entry >> > and use it instead. >> >> ___ >> OmniOS-discuss mailing list >> OmniOS-discuss@lists.omniti.com >> http://lists.omniti.com/mailman/listinfo/omnios-discuss >> > > > > -- > Ian Kaufman > Research Systems Administrator > UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu > > > -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] LDAP and Active Directory via rfc2307
Does your AD have SFU (or whatever it is called these days) set up? Ian On Fri, Apr 22, 2016 at 12:58 PM, Michael Talbott <mtalb...@lji.org> wrote: > You're exactly right. The DN in ad is the full name and if I create a user > where the DN and shortname match, then everything works great. > Unfortunately, I'm not sure if updating all the DNs to match the short name > will break other dependancies of it deployed in existing software > elsewhere. One day when I'm feeling brave and have a little downtime > scheduled, I'll batch update all the entries and see if anything breaks. > But, I suppose I'm stuck with winbind for the time being. But thank you for > all the help. > > > > > On Apr 22, 2016, at 11:27 AM, Paul B. Henson <hen...@acm.org> wrote: > > > > On Thu, Apr 21, 2016 at 11:35:56PM -0700, Michael Talbott wrote: > > > >> all the group members are listed as "John Doe" rather than jdoe which > >> means that when jdoe logs in, he can't access his groups due to the > >> naming disconnect. Any ideas of how to fix that? Somehow map the group > >> members to samAccountName rather than the DN? > > > > How is your AD structured? It sounds like it's using full names for DN's > > rather than usernames? If so, that's not going to work. > > > > Our AD uses usernames for DN's; for example, I'm: > > > > dn: CN=henson,OU=user,DC=ad,DC=cpp,DC=edu > > cn: henson > > sn: Henson > > givenName: Paul > > initials: B. > > distinguishedName: CN=henson,OU=user,DC=ad,DC=cpp,DC=edu > > displayName: Paul B. Henson > > sAMAccountName: henson > > > > and if you look at a group I'm in: > > > > dn: CN=netadmin,OU=group,DC=ad,DC=cpp,DC=edu > > cn: netadmin > > description: Network admins > > member: CN=henson,OU=user,DC=ad,DC=cpp,DC=edu > > distinguishedName: CN=netadmin,OU=group,DC=ad,DC=cpp,DC=edu > > sAMAccountName: netadmin > > > > So the RDN for both users and groups is the short name that a unix box > > expects to see, and the long name is in the displayName or description. > > I'm guessing you're using the full name as the CN and your users look > > like: > > > > dn: CN=Paul B. Henson,OU=user,DC=ad,DC=cpp,DC=edu > > > > so your group members look like: > > > > member: CN=Paul B. Henson,OU=user,DC=ad,DC=cpp,DC=edu > > > > If that's the case, I don't think there's any way you can get it to > > work. The rfc2307bis group support expects the RDN to be the username, > > there's no way to get it to look up some other attribute of the entry > > and use it instead. > > ___ > OmniOS-discuss mailing list > OmniOS-discuss@lists.omniti.com > http://lists.omniti.com/mailman/listinfo/omnios-discuss > -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] big zfs storage?
Hi Linda, I have two dual-headed OmniOS system attached to two 60 bay JBODs w/ 3TB drives. Using raidz2 and hot spares, each JBOD has about 106TB usable, and i use a single filesystem per JBOD. We may or may not add another JBOD, but we will be expanding into the PB realm eventually. I use napp-it to make things easier to manage, and I will start testing some HA/Failover. I also have two ~70TB usable systems. Ian On Thu, Jul 9, 2015 at 2:21 PM, Linda Kateley lkate...@kateley.com wrote: Hey is there anyone out there running big zfs on omni? I have been doing mostly zol and freebsd for the last year but have to build a 300+TB box and i want to come back home to roots(solaris). Feeling kind of hesitant :) Also, if you had to do over, is there anything you would do different. Also, what is the go to HBA these days? Seems like i saw stable code for lsi 3008? TIA linda ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] NFS v4.1 in OmniOS stable?
Lustre is great for HPC. It lacks the sanctity of data that other solutions have. It is getting there - using ZFS is a huge step forward, but Lustre is by no means a general purpose solution at this point. Ian On Fri, Mar 20, 2015 at 1:09 PM, Andrew Gabriel illu...@cucumber.demon.co.uk wrote: Dan McDonald wrote: On Mar 20, 2015, at 2:27 PM, Jeff Stockett jstock...@molalla.com wrote: Does OmniOS support NFS v4.1? 4.1 support is a new feature in esxi v6, and I was trying to set it up as described here: http://wahlnetwork.com/2015/02/02/nfs-v4-1/ Things of course work fine if I use NFS v3, but if I try v4.1, I get a timeout error when it tries to attach the data store. Both the omnios server and the esxi client are properly joined to Active Directory so I think the required Kerberos stuff should be working. We have NFS4.0 and earlier. We do not have NFS4.1. It would be a very sizeable undertaking, requiring illumos community support. If anyone would lead the charge on that, it'd be a storage-oriented firm, like Nexenta, or Delphix. Does anyone seriously use (or intend to use) 4.1 anymore? Lustre has the parallel file server market (with some pockets of AFS too). The large growth of parallel storage servers now tends to be object storage, and S3 (as a protocol) has become the standard for that. Kind of makes me wonder what the market for NFSv4.1 is? -- Andrew ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] High Availability storage with ZFS
Hi all, I have modified and played with some simple scripts to create my own heartbeat, STONITH, failover set up. I currently have a pair of redundant systems, each one having redundant heads connected to shared JBODs, with the data replicated to the backup system via ZFS send/recv. I am going to do further testing, as this is a new set up, but I had it down to a 10 second failover without too much hassle between heads at one point, and the delta for the data on the backup system was down to about 30 minutes at one point. Ian On Tue, Jan 6, 2015 at 7:19 AM, Stephan Budach stephan.bud...@jvm.de wrote: Am 06.01.15 um 14:08 schrieb Filip Marvan: Hi Vincenzo, your solution is much more better, so thank you very much for your notes. I will try that too! Filip From: Vincenzo Pii [mailto:p...@zhaw.ch] Sent: Tuesday, January 06, 2015 1:54 PM To: Filip Marvan Cc: omnios-discuss@lists.omniti.com Subject: Re: [OmniOS-discuss] High Availability storage with ZFS 2015-01-06 12:16 GMT+01:00 Filip Marvan filip.mar...@aira.cz: Hi as few guys before, I'm thinking again about High Availability storage with ZFS. I know, that there is great commercial RSF-1, but that's quite expensive for my needs. I know, that Sašo did a great job about that on his blog http://zfs-create.blogspot.cz but I never found the way, how to successfully configure that on current OmniOS versions. So I'm thinking about something more simple. Arrange two LUNs from two OmniOS ZFS storages in one software mirror through fibrechannel. Arrange that mirror in client, for example mdadm in Linux. I know, that it will have performance affect and I will lost some ZFS advantages, but I still can use snapshots, backups with send/receive and some other interesting ZFS things, so it could be usable for some projects. Is there anyone, who tried that before? Any eperience with that? Thank you, Filip Marvan Hi Filip, I am not directly answering your question, but I've gone through the configuration of HA (with pacemaker) on OmniOS in the past months and collected all my notes here: http://blog.zhaw.ch/icclab/use-pacemaker-and-corosync-on-illumos-omnios-to-run-a-ha-activepassive-cluster/, maybe it can be useful for you. In my experience, running pacemaker correctly on OmniOS is just the tip of the iceberg, then comes the implementation/configuration of the resource agents (and the cluster itself!). If this way is worth it, rather than a quicker and more custom solution, depends on the long term plans :). Best regards, Vincenzo. Have you looked at the setup that Saso Kiselkov describes in his blog here: http://zfs-create.blogspot.nl/2013/06/building-zfs-storage-appliance-part-1.html It seems to cover most of a pacemaker setup, including the resource agents. Cheers, budy ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
[OmniOS-discuss] weird library issue
Hi all, I have a situation where a bunch of libs are complaining about missing the following: libc.so.1 (ILLUMOS_0.7) or libc.so.1 (ILLUMOS_0.6) This is happening on one system, but I have others that are fine. A standard libc.so.1 exists, but various libs are looking for the tagged one. Any ideas? Thanks, Ian -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] weird library issue
It looks like this system is in some weird state where there are packages from 151006 and 151008. For example, /etc/release shows: root@massive02-b:~# cat /etc/release OmniOS v11 r151008 Copyright 2013 OmniTI Computer Consulting, Inc. All rights reserved. Use is subject to license terms. but, when I try to do a pkg update, I see this (multiple repeating lines): No suitable version of required package pkg://omnios/incorporation/jeos/omnios-userland@11,5.11-0.151008:20131204T231613Z found: Reject: pkg://omnios/incorporation/jeos/omnios-userland@11,5.11-0.151008:20131204T231613Z Reason: A version for 'incorporate' dependency on pkg:/library/security/openssl@1.0.1,5.11-0.151008 cannot be found No suitable version of required package pkg://omnios/incorporation/jeos/omnios-userland@11,5.11-0.151004:20121024T180535Z found: Reject: pkg://omnios/incorporation/jeos/omnios-userland@11,5.11-0.151004:20121024T180535Z Reason: A version for 'incorporate' dependency on pkg:/archiver/gnu-tar@1.26,5.11-0.151004 cannot be found No suitable version of required package pkg://omnios/incorporation/jeos/omnios-userland@11,5.11-0.151006:20130506T214442Z found: Reject: pkg://omnios/incorporation/jeos/omnios-userland@11,5.11-0.151006:20130506T214442Z Reason: A version for 'incorporate' dependency on pkg:/archiver/gnu-tar@1.26,5.11-0.151006 cannot be found and for openssl, I see this: root@massive02-b:~# pkg info library/security/openssl Name: library/security/openssl Summary: openssl - A toolkit for Secure Sockets Layer (SSL v2/v3) and Transport Layer (TLS v1) protocols and general purpose cryptographic library State: Installed Publisher: omnios Version: 1.0.1.10 (1.0.1j) Build Release: 5.11 Branch: 0.151006 Packaging Date: October 15, 2014 03:40:22 PM Size: 18.08 MB FMRI: pkg://omnios/library/security/openssl@1.0.1.10,5.11-0.151006:20141015T154022Z root@massive02-b:~# pkg install pkg:/library/security/openssl@1.0.1,5.11-0.151008 Creating Plan / pkg install: No matching version of library/security/openssl can be installed: Reject: pkg://omnios/library/security/openssl@1.0.1.5,5.11-0.151008:20131204T024253Z pkg://omnios/library/security/openssl@1.0.1.6,5.11-0.151008:20140110T173804Z pkg://omnios/library/security/openssl@1.0.1.7,5.11-0.151008:20140407T220403Z pkg://omnios/library/security/openssl@1.0.1.7,5.11-0.151008:20140408T142844Z pkg://omnios/library/security/openssl@1.0.1.8,5.11-0.151008:20140605T140630Z pkg://omnios/library/security/openssl@1.0.1.9,5.11-0.151008:20140807T035111Z Reason: Newer version pkg://omnios/library/security/openssl@1.0.1.10,5.11-0.151006:20141015T154022Z is already installed Ian On Wed, Dec 10, 2014 at 10:15 AM, Dan McDonald dan...@omniti.com wrote: On Dec 10, 2014, at 12:01 PM, Ian Kaufman ikauf...@eng.ucsd.edu wrote: Hi all, I have a situation where a bunch of libs are complaining about missing the following: libc.so.1 (ILLUMOS_0.7) or libc.so.1 (ILLUMOS_0.6) This is happening on one system, but I have others that are fine. A standard libc.so.1 exists, but various libs are looking for the tagged one. Any ideas? You'll need to provide more data. It's as if something you compiled was compiled against a later version of libc, and then moved to an earlier version of libc. Dan -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] NFSv4 id mapping only working on client but not server?
No, what we are saying is NFSv4 and RPC are not compatible right now, and thus AUTH_SYS/AUTH_UNIX will not map UIDs by name, but by number over RPC. If you are not going to use Kerberos or LDAP, then you need to sync UIDs/UIDNumber in /etc/passwd. It's not that Solaris and Linux are incompatible, it's that RPC does not support NFSv4. Ian On Mon, Dec 8, 2014 at 5:25 AM, John Klimek jkli...@gmail.com wrote: Thanks everybody. Are you guys saying that Solaris and Linux are incompatible for sharing NFSv4 ACLs (because of RPC differences, etc) ? On Sun, Dec 7, 2014 at 5:43 PM, Frank Pittel f...@deepthought.com wrote: I've tried doing this with NFS4 a couple of years ago and what I found was that while the UID mapping worked for things like ls anytime you tried to actually do anything with the file RPC calls were made and since that's outside of NFS they failed. It was a couple of years ago now since I went through it but the upshot is that at least then uid mapping didn't work in any meaningful way! The Other Frank On Thu, Dec 04, 2014 at 01:55:11PM -0500, John Klimek wrote: Thanks, I will give this a try. However, when I create a file if I check the acl on Omni, it does have extended permissions and looks correct. It's only the uid and gid that I'm trying to get mapped correctly. Do you still think that noacl will solve that problem? (I'm migrating some virtual machines so I can't try out the setting you mentioned until a few more minutes...) On Thu, Dec 4, 2014 at 1:37 PM, Michael Rasmussen m...@miras.org wrote: On Thu, 4 Dec 2014 13:20:04 -0500 John Klimek jkli...@gmail.com wrote: What can I have setup wrong? I can't find any debugging or logging options for nfsmapid... Also, does the uidmap and gidmap share.nfs options only work for NFSv3? Solaris and derivatives implementation of NFS ACL is not compliant to the Linux NFS ACL. More directly it is the ACL in Linux which is not POSIX conformant so to avoid problems you should add the mount option noacl in your Linux fstab file. Noacl will instruct Omnios NFS to revert to plain old uid/gid. -- Hilsen/Regards Michael Rasmussen Get my public GnuPG keys: michael at rasmussen dot cc http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xD3C9A00E mir at datanom dot net http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xE501F51C mir at miras dot org http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xE3E80917 -- /usr/games/fortune -es says: Don't just echo the code with comments - make every comment count. - The Elements of Programming Style (Kernighan Plaugher) ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] NFSv4 id mapping only working on client but not server?
Yes, I oversimplified things. The issue is that AUTH_SYS/AUTH_UNIX does not support NFSv4. AUTH_SYS uses the UID, not the name, so the mapping fails. So, when RPC uses AUTH_SYS, NFSv4 is SOL. http://thread.gmane.org/gmane.linux.nfsv4/7103/focus=7105 Supposedly. at least on the client side, this has been fixed somewhere upstream. However, the server side is not. https://bugzilla.linux-nfs.org/show_bug.cgi?id=226 Regardless, my point is that this is not a Solaris/Linux issue, as a Linux server and Linux clients would be in the same boat. On Mon, Dec 8, 2014 at 12:23 PM, Paul B. Henson hen...@acm.org wrote: From: Ian Kaufman Sent: Monday, December 08, 2014 8:54 AM No, what we are saying is NFSv4 and RPC are not compatible right now, and thus AUTH_SYS/AUTH_UNIX will not map UIDs by name, but by number over RPC. If you are not going to use Kerberos or LDAP, then you need to sync UIDs/UIDNumber in /etc/passwd. It's not that Solaris and Linux are incompatible, it's that RPC does not support NFSv4. Well, I don't think it's technically accurate to say NFSv4 and RPC are not compatible, nor that RPC does not support NFSv4. It's really more of an issue that the NFSv4 protocol failed to address ID mapping completely for nonsecure RPC methods. For secure RPC (such as Kerberos), identifiers are passed over the wire symbolically as strings, and the ID mapper on each side translates them to numeric identifiers. However, for insecure RPC using AUTH_SYS, NFSv4 continues to pass raw uid/gid identifiers over the wire. This limitation of the protocol renders ID mapping somewhat useless for NFSv4 over insecure RPC :(. Ideally they could address it in v4.1 or v4.2 by creating a new insecure AUTH_SYS_NAMES protocol, which would simply use symbolic string identifiers over the wire like secure NFS does. I guess there is not much interest or concern about this in the communities that deal with NFS protocol specifications 8-/. Perhaps the majority of deployments for people in decision-making capacity use secure NFS and thus are not impacted by this deficiency... -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] NFSv4 id mapping only working on client but not server?
As far as I recall, AUTH_UNIX (aka AUTH_SYS) uses RPC, and RPC has not been augmented to support NFSv4 yet. From an old thread in the OpenSolaris boards: - Note also, that although NFSv4 uses strings for uid/gid/domain, the underlying RPC layer uses the same authentication credentials as in previous NFS versions and other RPC programs. Since it's the RPC authentication that is used to control access, etc, don't expect too much from the NFSv4 name/id mapping. It's useful for ls -l listings, etc, but not for authentication. - On Fri, Dec 5, 2014 at 5:32 AM, John Klimek jkli...@gmail.com wrote: Paul, thanks! You understand exactly what I'm asking and what my problem is. Something seems to be going wrong (or incorrectly configured) with the NFS id mapping services on my Linux client or OmniOS server. I've setup the nfsmapid_domain and I believe it's working because before configuring it, everything was showing as nobody/nobody, but afterwards it shows the correct names. (on the client) I can also see in the client log and it's correctly mapping u...@domain.com in the Linux rpc.idmapd domain. The problem is that when I create a file on the client, the server (OmniOS) is showing the UID of the user that created the file instead of mapping it using the name to the local UID. It looks like I can avoid the problem if I synchronize UIDs and GIDs across all servers (it's a small home network so it's not a big problem and I'm not using LDAP or NIS), but NFSv4 was supposed to avoid this problem by using name mapping so you didn't have to synchronize UIDs/GIDs. On Thu, Dec 4, 2014 at 3:59 PM, Paul B. Henson hen...@acm.org wrote: From: Michael Rasmussen Sent: Thursday, December 04, 2014 11:11 AM Yes, because you want to avoid Omnios presents ACL which is incompatible with Linux ACL. I don't believe the ACL has anything to do with NFSv4 id mapping? And a ZFS ACL presented over NFSv4 is perfectly compatible with Linux. It's not a Linux POSIX ACL, and cannot be manipulated with getfacl/setfacl, you need to use the nfs4 ACL tools, but it works fine. http://forum.proxmox.com/threads/15793-CT-creation-on-NFS- Share?p=81530#post81530 In that thread, the user fails to chmod via NFS: chmod: changing permissions of `/mnt/pve/proxCT/private/108.tmp': Operation not permitted The root cause of which was a setting of restricted for aclmode: vdev1/proxCT aclmode restricted local Per the man page An aclmode property of restricted will cause the chmod(2) operation to return an error when used on any file or directory which has a non-trivial ACL whose entries can not be represented by a mode. The user could have set the inherited ACL on the initial filesystem to a trivial ACL, in which case chmod would've worked fine over NFS. In any case, I don't see anything in that thread that seems relevant to NFSv4 id mapping, which unless I misunderstand is the problem the OP is trying to resolve. On that subject, NFSv4 id mapping seems to be working fine for me between an omnios client and server. On the server, the file system is mounted as: /export/user/henson on export/user/henson read/write/nosetuid/nodevices/nonbmand/exec/xattr/atime/dev=2c5025c And exported as: /export/user/henson - nfs nosuid,sec=krb5i,sec=krb5p with the domain set: $ sharectl get -p nfsmapid_domain nfs nfsmapid_domain=csupomona.edu if I create a file on the server, it has the correct ownership: $ touch test_server $ ls -l test_server -rw-r--r--+ 1 henson csupomona 0 Dec 4 12:50 test_server on the client, the NFS export is mounted as: /mnt on files-www.csupomona.edu:/export/user/henson remote/read/write/setuid/devices/sec=krb5p/xattr/dev=85c0008 on Thu Dec 4 12:50:01 2014 the client has the same domain: $ sharectl get -p nfsmapid_domain nfs nfsmapid_domain=csupomona.edu The file created on the server shows up with the correct ownership: $ ls -l test_server -rw-r--r--+ 1 henson csupomona 0 Dec 4 12:50 test_server A file created on the client has the correct ownership: $ touch test_client $ ls -l test_client -rw-r--r--+ 1 henson csupomona 0 Dec 4 12:52 test_client And viewed back on the server, still correct: $ ls -l test_client -rw-r--r--+ 1 henson csupomona 0 Dec 4 12:52 test_client ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] infiniband
I have had an issue with the Hermon driver and Mellanox ConnectX2 QDR cards. On one system, I saw this in the log: massive02-a hermon: [ID 492207 kern.info] hermon1: CQE RNR retry counter exceeded and the card went offline. Now, the interface does not even come up. Three other identical systems are fine, as are two other systems with the same card running Illumian 1.0a. I haven't found a way to reset the RNR counter. Ian On Thu, Oct 23, 2014 at 4:49 PM, Michael Rasmussen m...@miras.org wrote: On Thu, 23 Oct 2014 19:25:47 -0400 Dan McDonald dan...@omniti.com wrote: Only older cards. By older cards do you the mean only 10 gbps cards? It is these cards I had in mind: Voltaire 500 EX-D Dual Port Host Channel Adapter PCIe 20Gbps DDR InfiniBand Card Do you know the PCI ID of that card? I can double-check, but I'm guessing the answer will be no. Dan prtconf -v from Solaris 11.1 https://forums.servethehome.com/index.php?threads/connectx-vpi-infiniband-card-and-solaris-11-1.3211/ -- Hilsen/Regards Michael Rasmussen Get my public GnuPG keys: michael at rasmussen dot cc http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xD3C9A00E mir at datanom dot net http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xE501F51C mir at miras dot org http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xE3E80917 -- /usr/games/fortune -es says: [Sir Stafford Cripps] has all the virtues I dislike and none of the vices I admire. -- Winston Churchill ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] infiniband
Mellanox ConnectX-2 cards came in SDR (10Gbps) , DDR (20Gbps), and QDR (40Gpbs) configs. Ian On Thu, Oct 23, 2014 at 5:57 PM, Michael Rasmussen m...@miras.org wrote: On Fri, 24 Oct 2014 11:51:30 +1100 David Bomba turbo...@gmail.com wrote: I have the exact same issue Ian. I have multiple XenServers connected to OmniOS storage units. Under specific circumstances, I get iscsi disconnects which prove fatal for the XenServer host guests. These machines run a mixed bag of ConnectX and ConnectX-2 cards I have managed to get the system stable, but with severe compromises. Previously using 10Gb cards, I didnt have any problems what so ever. ConnectX and ConnectX-2 is 40 gpbs cards? -- Hilsen/Regards Michael Rasmussen Get my public GnuPG keys: michael at rasmussen dot cc http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xD3C9A00E mir at datanom dot net http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xE501F51C mir at miras dot org http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xE3E80917 -- /usr/games/fortune -es says: Indifference will certainly be the downfall of mankind, but who cares? ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Illumos and Infiniband
On my systems, the X-2 cards claim 32 gbps. I sued IPoIB, and have seen transfers NFS transfers over 6 gbps and snapshot send/receive as high as 1.8 gbps while the system was simultaneously utilized during cluster computation. Ian On Wed, Jan 29, 2014 at 11:17 AM, David Bomba turbo...@gmail.com wrote: Hi Chris, We have an environment running OmniOS with both ConnectX and ConnectX-2 cards. The Hermon and Tavor drivers work very well. The only problem we had was that the CX-2 cards would only sync at 20gb/s. We export Comstar iSER targets and they have been rock solid for us for over a year. Dave On 30 January 2014 05:38, Chris Zembower zembo...@criterion.com wrote: Hello, I'm trying to dig up some information on Illumos (or Solaris) compatibility with the Mellanox ConnectX-3 VPI HCA. From what I can gather, the Hermon driver (for 1st-gen ConnectX) may work with this hardware with tweaking, but I can't seem to confirm that. OmniOS is my server OS of choice (I have several systems running), but the necessity of a working IB fabric here has led me to implement a ZFS on Linux box for this particular purpose. I'm currently using LIO/targetcli to export block SRP targets, and I'm not happy with it at all. Really wishing I had Comstar for this. Anyone here using IB with OmniOS? Anyone aware of a roadmap or active development of new IB drivers? Why isn't there an Illumos OFED port happening? No interest? It's thriving on the Linux side of the world... Please don't lecture me on the merits of 10GbE/FC over Infiniband, I've heard it all. :-) Any insights or suggestions are very much appreciated! Best, Chris ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Illumos and Infiniband
Heh - mistype. But that cease and desist we sent to IPoIB ... :) On Wed, Jan 29, 2014 at 11:42 AM, Dan Swartzendruber dswa...@druber.com wrote: On my systems, the X-2 cards claim 32 gbps. I sued IPoIB, and have seen transfers NFS transfers over 6 gbps and snapshot send/receive as high as 1.8 gbps while the system was simultaneously utilized during cluster computation. Ian I assume you meant 'used', not 'sued'? If not, your lawyer is in the wrong line of work :) -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Newbie to OmniOS
If the subnet mask is 255.255.252.0, the broadcast should be 129.82.31.255. Ian On Mon, Jan 13, 2014 at 3:02 PM, CJ Keist cj.ke...@colostate.edu wrote: Michael, Broadcast is set right (255.255.252.0) so 129.82.23.255 is correct for the broadcast. The 129.82.20.0 129.82.20.15 in the route table is consistent with other Solaris servers I have on the network (non-virtual). On 1/13/14, 3:45 PM, Michael Rasmussen wrote: On Mon, 13 Jan 2014 15:03:40 -0700 CJ Keist cj.ke...@colostate.edu wrote: Excuse the pics, but don't have cut paste ability while working through the java console. Here is all the network info I believe. I think you have entered wrong network data. IP 129.82.20.15 GW 129.82.20.1 BA 129.82.23.255 NW 129.82.20.0 ??? Something does not compute here. You are sure your broadcast address should not have been 129.82.20.255 instead of 129.82.23.255? ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -- C. J. Keist Email: cj.ke...@colostate.edu Systems Group Manager Solaris 10 OS (SAI) Engineering Network ServicesPhone: 970-491-0630 College of Engineering, CSU Fax: 970-491-5569 Ft. Collins, CO 80523-1301 All I want is a chance to prove 'Money can't buy happiness' ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Newbie to OmniOS
Oops, yup, my bad. For some reason, I was using 129.82.29.15 as the host ... Ian On Mon, Jan 13, 2014 at 3:35 PM, CJ Keist cj.ke...@colostate.edu wrote: Nope, 255.255.252.0/22 is 1022 host sub-net. On 1/13/14, 4:21 PM, Ian Kaufman wrote: If the subnet mask is 255.255.252.0, the broadcast should be 129.82.31.255. Ian On Mon, Jan 13, 2014 at 3:02 PM, CJ Keist cj.ke...@colostate.edu wrote: Michael, Broadcast is set right (255.255.252.0) so 129.82.23.255 is correct for the broadcast. The 129.82.20.0 129.82.20.15 in the route table is consistent with other Solaris servers I have on the network (non-virtual). On 1/13/14, 3:45 PM, Michael Rasmussen wrote: On Mon, 13 Jan 2014 15:03:40 -0700 CJ Keist cj.ke...@colostate.edu wrote: Excuse the pics, but don't have cut paste ability while working through the java console. Here is all the network info I believe. I think you have entered wrong network data. IP 129.82.20.15 GW 129.82.20.1 BA 129.82.23.255 NW 129.82.20.0 ??? Something does not compute here. You are sure your broadcast address should not have been 129.82.20.255 instead of 129.82.23.255? ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -- C. J. Keist Email: cj.ke...@colostate.edu Systems Group Manager Solaris 10 OS (SAI) Engineering Network ServicesPhone: 970-491-0630 College of Engineering, CSU Fax: 970-491-5569 Ft. Collins, CO 80523-1301 All I want is a chance to prove 'Money can't buy happiness' ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -- C. J. Keist Email: cj.ke...@colostate.edu Systems Group Manager Solaris 10 OS (SAI) Engineering Network ServicesPhone: 970-491-0630 College of Engineering, CSU Fax: 970-491-5569 Ft. Collins, CO 80523-1301 All I want is a chance to prove 'Money can't buy happiness' -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Physical slot based disk names for LSI SAS on OmniOS?
You can also use the sas2ircu directly via the command line. However,
Günther's efforts and integration into the napp-it GUI make it so much
easier to use.
Ian
On Thu, Oct 31, 2013 at 2:37 PM, alka a...@hfg-gmuend.de wrote:
hi Felix
its part of the monitor extension
(free for less than 8 disks)
see Menu disks SAS2 extension
Am 31.10.2013 um 19:26 schrieb Felix Nielsen:
Hi Günther,
Is that feature included in napp-it? If so where and if not how do I get it
:)
Thanks
Felix
Btw. napp-it rocks :)
Den 31/10/2013 kl. 19.03 skrev Günther Alka a...@hfg-gmuend.de:
I have added a physical slot detection that displays physical
Slot, WWN and serials together with the option to switch on the red
alert led on supported backplanes within napp-it with the help of
sas2ircu (a LSI tool that displays slot and disk serial).
On 30.10.2013 21:25, Chris Siebenmann wrote:
This is a long shot and I suspect that the answer is no, but: in
OmniOS, is it possible somehow to have disk device names for disks
behind LSI SAS controllers that are based on the physical slot ('phy')
that the disk is found in instead of the disk's reported WNN or serial
number?
(I understand that this is only easy if there are no SAS expanders
involved, which is the situation that I care about.)
I know that I can recover this information from prtconf -v with
appropriate mangling, but our contemplated environment would be much
more manageable if we could directly use physical slot based device
naming.
Thanks in advance. I'll post a summary if there are any private
replies (for anything besides 'nope, can't do it').
- cks
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
--
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
--
Ian Kaufman
Research Systems Administrator
UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] rsync issues
Hi all, It appears that the issue is the default set of ACLs created when setting up the filesystems. I will dig a little deeper, but I wound up deleting one of the smaller filesystems, and then doing a ZFS send/receive to recreate it from the Illumian 1.0a5 server. The ACLs changed to match the Illumian systems, and rsync appears to behave as expected now. Ian On Tue, Sep 10, 2013 at 1:27 PM, Tim Rice t...@multitalents.net wrote: On Mon, 9 Sep 2013, Ian Kaufman wrote: Hi all, I am trying to slurp data over from one IllumOS based system (Illumian) to another (latest stable OmniOS), and I am running into a problem. I keep seeing the error: rsync: failed to set permissions on FILENAME: Not owner (1) which is leaving me with all of my dirs/files either 777 or 666. I am using the -a flag, both as root and as the directory/file owner, and the namespace/user ids are consistent across the system. My idmapd domain is also consistent. Has anyone else seen this problem? I just go through migrating the data from my UnixWare server to OmniOS (omnios-b281e50) and saw a very similar error message. I say similar because if I remember correctly it came from tar not rsync. Then doing a rsync tuned up things nicely. The problem files all seemed to be suid root. Thanks, Ian -- Tim RiceMultitalents t...@multitalents.net -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
[OmniOS-discuss] rsync issues
Hi all, I am trying to slurp data over from one IllumOS based system (Illumian) to another (latest stable OmniOS), and I am running into a problem. I keep seeing the error: rsync: failed to set permissions on FILENAME: Not owner (1) which is leaving me with all of my dirs/files either 777 or 666. I am using the -a flag, both as root and as the directory/file owner, and the namespace/user ids are consistent across the system. My idmapd domain is also consistent. Has anyone else seen this problem? Thanks, Ian -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] rsync issues
Sadly, no, nothing is immutable as I was able to rsync between Illumian boxes without any problems. I think something is wrong with rsync in OmniOS. Ian On Mon, Sep 9, 2013 at 2:00 PM, Jim Klimov jimkli...@cos.ru wrote: On 2013-09-09 20:40, Ian Kaufman wrote: Hi all, I am trying to slurp data over from one IllumOS based system (Illumian) to another (latest stable OmniOS), and I am running into a problem. I keep seeing the error: rsync: failed to set permissions on FILENAME: Not owner (1) which is leaving me with all of my dirs/files either 777 or 666. I am using the -a flag, both as root and as the directory/file owner, and the namespace/user ids are consistent across the system. My idmapd domain is also consistent. One wild guess, especially if you are root on local system, is that maybe the original FS objects had extended attributes like immutable, and if that is set on a directory - that might somehow forbid further receiving of files into it? This is a wild guess again, but if it is right - you might want to retry, by receiving data and hard/symlinks first (without much care for attributes), and then pass over this with a more complete rsync option set, which would set FS attributes as well. HTH, //Jim ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Ian Kaufman Research Systems Administrator UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
