Re: RFC: nfsd in a vnet jail

[Julian Elischer]

On 11/27/22 11:13 AM, Bjoern A. Zeeb wrote:

On Sun, 27 Nov 2022, James Gritton wrote:


On 2022-11-25 15:17, Rick Macklem wrote:


Hi,

bz@ has encouraged me to fiddle with the nfsd
so that it works in a vnet jail.
I have now basically done so, specifically for
NFSv4, since NFSv3 presents various issues.

What I have not yet done is put global variables
in the vnet. This needs to be done so that the nfsd
can be run in multiple jail instances and/or in and
outside of a jail.
The problem is that there are 100s of global variables.

I can see two approaches:
1 - Move them all into the vnet jail. This would imply
that all the sysctls need to somehow be changed,
which would seem to be a POLA violation.


Not a POLA.  The sysctl (names) don't change.  Just the values are
duplicated per-jail.





As Marko and I (mostly Marko) were assigning different variable and 
sysctls
when Vnet first hit, it was generally  pretty obvious which were local 
and which were global.
The sysctls values need to mean the same thing in the jail as they do 
in an unjailed system,

so the sysctl names don't change, just the value reported changes.
Some systls become read-only or invisible in jails.
Sometimes this takes adding some boilerplate virtualization code to 
each sysctl howeve
there is already some inbuilt support in the SYSCTL framework for VNET 
isolation.

You just need to set the CTLFLAG_VNET bit in the sysctl definition.
I agree with what Bjorn says below.
There is a slight added complication in that it is not just vnet but 
jailing as well
that needs to be take into account because vnet doesn't affect VFS, 
but jails do.





It also implies a lot of stuff in the vnet.
2 - Just move the global variables that will always
differ from one nfsd to another (this would make
the sysctls global and apply to all nfsds).
This will keep the number of globals in the vnet
smaller.

I am currently leaning towards #2, put what do others
think?

rick
ps: Personally, I don't know what use there is of
running the nfsd inside a vnet jail, but bz@ has
some use case.


I would prefer closer to #2, unless you want to support only one 
jail running nfsd (which is admittedly one of the more likely 
scenarios).  I imagine it's a case-by-case judgement call, as to 
whether a particular knob should be global or per-jail.



I think the call is:  everything that if changed in a vnet jail that
could cause the entire system to be DoSed by changing the setting in 
the

jail defintitvely stays global.

Everything which needs to be writeable on a per-instance base probably
needs to be virtualised.

My main concern with virtualising the variables will be early boot and
and NFSROOT szenarios that will need access to them early on before the
virtual network stacks are properly initialized;  I can help sorting
that out if my concerns become real.  Most probably was sorted before
for NFSROOT with the IP stack so it's likely an okay job now.


Also given I have once done it before for another subsystem;  we could
think of a V_FS bit (and I write it that way to not confuse it with
VFS).  Now NFS sits somewhere between FS and NET so I am not surt where
it should belong should we ponder that route as yet another option (and
if we think that VNETs will be way too big one day? -- there's probably
a lot other fish still to fry though some of that has been burnt in the
past already).

I think I have another email or two on the subject (possibly 
privately);

sorry Rick that I haven't gotten to them more timely.  I'll have a look
later tonight.

/bz

Curious minds .. etc

[Julian Elischer]

Several years ago (OK, maybe 12 years ago) I did an experiment where I 
unpacked a
freebsd 1.1 (or maybe 2.0?) image into a subdirectory, and after 
installing various
compat packagesand options and a.out support and changing MAX_PID to 
be 6, I was able to

chroot to it and do a "make world". Things were stupidly fast.


Has anyone been able to do such a thing in recent years? One wonders 
what options one

would need and what the oldest Version we could run in this way was..


J

Re: Building ZFS disk images

[Julian Elischer]

On 9/28/21 9:15 AM, Rodney W. Grimes wrote:

On Tue, Sep 28, 2021 at 9:48 AM Rodney W. Grimes
 wrote:

^^^

re-guid the pool on first boot.

Isnt the proper place to solve this lack of Unique UUID creation
in the tool(s) that are creating the zfs pool in the first place.

Fixing it "post boot" seems to be a far to late hack and doesnt
fix any of the situations where one might import these pools
between creation and first boot.

No, because you might create a VM image once, then instantiate it
dozens or thousands of times.  The firstboot solution is great because
it lets you reuse the same image file.

I would continue to argue that the place to fix this is in the
"instantiate tool".  ESXI vmfs deals with this all the time
when you clone a disk.  And again the "fix at boot" does not
deal with the problem in that if I "instatiate" 10 copies of
a zpool for VM's and then try to mount 2 of them at once on
the host this problem rares it head.   Fix the problem as close
to point of creation as possible for minimal issues in all
operations for everyone.


Define a special magic "change to something else" ID that will always 
do the right thing exactly once.

Re: how to use the ktls

[Julian Elischer]

On 1/9/20 2:53 PM, Rick Macklem wrote:

John Baldwin wrote:

On 1/7/20 3:02 PM, Rick Macklem wrote:


Someone once told me they were working on a netgraph node that did TLS 
encapsulation of a stream.


I can not remember who it was, but I do remember they were dubious 
about being allowed to give it back.  :-(


I only mention this as it MAY be an architectural avenue to investigate.

Julian



Hi,

Now that I've completed NFSv4.2 I'm on to the next project, which is making NFS
work over TLS.
Of course, I know absolutely nothing about TLS, which will make this an 
interesting
exercise for me.
I did find simple server code in the OpenSSL doc. which at least gives me a 
starting
point for the initialization stuff.
As I understand it, this initialization must be done in userspace?

Then somehow, the ktls takes over and does the encryption of the
data being sent on the socket via sosend_generic(). Does that sound right?

So, how does the kernel know the stuff that the initialization phase (handshake)
figures out, or is it magic I don't have to worry about?

Don't waste much time replying to this. A few quick hints will keep me going for
now. (From what I've seen sofar, this TLS stuff isn't simple. And I thought 
Kerberos
was a pain.;-)

Thanks in advance for any hints, rick

Hmmm, this might be a fair bit of work indeed.

If it was easy,  it wouldn't be fun;-) FreeBSD13 is a ways off and if it 
doesn't make that, oh well..


Right now KTLS only works for transmit (though I have some WIP for receive).

Hopefully your WIP will make progress someday, or I might be able to work on it.


KTLS does assumes that the initial handshake and key negotiation is handled by
OpenSSL.  OpenSSL uses custom setockopt() calls to tell the kernel which
session keys to use.

Yea, I figured I'd need a daemon like the gssd for this. The krpc makes it a 
little
more fun, since it handles TCP connections in the kernel.


I think what you would want to do is use something like OpenSSL_connect() in
userspace, and then check to see if KTLS "worked".

Thanks (and for the code below). I found the simple server code in the OpenSSL 
doc,
but the client code gets a web page and is quite involved.


If it did, you can tell
the kernel it can write to the socket directly, otherwise you will have to
bounce data back out to userspace to run it through SSL_write() and have
userspace do SSL_read() and then feed data into the kernel.

I don't think bouncing the data up/down to/from userland would work well.
I'd say "if it can't be done in the kernel, too bad". The above could be used 
for
a NULL RPC to see it is working, for the client.


The pseudo-code might look something like:

SSL *s;

s = SSL_new(...);

/* fd is the existing TCP socket */
SSL_set_fd(s, fd);
OpenSSL_connect(s);
if (BIO_get_ktls_send(SSL_get_wbio(s)) {
  /* Can use KTLS for transmit. */
}
if (BIO_get_ktls_recv(SSL_get_rbio(s)) {
   /* Can use KTLS for receive. */
}

Thanks John, rick


--
John Baldwin
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Reverting -current by date.

[Julian Elischer]

On 11/20/19 12:02 PM, bob prohaska wrote:

On Wed, Nov 20, 2019 at 11:18:41AM -0700, Warner Losh wrote:

On Wed, Nov 20, 2019 at 10:39 AM bob prohaska  wrote:


 From time to time it would be handy to revert freebsd-current to
an older, well-behaved revision.

Is there a mechanism for identifying revision numbers that
will at least compile and boot, by date?


Almost all of them will compile. Almost all of those will boot. While some
build breakage sneaks through, the default assumption is that it's good.
That's certainly been my experience randomly updating to -current. There's
some that are more or less performant, mind you, and some that are more or
less stable, it is true. But the overwhelming vast majority will compile
and boot, at least for amd64. I have issues less than 1% of the time when
updating to whatever is current at the moment I fancy an update.


Are commits that depend on one another somehow grouped in a single revision?

Unlike in the old CVS days,  yes, our commit rules require any
single commit to not break the build or system, which, as a corollary 
means
that people should commit all changes required to do so in a single 
commit.
  

There's some hardware that gets broken from time to time, but we don't
track that specifically. And non-amd64 architectures takes more care and
planning as any build breakage for those platforms lasts longer, in direct
proportion to how popular the platform is


Point taken. I'm interested in aarch64, which puts me somewhat in the weeds.
  

It's all in the commit logs. If you run -current you need to read them.
They will also tell you almost always if you pick revision X if there was a
subsequent fix that made things compile you should go with.


I take it the strategy would be go back in the log to a rough date,
then browse forward in time looking for signs of major trouble. When
the commits turn minor/benign, select a revision from that timeframe.
  

basically yes. I usually do a (more or less) binary search, informed by
examination of the sources and commit history.

Study the commit logs? I know I'm harping on that, but when things go
wrong, that's what I do.


I hoped for a more mechanical approach. For example, snapshots are
generated from time to time. Presumably, they're vetted in some way
and knowing what revisions made it to the snapshot stage might be a
starting point. The snapshot server does not appear to contain that
information for earlier offerings.

yes there are snapshots but they are more likely to be vetted for
compiling than for running successfully on your hardware.



Also -DNO_CLEAN builds help a lot if you're worried about it not even
building, though from time to time you run into issues with a NO_CLEAN
build due to a recent commit that wasn't appreciated at the time of the
commit, but was later and fixed.


Does -DNO_CLEAN behave sanely (and usefully) when going backwards in time?
I commonly use it for small forward steps, but time reversal is tricky 8-)
Good question.. that would depend on whether the source control 
program you use updates the timestamps of the files to be the time of  
commit or time of update.. I am pretty sure svn just uses the time 
that you ran the command. However SOMETIMES there are changes that are 
made to support forward migration that may not work well with 
backwards migration..  I tend to do several NO_CLEAN builds but every 
now and then  I'll do a full build if I've gone too far or if things 
seem a bit odd.


Thanks for replying!

bob prohaska



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: g_vfs_done():ufs/rootfs[WRITE flood on rpi3

[Julian Elischer]

On 11/20/19 2:37 PM, bob prohaska wrote:

While compiling a kernel at r354909 using a system at
r354845 the serial console flooded with messages of the
form
g_vfs_done():ufs/rootfs[WRITE(offset=266534912, length=32768)]error = 5

occasionally interspersed with

mmcsd0: Error indicated: 1 Timeout
g_vfs_done():ufs/rootfs[WRITE(offset=268697600, length=32768)]emmcsd0: Error 
indicated: 1 Timeout

It looked like the microSD card had finally failed, but breaking to
the debugger and rebooting seems to have recovered normally: The kernel
build picked up where it left off and the new kernel appears to boot
and run.

There are a few explanations as to how this succeeded.

One thing to note is that during builds, there is a large chance that 
any given write does not to be persisted as it is a temporary write of 
some kind.  As long as the next reader of the file can get it from 
cache, things will succeed.


There is also of course the chance that the write actually succeeded 
but that it happened later or that the response was bad rather than 
the write..  This can happen due to bugs in the firmware of the card.


And finally, potentially when you rebuilt, the failed steps were 
redone.  UFS Soft updates would help that result as it will not allow 
the metadata to be written until the data for the file was 
successfully written..


It will however be worth keeping an eye on that device..

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Reverting -current by date.

[Julian Elischer]

On 11/20/19 1:51 PM, Mark Millard wrote:

Bob P. wrote for an aarch64 context:


 From time to time it would be handy to revert freebsd-current to
an older, well-behaved revision.

Is there a mechanism for identifying revision numbers that
will at least compile and boot, by date?

In my case buildworld seems to be markedly slower than, say,
six months ago. Maybe it's hardware, maybe something else. Is
there a way to pick a revision number to revert to, that's
better than merely guessing?

You can explore the history of installable world/kernel materials
at places matching the pattern:

https://artifact.ci.freebsd.org/snapshot/head/r*/arm64/aarch64/*


I find that https://svnweb.freebsd.org/base/ is good enough to allow 
exploration of revisions. If you add a revision in the given field you 
will only see revs show up that are older than that.




Some r* will exist without having arm64/aarch64 material but having
mateirals for other platforms. The * matches a svn revision number.

These are from ci.freebsd.org builds, not the announced/released
snaphots. These go back a long ways. ci.freebsd.org does not try
to build every svn revision. (It does not build that fast relative
to svn updating.)

I sometimes use these for approximate bisection without building.

The https://artifact.ci.freebsd.org/snapshot/head/r*/arm64/aarch64/*
materials are compressed tar archives (*.txz) and a MANIFEST file.
bsdinstall uses these files, although I've at times done basic
testing activities based on just manually expanding the tars.

(The materials do not span u-boot or other such and need to have
file system(s) to expand into.)

(From a different message:)

I hoped for a more mechanical approach. For example, snapshots are
generated from time to time. Presumably, they're vetted in some way
and knowing what revisions made it to the snapshot stage might be a
starting point. The snapshot server does not appear to contain that
information for earlier offerings.

Snapshots are not vetted as far as I know --and sometimes do not boot
in contexts that I've tried. This applies both to artifact.ci.freebsd.org
materials and the announced snapshots put under
https://download.freebsd.org/ftp/snapshots/ .

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: panic: Assertion in_epoch(net_epoch_preempt) failed at ... src/sys/net/if.c:3694

[Julian Elischer]

On 10/8/19 4:22 PM, Ryan Stone wrote:

I haven't found any good references on the subject, but here's my understanding:

- epoch_enter() and epoch_exit() are very inexpensive operations
(cheaper than mtx, rw_lock or rm_lock operations) that are use to mark
read-only critical sections
- epoch_wait() guarantees that no threads that were in the critical
section when it was first called are still in the critical section
when it completes

With this guarantee, you can safely destroy an object with the
following procedure:

1. Atomically remove all global pointers to the object (e.g. remove it
from any lists that the critical sections might look it up in).  This
must be done atomically because read-only threads can be concurrently
running in the critical section.  This guarantees that no more threads
can get a pointer to it.
2. Call epoch_wait() to drain all threads that already held pointers
to it before step 1.
3. You now hold the only pointer to the object, so you are free to
destroy it as you please.


Ok thanks. Gottit.  thanks for the description.



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: panic: Assertion in_epoch(net_epoch_preempt) failed at ... src/sys/net/if.c:3694

[Julian Elischer]

On 10/8/19 2:42 PM, Walter Parker wrote:

See the Wikipedia page at https://en.wikipedia.org/wiki/Unix_time


but I was refering to the new "epoch" faciity in the kernel..  ( man 9 
epoch  )


https://www.freebsd.org/cgi/man.cgi?query=epoch=0=9=FreeBSD+13-current=default=html





Walter

On Tue, Oct 8, 2019 at 2:37 PM Julian Elischer <mailto:jul...@freebsd.org>> wrote:


Is there a paper or good description of the epoch concept? (more
readable than epoch(9)).
Haven't been to enough BSD confs recently.

julian
___
freebsd-current@freebsd.org <mailto:freebsd-current@freebsd.org>
mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to
"freebsd-current-unsubscr...@freebsd.org
<mailto:freebsd-current-unsubscr...@freebsd.org>"



--
The greatest dangers to liberty lurk in insidious encroachment by 
men of zeal, well-meaning but without understanding.   -- Justice 
Louis D. Brandeis



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: panic: Assertion in_epoch(net_epoch_preempt) failed at ... src/sys/net/if.c:3694

[Julian Elischer]

On 10/8/19 2:45 PM, Matthew Macy wrote:



On Tue, Oct 8, 2019 at 14:42 Walter Parker <mailto:walt...@gmail.com>> wrote:


See the Wikipedia page at https://en.wikipedia.org/wiki/Unix_time



Unrelated.
http://concurrencykit.org/slides.html

And see also Keir Fraser’s thesis where the idea originated.


Walter

On Tue, Oct 8, 2019 at 2:37 PM Julian Elischer
mailto:jul...@freebsd.org>> wrote:

> Is there a paper or good description of the epoch concept? (more
> readable than epoch(9)).
> Haven't been to enough BSD confs recently.
>
> julian
> ___
> freebsd-current@freebsd.org
<mailto:freebsd-current@freebsd.org> mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
"freebsd-current-unsubscr...@freebsd.org
<mailto:freebsd-current-unsubscr...@freebsd.org>"
>


-- 
The greatest dangers to liberty lurk in insidious encroachment

by men of
zeal, well-meaning but without understanding.   -- Justice Louis
D. Brandeis
___
freebsd-current@freebsd.org <mailto:freebsd-current@freebsd.org>
mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to
"freebsd-current-unsubscr...@freebsd.org
<mailto:freebsd-current-unsubscr...@freebsd.org>"


thanks for the answers


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: panic: Assertion in_epoch(net_epoch_preempt) failed at ... src/sys/net/if.c:3694

[Julian Elischer]

Is there a paper or good description of the epoch concept? (more 
readable than epoch(9)).

Haven't been to enough BSD confs recently.

julian
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: svn commit: r339898 - head/lib/libc/amd64/sys

[Julian Elischer]

what's an ifunc?


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: vnet & firewalls in 12.0

[Julian Elischer]

I will only discuss ipfw.. I dont' use pf.


On 18/10/18 11:33 am, Ernie Luzar wrote:
Wanting to get a head start on using 12.0 and vnet jails with in 
jail firewall.


1. Will Vimage be compiled as a module in the 12.0 kernel and be 
included in the base system release?

it's in base.. not  a module


1.a. Has the boot time console log message about vimage being 
"highly experimental" been removed?


2. Has the pf firewall been fixed so it can now run in a vnet jail 
or multiple vnet jails with out concern for which firewall is 
running on the host?


2.a. Is each vnet/pf log only viewable from it's vnet jail console?

2.b. Will pf/kernel module auto load on first call from a vnet jail?

2.c. Does vnet/pf NAT work?



3. Does the ipfw firewall still have the 11.x release mandatory 
requirements that the host must also be running ipfw for the vnet 
jailed ipfw to work?

never heard about that..
effectively each network stack can have its own firewall. The ipfw 
module must be loaded so it will be 'hooked into' each stack.

whether you use it or not is up to you.


3.a. Are all vnet/ipfw log messages still intermixed with the host's 
ipfw log messages?
that is probably the case.  there is no per-jail kernel logging 
facility. (Sounds like a good idea!  send patches!)


3.b. Does vnet/ipfw NAT work?

last I checked it did.



4. Has any work been done to ipf (ipfilter) so it will function when 
used in a vnet jail?

I don't know how many people are using that... not a lot.


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 
"freebsd-current-unsubscr...@freebsd.org"




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

a little secret info on AZURE VMs..

[Julian Elischer]

Apparently if you publish an Azure image to their marketplace, they
blindly store billing information at location 65536 of the VHD file..
so you need to ensure that your first partitions start after that.
if you use a layout with your first partition starting at 64 sectors
in, this location falls in the middle of your boot code.
So it fails to boot.

I haven't found any documentation of this yet..


Julian

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: gcc/clang interoperability problem with a custom "samba" build in recent -current.

[Julian Elischer]

On 25/7/18 12:40 am, Julian Elischer wrote:

On 22/7/18 4:32 am, Dimitry Andric wrote:

On 21 Jul 2018, at 21:11, Yuri Pankov  wrote:

Yuri Pankov wrote:

Julian Elischer wrote:

...

anyone know if there is a clang equivalent of -Wp, -E,-lang-asm?

In later GCC versions the cpp's -lang-asm seems to be deprecated in
favor of -x assembler-with-cpp as it conflicts with -l option.
Could you try changing the -Wp,-E,-lang-asm to 
-Wp,-E,-xassembler-with-cpp?
I did this but if failed.. I had to reread this email  a few times to 
think of replacing the whole

-Wp,-E,-lang-asm  with just  -xassembler-with-cpp!


Just tried it myself, and if you indeed mean the 
third_party/aesni-intel/aesni-intel_asm.c, the following seems to 
work for me:


clang -xassembler-with-cpp -c 
third_party/aesni-intel/aesni-intel_asm.c

Yes, that is exactly what I suggested to Julian on IRC.  The point is
that the ".c" extension is misleading, it should more likely be a ".S"
extension.  But maybe this source file is used for multiple purposes.

Note that -x assembler-with-cpp should also work fine for gcc.

Ah..  the trick is to remove the -Wp,-E as well...


-Dimitry


thanks

I tried that but the version of the file we have has several lines 
that caused problems...


a lot of the assembler has assembler comments (starting with '#') 
which clang complained about and died..


it also had #.align 4

which I HOPE is just a commented out line..
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 
"freebsd-current-unsubscr...@freebsd.org"




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: gcc/clang interoperability problem with a custom "samba" build in recent -current.

[Julian Elischer]

On 22/7/18 3:11 am, Yuri Pankov wrote:

Yuri Pankov wrote:

Julian Elischer wrote:

I would really like ot get some pointers as to who are our tools
committers at the moment, in particular who might know about these 
issues.

The main issue for me at the moment is the ability to compile the
aesni code in Samba from clang..

Julian


On 20/7/18 7:32 pm, Julian Elischer wrote:

compiling our samba with gcc 4.2.1 in 12 gave us some off behaviour
when lld became the linker I think..

1/ linking needed some directories added to some of the build
scripts because previously apparently it looked in $SYSROOT/usr/lib
by default and now it doesn't.

2/ compiling our samba produces a libtdb.so that has various symbols
in it, (according to nm(1) ), but when we try link against it we get
complaints about those symbols not being defined.

3/ an attempt to switch to using clang to compile everything 
leads to:



"--aes-accel=intelaesni selected and compiler rejects 
-Wp,-E,-lang-asm.


One wonders whether there is a clang equivalent of 
"-Wp,-E,-lang-asm"


The AES acceleration is a configure option for the samba package.

Apparently turning it on requires -Wp,-E,-lang-asm.

which apparently gcc 4.2.1 has, but clang doesn't have.

     FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) 
(based

     on LLVM 6.0.1)
     Target: x86_64-unknown-freebsd12.0
     Thread model: posix
     InstalledDir: /usr/bin

anyone know if there is a clang equivalent of -Wp, -E,-lang-asm?


In later GCC versions the cpp's -lang-asm seems to be deprecated in
favor of -x assembler-with-cpp as it conflicts with -l option.

Could you try changing the -Wp,-E,-lang-asm to 
-Wp,-E,-xassembler-with-cpp?


Just tried it myself, and if you indeed mean the 
third_party/aesni-intel/aesni-intel_asm.c, the following seems to 
work for me:


clang -xassembler-with-cpp -c third_party/aesni-intel/aesni-intel_asm.c


when I try it I get lots of errors due to hte fact that there are 
assembled comments that start with #

and they are all cited as errors by clang.
I had to put '//' before about 80 lines line that.





possible work arrounds include:

1/ Get gcc/lld to produce a library from which lld can find the 
symbols


2/ find a way to compile this with clang but everything else with 
gcc?


3/ find a way to allow clang to use
-Wp,-E,-lang-asm

whatever that means


Thoughts from any tools people?





___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: gcc/clang interoperability problem with a custom "samba" build in recent -current.

[Julian Elischer]

On 22/7/18 4:32 am, Dimitry Andric wrote:

On 21 Jul 2018, at 21:11, Yuri Pankov  wrote:

Yuri Pankov wrote:

Julian Elischer wrote:

...

anyone know if there is a clang equivalent of -Wp, -E,-lang-asm?

In later GCC versions the cpp's -lang-asm seems to be deprecated in
favor of -x assembler-with-cpp as it conflicts with -l option.
Could you try changing the -Wp,-E,-lang-asm to -Wp,-E,-xassembler-with-cpp?

Just tried it myself, and if you indeed mean the 
third_party/aesni-intel/aesni-intel_asm.c, the following seems to work for me:

clang -xassembler-with-cpp -c third_party/aesni-intel/aesni-intel_asm.c

Yes, that is exactly what I suggested to Julian on IRC.  The point is
that the ".c" extension is misleading, it should more likely be a ".S"
extension.  But maybe this source file is used for multiple purposes.

Note that -x assembler-with-cpp should also work fine for gcc.

-Dimitry


thanks

I tried that but the version of the file we have has several lines 
that caused problems...


a lot of the assembler has assembler comments (starting with '#') 
which clang complained about and died..


it also had #.align 4

which I HOPE is just a commented out line..
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: gcc/clang interoperability problem with a custom "samba" build in recent -current.

[Julian Elischer]

I would really like ot get some pointers as to who are our tools 
committers at the moment, in particular who might know about these issues.
The main issue for me at the moment is the ability to compile the 
aesni code in Samba from clang..


Julian


On 20/7/18 7:32 pm, Julian Elischer wrote:
compiling our samba with gcc 4.2.1 in 12 gave us some off behaviour 
when lld became the linker I think..


1/ linking needed some directories added to some of the build 
scripts because previously apparently it looked in $SYSROOT/usr/lib 
by default and now it doesn't.


2/ compiling our samba produces a libtdb.so that has various symbols 
in it, (according to nm(1) ), but when we try link against it we get 
complaints about those symbols not being defined.


3/ an attempt to switch to using clang to compile everything leads to:


"--aes-accel=intelaesni selected and compiler rejects -Wp,-E,-lang-asm.

One wonders whether there is a clang equivalent of "-Wp,-E,-lang-asm"

The AES acceleration is a configure option for the samba package.

Apparently turning it on requires -Wp,-E,-lang-asm.

which apparently gcc 4.2.1 has, but clang doesn't have.

   FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based
   on LLVM 6.0.1)
   Target: x86_64-unknown-freebsd12.0
   Thread model: posix
   InstalledDir: /usr/bin

anyone know if there is a clang equivalent of -Wp, -E,-lang-asm?

possible work arrounds include:

1/ Get gcc/lld to produce a library from which lld can find the symbols

2/ find a way to compile this with clang but everything else with gcc?

3/ find a way to allow clang to use
-Wp,-E,-lang-asm

whatever that means


Thoughts from any tools people?

Julian

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 
"freebsd-current-unsubscr...@freebsd.org"




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

gcc/clang interoperability problem with a custom "samba" build in recent -current.

[Julian Elischer]

compiling our samba with gcc 4.2.1 in 12 gave us some off behaviour 
when lld became the linker I think..


1/ linking needed some directories added to some of the build scripts 
because previously apparently it looked in $SYSROOT/usr/lib by default 
and now it doesn't.


2/ compiling our samba produces a libtdb.so that has various symbols 
in it, (according to nm(1) ), but when we try link against it we get 
complaints about those symbols not being defined.


3/ an attempt to switch to using clang to compile everything leads to:


"--aes-accel=intelaesni selected and compiler rejects -Wp,-E,-lang-asm.

One wonders whether there is a clang equivalent of "-Wp,-E,-lang-asm"

The AES acceleration is a configure option for the samba package.

Apparently turning it on requires -Wp,-E,-lang-asm.

which apparently gcc 4.2.1 has, but clang doesn't have.

   FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based
   on LLVM 6.0.1)
   Target: x86_64-unknown-freebsd12.0
   Thread model: posix
   InstalledDir: /usr/bin

anyone know if there is a clang equivalent of -Wp, -E,-lang-asm?

possible work arrounds include:

1/ Get gcc/lld to produce a library from which lld can find the symbols

2/ find a way to compile this with clang but everything else with gcc?

3/ find a way to allow clang to use
-Wp,-E,-lang-asm

whatever that means


Thoughts from any tools people?

Julian

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: GELI attach issue from r326073 -> r334996

[Julian Elischer]

On 14/6/18 4:46 am, Michael Jung wrote:

On 2018-06-13 15:27, Ian Lepore wrote:

On Wed, 2018-06-13 at 14:29 -0400, Michael Jung wrote:

Hi!

I just tried updating current from r326073 -> r334996 and when
I try 'geli attach' I get the following error:

# geli attach -p -k mykey.key /dev/gpt/da14
geli: Missing keyno argument
#

If I boot the old kernel GELI attaches just fine.

I ran into this once before but can not find the thread.  I recall
it being a bug... with no resolution.

I mount zfs partitions over GELI - my painful solution was to
nuke each GPT partition in the zpool, resilver, repeat and
rinse until everything was non-encrypted and repeat the cycle
to re-encrypt.  NOT FUN.

Looking for suggestions to supply additional information to debug
and resolve.

dmesg attached from working kernel.

Thanks!


r333439 added a '-n keyno' parameter to geli attach, but it's supposed
to default to trying all keys if you don't use -n. Is it possible
you're running a newer kernel (or geom_eli module) than your userland
or vice versa?

-- Ian


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 
"freebsd-current-unsubscr...@freebsd.org"


Ian:

Yes you are correct Maybe not the best method but I normally 
installkernel -
boot into single user mode - GELI attach, zfs mount -av, then 
installworld.


Yes that is the prescribed method. if it doesn't work then whoever 
changed the

geli code should fix it to handle this.
You should be able to run older code on newer kernels with very few 
exceptions.

I don't know if this also affects upgrades form 11. Have you tested that?


My boot volume is UFS, but many of the mount points are on zpools.

What would be the best way to test a new kernel without a full 
installworld

with new userland geli bits?

I currently don't have a way to backup my 35TB of data :-( and don't 
want to
lose anything. and I need a back out method should a full 
installworld fail.


Thanks for you quick reply.

--mikej
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 
"freebsd-current-unsubscr...@freebsd.org"





___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: head@r334204, Bad link elm in callout_process()

[Julian Elischer]

On 29/5/18 9:53 pm, Andriy Gapon wrote:

On 29/05/2018 14:53, Hans Petter Selasky wrote:

On 05/29/18 13:20, Andriy Gapon wrote:

(kgdb) p *$4.lh_first->c_links.le.le_next
$6 = {
    c_links = {
  le = {
    le_next = 0x0,
    le_prev = 0xfe0003999f98

Where does the le_prev point?

Typically happens when callouts are not properly drained before freeing memory.

Yeah, this could have been a pilot error.
I added some debugging code and that code could do callout_init + callout_reset
on an active or pending callout.  I am not seeing the problem without the
debugging code.
Sorry for the noise!

did you add code to that machine?? (bob.$work)






___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

crash on process exit.. current at about r332467

[Julian Elischer]

back trace at:  http://www.freebsd.org/~julian/bob-crash.png

If anyone wants to take a look..

In the exit syscall, while deallocating a vm object.

I haven't see references to a similar crash in the last 10 days or 
so.. But if it rings any bells...




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: SCHED_ULE makes 256Mbyte i386 unusable

[Julian Elischer]

On 22/4/18 9:43 pm, Rick Macklem wrote:

Konstantin Belousov wrote:

On Sat, Apr 21, 2018 at 11:30:55PM +, Rick Macklem wrote:

Konstantin Belousov wrote:

On Sat, Apr 21, 2018 at 07:21:58PM +, Rick Macklem wrote:

I decided to start a new thread on current related to SCHED_ULE, since I see
more than just performance degradation and on a recent current kernel.
(I cc'd a couple of the people discussing performance problems in freebsd-stable
  recently under a subject line of "Re: kern.sched.quantum: Creepy, sadistic 
scheduler".

When testing a pNFS server on a single core i386 with 256Mbytes using a Dec. 
2017
current/head kernel, I would see about a 30% performance degradation (elapsed
run time for a kernel build over NFSv4.1) when the server kernel was built with
options SCHED_ULE
instead of
options SCHED_4BSD

So, now that I have decreased the number of nfsd kernel threads to 32, it works
with both schedulers and with essentially the same performance. (ie. The 30%
performance degradation has disappeared.)


Now, with a kernel from a couple of days ago, the
options SCHED_ULE
kernel becomes unusable shortly after starting testing.
I have seen two variants of this:
- Became essentially hung. All I could do was ping the machine from the network.
- Reported "vm_thread_new: kstack allocation failed
   and then any attempt to do anything gets "No more processes".

This is strange.  It usually means that you get KVA either exhausted or
severly fragmented.

Yes. I reduced the number of nfsd threads from 256->32 and the SCHED_ULE
kernel is working ok now. I haven't done enough to compare performance yet.
Maybe I'll post again when I have some numbers.


Enter ddb, it should be operational since pings are replied.  Try to see
where the threads are stuck.

I didn't do this, since reducing the number of kernel threads seems to have 
fixed
the problem. For the pNFS server, the nfsd threads will spawn additional kernel
threads to do proxies to the mirrored DS servers.


with the only difference being a kernel built with
options SCHED_4BSD
everything works and performs the same as the Dec 2017 kernel.

I can try rolling back through the revisions, but it would be nice if someone
could suggest where to start, because it takes a couple of hours to build a
kernel on this system.

So, something has made things worse for a head/current kernel this winter, rick

There are at least two potentially relevant changes.

First is r326758 Dec 11 which bumped KSTACK_PAGES on i386 to 4.

I've been running this machine with KSTACK_PAGES=4 for some time, so no change.

W.r.t. Rodney Grimes comments about this (which didn't end up in this messages
in the thread):
I didn't see any instability when using KSTACK_PAGES=4 for this until this 
cropped
up and seemed to be scheduler related (but not really, it seems).
I bumped it to KSTACK_PAGES=4 because I needed that for the pNFS Metadata
Server code.

Yes, NFS does use quite a bit of kernel stack. Unfortunately, it isn't one big
item getting allocated on the stack, but many moderate sized ones.
(A part of it is multiple instances of "struct vattr", some buried in "struct 
nfsvattr",
  that NFS needs to use. I don't think these are large enough to justify 
malloc/free,
  but it has to use several of them.)

One case I did try fixing was about 6 cases where "struct nfsstate" ended up on
the stack. I changes the code to malloc/free them and then when testing, to
my surprise I had a 20% performance hit and shelved the patch.


you might try using uma. especially setting up a non-freeing zone, 
where he system allocates what it needs and then just recycles them.

(man uma)

Now that I know that the server was running near its limit, I might try this one
again, to see if the performance hit doesn't occur when the machine has adequate
memory. If the performance hit goes away, I could commit this, but it wouldn't 
have that much effect on the kstack usage. (It's interesting how this patch 
ended
up related to the issue this thread discussed.)


Second is r332489 Apr 13, which introduced 4/4G KVA/UVA split.

Could this change have resulted in the system being able to allocate fewer
kernel threads/stacks for some reason?

Well, it could, as anything can be buggy. But the intent of the change
was to give 4G KVA, and it did.

Righto. No concern here. I suspect the Dec. 2017 kernel was close to the limit
(see performance issue that went away, noted above) and any change could
have pushed it across the line, I think.


Consequences of the first one are obvious, it is much harder to find
the place to map the stack.  Second change, on the other hand, provides
almost full 4G for KVA and should have mostly compensate for the negative
effects of the first.

And, I cannot see how changing the scheduler would fix or even affect that
behaviour.

My hunch is that the system was running near its limit for kernel 
threads/stacks.
Then, somehow, the timing SCHED_ULE caused resulted in the nfsd trying to get

Re: SCHED_ULE makes 256Mbyte i386 unusable

[Julian Elischer]

On 22/4/18 10:36 pm, Rodney W. Grimes wrote:

Konstantin Belousov wrote:

On Sat, Apr 21, 2018 at 11:30:55PM +, Rick Macklem wrote:

Konstantin Belousov wrote:

On Sat, Apr 21, 2018 at 07:21:58PM +, Rick Macklem wrote:

I decided to start a new thread on current related to SCHED_ULE, since I see
more than just performance degradation and on a recent current kernel.
(I cc'd a couple of the people discussing performance problems in freebsd-stable
  recently under a subject line of "Re: kern.sched.quantum: Creepy, sadistic 
scheduler".

When testing a pNFS server on a single core i386 with 256Mbytes using a Dec. 
2017
current/head kernel, I would see about a 30% performance degradation (elapsed
run time for a kernel build over NFSv4.1) when the server kernel was built with
options SCHED_ULE
instead of
options SCHED_4BSD

So, now that I have decreased the number of nfsd kernel threads to 32, it works
with both schedulers and with essentially the same performance. (ie. The 30%
performance degradation has disappeared.)


Now, with a kernel from a couple of days ago, the
options SCHED_ULE
kernel becomes unusable shortly after starting testing.
I have seen two variants of this:
- Became essentially hung. All I could do was ping the machine from the network.
- Reported "vm_thread_new: kstack allocation failed
   and then any attempt to do anything gets "No more processes".

This is strange.  It usually means that you get KVA either exhausted or
severly fragmented.

Yes. I reduced the number of nfsd threads from 256->32 and the SCHED_ULE
kernel is working ok now. I haven't done enough to compare performance yet.
Maybe I'll post again when I have some numbers.


Enter ddb, it should be operational since pings are replied.  Try to see
where the threads are stuck.

I didn't do this, since reducing the number of kernel threads seems to have 
fixed
the problem. For the pNFS server, the nfsd threads will spawn additional kernel
threads to do proxies to the mirrored DS servers.


with the only difference being a kernel built with
options SCHED_4BSD
everything works and performs the same as the Dec 2017 kernel.

I can try rolling back through the revisions, but it would be nice if someone
could suggest where to start, because it takes a couple of hours to build a
kernel on this system.

So, something has made things worse for a head/current kernel this winter, rick

There are at least two potentially relevant changes.

First is r326758 Dec 11 which bumped KSTACK_PAGES on i386 to 4.

I've been running this machine with KSTACK_PAGES=4 for some time, so no change.

W.r.t. Rodney Grimes comments about this (which didn't end up in this messages
in the thread):
I didn't see any instability when using KSTACK_PAGES=4 for this until this 
cropped
up and seemed to be scheduler related (but not really, it seems).
I bumped it to KSTACK_PAGES=4 because I needed that for the pNFS Metadata
Server code.

Yes, NFS does use quite a bit of kernel stack. Unfortunately, it isn't one big
item getting allocated on the stack, but many moderate sized ones.
(A part of it is multiple instances of "struct vattr", some buried in "struct 
nfsvattr",
  that NFS needs to use. I don't think these are large enough to justify 
malloc/free,
  but it has to use several of them.)

One case I did try fixing was about 6 cases where "struct nfsstate" ended up on
the stack. I changes the code to malloc/free them and then when testing, to
my surprise I had a 20% performance hit and shelved the patch.
Now that I know that the server was running near its limit, I might try this one
again, to see if the performance hit doesn't occur when the machine has adequate
memory. If the performance hit goes away, I could commit this, but it wouldn't
have that much effect on the kstack usage. (It's interesting how this patch 
ended
up related to the issue this thread discussed.)

Anything we can do to help relieve KSTACK usage, especially on i386
is helpfull.  These is a thread back quite some time where someone
came up with a compile time static "this functions uses X bytes of
local stack" and a bit of clean up was done.  We should persue
this issue further.


that was me.

use
|-Wframe-larger-than||=|¶ 
 
and set it to something like 512 bytes




My experiece with the i386/KSTACK issues was attempting to do installs
from snapshot .iso's, I usually had to change to a custom kernel without
INVARIANTS and WITNESS, or reduce KSTACK to 2 and suffer the small stack
problem (ie, dont use NFS during install).  Neither was very pleasant.

I have found it in practical to run the 4 page KSTACK in production
VM's using i386 due to memory requirements.  I run many very lean
i386 VM's with 64MB of memory.  I suspect our user base also has
many people doing this, and it would be to our advantage to try
and reduce our kernel stack needs.



Second is r332489 Apr 13, which introduced 4/4G 

Re: SCHED_ULE makes 256Mbyte i386 unusable

[Julian Elischer]

On 22/4/18 10:36 pm, Rodney W. Grimes wrote:

Konstantin Belousov wrote:

On Sat, Apr 21, 2018 at 11:30:55PM +, Rick Macklem wrote:

Konstantin Belousov wrote:

On Sat, Apr 21, 2018 at 07:21:58PM +, Rick Macklem wrote:

I decided to start a new thread on current related to SCHED_ULE, since I see
more than just performance degradation and on a recent current kernel.
(I cc'd a couple of the people discussing performance problems in freebsd-stable
  recently under a subject line of "Re: kern.sched.quantum: Creepy, sadistic 
scheduler".

When testing a pNFS server on a single core i386 with 256Mbytes using a Dec. 
2017
current/head kernel, I would see about a 30% performance degradation (elapsed
run time for a kernel build over NFSv4.1) when the server kernel was built with
options SCHED_ULE
instead of
options SCHED_4BSD

So, now that I have decreased the number of nfsd kernel threads to 32, it works
with both schedulers and with essentially the same performance. (ie. The 30%
performance degradation has disappeared.)


Now, with a kernel from a couple of days ago, the
options SCHED_ULE
kernel becomes unusable shortly after starting testing.
I have seen two variants of this:
- Became essentially hung. All I could do was ping the machine from the network.
- Reported "vm_thread_new: kstack allocation failed
   and then any attempt to do anything gets "No more processes".

This is strange.  It usually means that you get KVA either exhausted or
severly fragmented.

Yes. I reduced the number of nfsd threads from 256->32 and the SCHED_ULE
kernel is working ok now. I haven't done enough to compare performance yet.
Maybe I'll post again when I have some numbers.


Enter ddb, it should be operational since pings are replied.  Try to see
where the threads are stuck.

I didn't do this, since reducing the number of kernel threads seems to have 
fixed
the problem. For the pNFS server, the nfsd threads will spawn additional kernel
threads to do proxies to the mirrored DS servers.


with the only difference being a kernel built with
options SCHED_4BSD
everything works and performs the same as the Dec 2017 kernel.

I can try rolling back through the revisions, but it would be nice if someone
could suggest where to start, because it takes a couple of hours to build a
kernel on this system.

So, something has made things worse for a head/current kernel this winter, rick

There are at least two potentially relevant changes.

First is r326758 Dec 11 which bumped KSTACK_PAGES on i386 to 4.

I've been running this machine with KSTACK_PAGES=4 for some time, so no change.

W.r.t. Rodney Grimes comments about this (which didn't end up in this messages
in the thread):
I didn't see any instability when using KSTACK_PAGES=4 for this until this 
cropped
up and seemed to be scheduler related (but not really, it seems).
I bumped it to KSTACK_PAGES=4 because I needed that for the pNFS Metadata
Server code.

Yes, NFS does use quite a bit of kernel stack. Unfortunately, it isn't one big
item getting allocated on the stack, but many moderate sized ones.
(A part of it is multiple instances of "struct vattr", some buried in "struct 
nfsvattr",
  that NFS needs to use. I don't think these are large enough to justify 
malloc/free,
  but it has to use several of them.)

One case I did try fixing was about 6 cases where "struct nfsstate" ended up on
the stack. I changes the code to malloc/free them and then when testing, to
my surprise I had a 20% performance hit and shelved the patch.
Now that I know that the server was running near its limit, I might try this one
again, to see if the performance hit doesn't occur when the machine has adequate
memory. If the performance hit goes away, I could commit this, but it wouldn't
have that much effect on the kstack usage. (It's interesting how this patch 
ended
up related to the issue this thread discussed.)

Anything we can do to help relieve KSTACK usage, especially on i386
is helpfull.  These is a thread back quite some time where someone
came up with a compile time static "this functions uses X bytes of
local stack" and a bit of clean up was done.  We should persue
this issue further.


that was me.

use
|-Wframe-larger-than||=|¶ 
 
and set it to something like 512 bytes (obviously you have to make 
warnings non fatal as well).






My experiece with the i386/KSTACK issues was attempting to do installs
from snapshot .iso's, I usually had to change to a custom kernel without
INVARIANTS and WITNESS, or reduce KSTACK to 2 and suffer the small stack
problem (ie, dont use NFS during install).  Neither was very pleasant.

I have found it in practical to run the 4 page KSTACK in production
VM's using i386 due to memory requirements.  I run many very lean
i386 VM's with 64MB of memory.  I suspect our user base also has
many people doing this, and it would be to our advantage to try
and reduce our kernel stack 

Re: anyone running with ngroups increased from 16?

[Julian Elischer]

On 16/4/18 6:37 pm, Julian Elischer wrote:
Windows users seem to have an almost unlimited number of groups and 
soem places seem to use them a LOT.
This gives Posix systems problems with deciding how to handle them 
all. Especially when getting

user credentials from winbindd (samba).

Does anyone know of any work done to either bypass this limit or to 
at least expand it?


I mean with the other applications such NFS usages etc.
I know mountd explodes with > 16..  has anyone done a cleaning pass?



Thanks

Julian

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 
"freebsd-current-unsubscr...@freebsd.org"




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Module compiles looking in /usr/src when alternate src tree is in use

[Julian Elischer]

On 11/4/18 8:29 am, Rodney W. Grimes wrote:

Rodney W. Grimes  wrote:


I am having a compile time issue for a patched that compiled fine on my
r329294 system, but now failes to compile with what looks like a wrong
header being included.

You may find it helpful to do something like:

make -dv -C sys/modules/vmm -V CFLAGS > /tmp/dvo 2>&1
egrep ':.PARSE|/usr/src/sys' /tmp/dvo | grep -B1 usr/src | more

The arg to -V doesn't matter btw.
You could narrow that down if you know what var -I/usr/src/sys is in
(probably CFLAGS but you never know)
the above should help find the makefile that is introducing the bogus -I


Thank you, that does help narrow it down:  (I backed up a vew lines
from the first place I saw src/.)

...
Global:.PARSEFILE = bsd.kmod.mk
Global:.PARSEDIR = /usr/src-topo/share/mk
Global:.PARSEFILE = bsd.kmod.mk
Result[] of :U is "/usr/src/sys"
Result[] of :U is "/usr/src/sys"
Global:SYSDIR = ${:U/usr/src/sys:tA}
Global:.PARSEDIR = /usr/src-topo/share/mk
Global:.PARSEFILE = bsd.kmod.mk
Result[] of :U is "/usr/src/sys"
Applying[] :t to "/usr/src/sys"
Result[] of :t is "/usr/src/sys"
Result[] of :U is "/usr/src/sys"
Applying[] :t to "/usr/src/sys"
Result[] of :t is "/usr/src/sys"
Result[] of :U is "/usr/src/sys"
Applying[] :t to "/usr/src/sys"
Result[] of :t is "/usr/src/sys"
Global:.MAKE.MAKEFILES = /usr/src-topo/share/mk/sys.mk 
/usr/src-topo/share/mk/local.sys.env.mk /usr/src-topo/share/mk/src.sys.env.mk 
/usr/src-topo/share/mk/bsd.mkopt.m
k /usr/src-topo/share/mk/src.sys.obj.mk /usr/src-topo/share/mk/auto.obj.mk 
/usr/src-topo/share/mk/bsd.suffixes.mk /usr/src-topo/share/mk/local.sys.mk 
/usr/src-topo/sha
re/mk/src.sys.mk /usr/src-topo/sys/modules/vmm/Makefile 
/usr/src-topo/share/mk/bsd.kmod.mk /usr/src/sys/conf/kmod.mk

^
Thats gona bust something some day

Global:.PARSEDIR = /usr/src/sys/conf
Global:.PARSEFILE = kmod.mk
Global:.INCLUDEDFROMDIR = /usr/src/sys/conf
Oh my!  Ug


So something in bsd.kmod.mk is going very wrong... it looks like it
starts to pull all sorts of stuff from /usr/src/sys!


I seem to remember that there is code in the Makefiles that looks for 
the sys directory.
I believe it can be directed to use a directory but in its absence it 
looks at some well
known locations, which would probably fail if there is already a 
DIFFERENT tree in /usr/src.



I have wrapped the long line so I can point to a difference between
r329294 and r332262 make log of this file.

r329294 make output:

cc  -O2 -pipe -DVMM_KEEP_STATS -DSMP  -fno-strict-aliasing -Werror -D_KERNEL \
-DKLD_MODULE -nostdinc  -I/usr/src-topo/sys/amd64/vmm \
-I/usr/src-topo/sys/amd64/vmm/io -I/usr/src-topo/sys/amd64/vmm/intel \
-I/usr/src-topo/sys/amd64/vmm/amd -I. -I/usr/src-topo/sys -fno-common  \
^ this is what I would 
expect




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

anyone running with ngroups increased from 16?

[Julian Elischer]

Windows users seem to have an almost unlimited number of groups and 
soem places seem to use them a LOT.
This gives Posix systems problems with deciding how to handle them 
all. Especially when getting

user credentials from winbindd (samba).

Does anyone know of any work done to either bypass this limit or to at 
least expand it?


Thanks

Julian

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: g_handleattr: md0 bio_length 24 len 31 -> EFAULT

[Julian Elischer]

On 8/4/18 10:48 pm, Kyle Evans wrote:

On Sun, Apr 8, 2018 at 5:53 AM, Peter Holm  wrote:

On Sun, Apr 08, 2018 at 02:36:08AM -0700, Michael Dexter wrote:

On 3/24/18 2:35 AM, O. Hartmann wrote:

Writing out memory (md) backed images of UFS2 filesystems (NanoBSD images, 
created via
the classical manual way, no makefs), my recent CURRENT system dumps the
console full of these error messages:

g_handleattr: md0 bio_length 24 len 31 -> EFAULT

I do not know what they are supposed to mean and I'd like to ask whether 
someone could
shed some light on this.

I am seeing this on the the latest snapshot when attempting to run
option_survey.sh which creates an md-attached disk image.

Anyone else seeing this?

Michael

Yes, I have:

[pho@freefall ~/public_html/stress/log]$ grep -a g_handleattr: `ls -rt`
numa025.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
numa025.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
numa025.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
numa025.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
numa025.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
numa025.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
numa025.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
numa025.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
numa025.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
kostik1104.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
kostik1104.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
kostik1104.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
kostik1104.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
kostik1104.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
kostik1104.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
kostik1104.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
kostik1104.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
kostik1104.txt:g_handleattr: md10 bio_length 24 len 31 -> EFAULT
[pho@freefall ~/public_html/stress/log]$


FYI- this should be fixed by r332070.

Thanks,

Kyle Evans
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


so I think current has other issues I want to avoid

can I just pull that commit and have have svn know how to upgrade past 
it later?



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

deadlock detected in -current (new).. NFS? Anyone with 'amd' knowledge?

[Julian Elischer]

Anyone seen these recently?



I just got the following:

panic: deadlkres: possible deadlock detected for (address)  blocked 
for (big number) ticks


anyone seen similar on very new (yesterday) -current GENERIC. ?
there was a prior message that may or may not be related:

newnfs: server pid741@gamma  error: fileid changed. fsid 0:0: expected 
fileid 0x1, got 0x2

(BROKEN NFS SERVER OR MIDDLEWARE)

(amd screwed up).

I suspect that amd is broken on -current
using known good config files results in a hung nfs mount
and even after killing amd, any attempt to touch the mountpoint 
results in a hung (but interruptable) proccess.
For example as /net  was an amd mountpoint,   ls -l /    results in a 
hung copy of ls.

(but you can get it back with ^C)

I suspect the two are related.

Julian


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

zfsloader: messy output on Dell iDrac serial console

[Julian Elischer]

running on *some* Dell servers the serial console looks messed up at 
the time of the FreeBSD menu.
it looks like it's dropping (or adding? but that's less likely) 
characters.


 |  5. [K]ernel: kernel (1 of 2)   |  .- .    -.
 |  6. Configure Boot [O]ptions... |2  --| y/   -.   
-/`   -o/
 |137. Select Boot [E]nvironment...    | `:`  
`:` ::/sy++:.
 | H| .-- `--. --  
/6H`:  :`

 | | .---..

IS there an RTS/CTS input we should be attending to?

I will add that some of the other modules (e.g. the mfi raid setup 
section) also get affected,

so it's not just a bug in our code.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

tzset (and friends) patch to watch /etc/localtime.. comments?

[Julian Elischer]

anyone interested in ctime, tzset, and localtime.c  please look 
athttps://reviews.freebsd.org/D14608


comments and improvements welcome.


Julian

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: A small procedural request

[Julian Elischer]

On 21/2/18 7:14 pm, Tomoaki AOKI wrote:

Hi.

+1. But have one suggestion for format.
Something like

  Broken by: rXXX
  Broken by: Unknown (Bugfix but the revision introduced it is unknown)

and optionally

  Broken by: No (To emphasize it's NOT a bugfix.)


I think that is probably too much, but the    Broken by:  would be 
good.


would be better for scripts already handling "MFC after: " or
"X-MFC-With: " etc. to support this.

If put on the top with "MFC rXX: Comments", it can be

  FIX rXX: Comments


possibly..
that Would allow some sort of collection of the data to  suggest good 
places to

retrospectively base your head following (but not too closely) branches.
but may be more work that people are willing to do..
For myself, just a hint of where the bug was introduced would help a lot.
further more if you have a branch/product based at some point in time, 
this would help

you to know when a patch needs to be cherry picked back to your code.


or for multiple revisions,

  FIX rXX rYY rZZ: Comments for multiple individuals
  FIX rXX-rYY: Comments for massive continuous range

would be better.

Regards.


On Wed, 21 Feb 2018 12:01:33 +0800
Julian Elischer <jul...@freebsd.org> wrote:


Hi,〓 I have a very small request to those committing into head.

If you commit a fix, then if it is possible to easily do so, can you
give the revision number in which the regression was introduced?

like "this was〓 broken in r329xxx"

this allows people who are looking for specific problems to say "Ok
that bug was introduced after the snapshot I'm working on and can't be
my issue".

(we are not always working on the very tip).


thanks

Julian


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"






___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

A small procedural request

[Julian Elischer]

Hi,  I have a very small request to those committing into head.

If you commit a fix, then if it is possible to easily do so, can you 
give the revision number in which the regression was introduced?


like "this was  broken in r329xxx"

this allows people who are looking for specific problems to say "Ok 
that bug was introduced after the snapshot I'm working on and can't be 
my issue".


(we are not always working on the very tip).


thanks

Julian


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: 12.0-CURRENT missing timezones

[Julian Elischer]

On 10/2/18 2:48 am, Warner Losh wrote:

OK. That makes sense again.


I've pushed two changes (r329064 and r329075) which should correct this.



thanks.. I was just about to go investigate this as I noticed my build 
last week had no timezone info.




Warner

On Fri, Feb 9, 2018 at 10:28 AM, David Boyd  wrote:


On Fri, 2018-02-09 at 10:16 -0700, Warner Losh wrote:



On Fri, Feb 9, 2018 at 10:13 AM, Warner Losh  wrote:



On Fri, Feb 9, 2018 at 9:56 AM, David Boyd  wrote:

In the most recent images for 12.0-CURRENT

 FreeBSD-12.0-CURRENT-amd64-20180208-r329009.vmdk.xz

 FreeBSD-12.0-CURRENT-amd64-20180208-r329009-disc1.iso

 FreeBSD-12.0-CURRENT-amd64-20180131-r328637-memstick.img

the /usr/share/zoneinfo directory is sparsely populated with
directories. The only file present is "zone.tab".


I think that may be my fault for changing find -s to find | sort, but I
think I neglected to add sort to ITOOLS to fix it. Building a test now.


I am surprised the 0131 is empty, though My change was after that.

Warner


Oops. My mistake (cut and paste error). All dates should be 20180208.


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Since last week (today) current on my Ryzen box is unstable

[Julian Elischer]

On 19/2/18 4:33 am, Gleb Smirnoff wrote:

On Sun, Feb 18, 2018 at 10:15:24PM +0200, Andriy Gapon wrote:
A> On 18/02/2018 15:26, Gleb Smirnoff wrote:
A> > My only point is that it is a performance improvement. IMHO that's enough 
:)
A>
A> I don't think that passing an invalid argument to a documented KPI is 
"enough"
A> for any optimization.

I don't see a sense in making this KPI so sacred. This is something used 
internally
in kernel, and not used outside. The KPI has changed several times in the past.

A> > If you can't suggest a more elegant way of doing that improvement, then all
A> > I can suggest is to document it and add its support to ZFS.
A>
A> In return I can only suggest that (1) you run your suggestion by arch@ -- 
unless
A> that's already been done and you can point me to the discussion,  (2) 
document
A> it and (3) double-check that all implementations confirm to it.

I can provide a patch for ZFS.



If any module outside of the code that implements it needs to know 
about it,

then it is in the KPI and should be documented in the KPI documentation
(e.g. man 9)


Since the Filesystems need to know about this, it must be an externally
visible feature and therefore needs to be documented.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: /usr/obj is 11GB huge on FreeBSD 12-current

[Julian Elischer]

On 16/12/17 2:39 am, Konstantin Belousov wrote:



Put the following into /etc/src.conf:


This brings up two questions:
when to use make.conf and when to use src.conf,

and..


WITHOUT_PROFILE=yes
WITHOUT_DEBUG_FILES=yes
WITHOUT_TESTS=yes

which of the following is correct and why?

WITH_DEBUG_FILES=no
WITHOUT_DEBUG_FILES=yes

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: RFC how to use kernel procs/threads efficiently

[Julian Elischer]

On 10/10/17 8:33 pm, Rick Macklem wrote:

Julian Elischer wrote:
[stuff snipped]

On 10/10/17 4:25 am, Rick Macklem wrote:

--> As such, having a fixed reasonable # of threads is probably the best
that can be done.
- The current patch has the # of threads as a sysctl with a default of 
32.

why not set it to ncpu or something?

Well, each of these threads will do an RPC, which means a couple of short
bursts of CPU and then sleep the rest of the time waiting for the RPC reply
to come back from the Data Server.
As such, it would seem to me that you would want a lot more threads than
CPUs on the machine?
However, setting the default to "N * ncpu" seems better than just a fixed "32"
to me. (For nfsd, the current default is 8 * ncpu, so maybe that is a good
default for this too?)
yeah I really just meant "some function of ncpu"..  not specifically 
"ncpu x 1"

What do you think?

Thanks for the comment, rick





___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: anyone know how to get rid of this error? (10.4/Gcc vs binutils port)

[Julian Elischer]

On 4/10/17 12:56 am, Ian Lepore wrote:

On Wed, 2017-10-04 at 00:18 +0800, Julian Elischer wrote:

Our 10.4 system is using gcc (for now).

when we compile the devel/binutils port, we get a failure with a
bunch
of these errors:


`_ZTSN12_GLOBAL__N_110Stub_tableILi64ELb1EEE' referenced in section
`.rodata' of aarch64.o: defined in discarded section
`.rodata._ZTSN12_GLOBAL__N_110Stub_tableILi64ELb1EEE[_ZTSN12_GLOBAL__
N_110Stub_tableILi64ELb1EEE]'
of aarch64.o
`_ZTSN12_GLOBAL__N_110Stub_tableILi64ELb0EEE' referenced in section
`.rodata' of aarch64.o: defined in discarded section
`.rodata._ZTSN12_GLOBAL__N_110Stub_tableILi64ELb0EEE[_ZTSN12_GLOBAL__
N_110Stub_tableILi64ELb0EEE]'
of aarch64.o


I managed to defeat these one before but I forget how.

possibly the answer is to use clang/clang++ for this item but I
tried
defining CC and CXX  to clang/clang++ in the Makefile but that
didn't
seem to help

there's probably a USE_CLANG option or something that I haven't seen.

We ran into the same thing recently at $work.  The root cause for us
involved having same-named classes in anonymous namespaces in different
compilation units.  The classes made reference to something that was
declared extern "C", bringing into play some rules about how C things
in anon namespaces really refer to the same global C object outside of
any namespace.  Then link-time optimization led to discarding the
object from one compilation unit, and that somehow resulted in
discarding the referenced extern "C" thing completely, even though it
was still referenced from the non-discarded instance of an object from
a different compilation unit.  Phew.

The same code has no problems with clang on freebsd 11, just with gcc
on 10.3.

So, for us the fix was a bit heavy-handed: we just renamed one of the
classes involved in the problem so that there were no longer any same-
named classes in anon namespaces in separate compilation units.
  Probably not a good option for you.  A fix involving compile options
might result in not discarding unreferenced segments at all, and with
templated code that might result in huge binaries.

Mixing clang-compiled and gcc-compiled c++ may give you grief with
exceptions and other things (it would on ARM on 10.x, but maybe not on
x86 arches).

-- Ian



thanks

the issue comes up in the binutils port which is a dependency of gdb.

I don't think we actually need to install the binutils port on our 
appliance, (and we only install gdb to generate backtraces on debug 
reports)


I just added CC=clang and CXX=clang++ in the makefile that called it 
and the problem seemed to go away.



All i wanted to do is get gdb compiled and I end up with gcc6, llvm 
and binutils (plus a whole lot more) as a bonus (plus an extra 30 
minutes of compile time)




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

anyone know how to get rid of this error? (10.4/Gcc vs binutils port)

[Julian Elischer]

Our 10.4 system is using gcc (for now).

when we compile the devel/binutils port, we get a failure with a bunch 
of these errors:



`_ZTSN12_GLOBAL__N_110Stub_tableILi64ELb1EEE' referenced in section 
`.rodata' of aarch64.o: defined in discarded section 
`.rodata._ZTSN12_GLOBAL__N_110Stub_tableILi64ELb1EEE[_ZTSN12_GLOBAL__N_110Stub_tableILi64ELb1EEE]' 
of aarch64.o
`_ZTSN12_GLOBAL__N_110Stub_tableILi64ELb0EEE' referenced in section 
`.rodata' of aarch64.o: defined in discarded section 
`.rodata._ZTSN12_GLOBAL__N_110Stub_tableILi64ELb0EEE[_ZTSN12_GLOBAL__N_110Stub_tableILi64ELb0EEE]' 
of aarch64.o



I managed to defeat these one before but I forget how.

possibly the answer is to use clang/clang++ for this item but I tried 
defining CC and CXX  to clang/clang++ in the Makefile but that didn't 
seem to help


there's probably a USE_CLANG option or something that I haven't seen.


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: extending the maximum filename length (pointer to patch)[request for input]

[Julian Elischer]

On 13/9/17 1:16 am, Jonathan Anderson wrote:

On 12 Sep 2017, at 14:38, Julian Elischer wrote:

“这是一个测试多字符文件名长度的文件目的是命名一个文件用中文或者日文或者韩文字符并且要求字符长度超过八十五个字符然后拷贝该文件到我们的共享文件夹看看是否能够拷贝该文件到我.txt” 

(I have no idea what that says but apparently it's a real filename 
from a windows machine that blew up when written via samba.)


Google Translate says, amusingly:
"This is a test file for the length of the file name. The purpose is 
to name a file in Chinese or Japanese or Korean characters and 
require the character to be longer than 85 characters and then copy 
the file into our shared folder to see if it can copy the file To 
me" (.txt, I guess)


No matter what number you choose for a path length, you're never 
going to win against that specific user. :)

ha!
it was give as an example to show us the limit.
Apparently however our company has difficulty selling in China due to 
this limit because Microsoft has a much larger limit and people use it.





Jon
--
Jonathan Anderson
jonat...@freebsd.org




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: extending the maximum filename length (pointer to patch)[request for input]

[Julian Elischer]

On 12/9/17 2:17 pm, Conrad Meyer wrote:

On Sat, Sep 9, 2017 at 9:09 AM, Julian Elischer <jul...@freebsd.org> wrote:

maybe we could get it into -current.
It'd be silly to have to have people re-inventing hte wheel all the time.
How about you put those changes into the reviews.freebsd.org and we can get
some general consensus on them.
We'll have to do similar for the Asian customers and anyone who uses UTF-8.
So it
would be silly to have to develop it all again (but subtly different of
course).

The key issue is how many system calls and other APIs would be broken,
and how many would be broken in a non backwards compatible way?

We would need it in a stable/10 and 11 branch but if the patch is isolated
enough we could carry it forward until we get to 12.

One has to allow people to do whatever they are used to with Windows.
And in this case the issue is serving files over samba to windows machines.

Hey Julian,

I've thrown the patch up at https://reviews.freebsd.org/D12330 .  I
haven't actually tested it on FreeBSD, but it does compile.  We also
have some patches against contrib/pjdfstest to fix those tests against
long file names, but I think we can hold off on those changes until
we've nailed down what the architectural change will be (if any).


thanks!

that looks a lot like a proof -of concept patch we derived a while 
back but never really tested.
The issue for us is that using UTF-8 the filenames become too short 
for common usage in China and Japan.

Apparently they routinely nae files with the contents of a small novella.

e.g.

“这是一个测试多字符文件名长度的文件目的是命名一个文件用中文或者日文或者韩文字符并且要求字符长度超过八十五个字符然后拷贝该文件到我们的共享文件夹看看是否能够拷贝该文件到我.txt”
(I have no idea what that says but apparently it's a real filename from a 
windows machine that blew up when written via samba.)

Does anyone else have any thoughts about whether FreeBSD 12 might grow longer 
path/filename support?
 (I'm told Linux uses 1K and 4096 for filenames and path length.)

Julian



It's quite possible this accidentally breaks even more APIs than
expected and we should do some fine tuning to reduce the damage.  Our
$WORK product mostly doesn't care about ABI, so we may not have
noticed some ABI breakage.

If anyone else is interested, please subscribe or add yourself as a
reviewer on the phabricator revision.

Best,
Conrad



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: extending the maximum filename length

[Julian Elischer]

On 9/9/17 1:28 am, Conrad Meyer wrote:

On Fri, Sep 8, 2017 at 10:15 AM, Julian Elischer <jul...@freebsd.org> wrote:

Has anyone using freeBSD ever increased NAME_MAX (filename maximum length)
and have any experience with it?

We ($JOB) would recompile the entire system so intra-system compatibility
would probably be ok, and we have our own filesystem which would support it.

But I wonder if anyone has tried it and hit unexpected problems.


reason:  Chinese and Japanese people who have gotten into the habit of
having a filename of 256 CHINESE/JAPANESE characters in Microsoft and want
to store their files on a FreeBSD based server witout having to rename
everything. (3 bytes for each of those characters giving a limit of 83
characters).

(though since each character is a word the names if I could read them must
be amazing)


NFS behaviour is one issue I don't know but would be interested in..  could
we SERVE such files?

Hey Julian,

We've done exactly this at Isilon.

Basically we bumped NAME_MAX and related constants out by 4x (maximum
length of a UTF-8 encoded Unicode character, in bytes).  So NAME_MAX
is 1023, PATH_MAX is 4096, and a bunch of follow-on equivalent/related
constants are bumped similarly.

To avoid breaking too many ABIs, we've retained a SHORT_NAME_MAX
constant of 255 which several non-filesystem NAME_MAX users were
converted to.  Stuff like module name APIs, procstat, etc.

Individual filesystems are free to implement and restrict maximum
component lengths in VOP_LOOKUP.  For us, we retained 255 bytes for
basically all filesystems (see e.g. r316509, r313475) aside from IFS
(the OneFS filesystem).  For IFS we check that component names are 255
("MAXNAMCHARS") unicode codepoints or fewer (not all names shorter
than NAME_MAX bytes are valid).

NFS commonly does not support >255 byte names, so we have a hack there
to export longer names as some shorter name plus a hash (total length
of 255 bytes or fewer) to uniquely identify the file.  SMB exports
longer names just fine.

Anyway, we'd love to shift some of these patches upstream, if there is
interest in supporting this kind of thing.

Sure, there are a few snags here and there and some ABIs change.  But
overall it seems to work pretty well.

Best,
Conrad


maybe we could get it into -current.
It'd be silly to have to have people re-inventing hte wheel all the time.
How about you put those changes into the reviews.freebsd.org and we 
can get some general consensus on them.
We'll have to do similar for the Asian customers and anyone who uses 
UTF-8. So it
would be silly to have to develop it all again (but subtly different 
of course).


The key issue is how many system calls and other APIs would be broken,
and how many would be broken in a non backwards compatible way?

We would need it in a stable/10 and 11 branch but if the patch is isolated
enough we could carry it forward until we get to 12.

One has to allow people to do whatever they are used to with Windows.
And in this case the issue is serving files over samba to windows 
machines.


Julian
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

extending the maximum filename length

[Julian Elischer]

Has anyone using freeBSD ever increased NAME_MAX (filename maximum 
length) and have any experience with it?


We ($JOB) would recompile the entire system so intra-system 
compatibility would probably be ok, and we have our own filesystem 
which would support it.


But I wonder if anyone has tried it and hit unexpected problems.


reason:  Chinese and Japanese people who have gotten into the habit of 
having a filename of 256 CHINESE/JAPANESE characters in Microsoft and 
want to store their files on a FreeBSD based server witout having to 
rename everything. (3 bytes for each of those characters giving a 
limit of 83 characters).


(though since each character is a word the names if I could read them 
must be amazing)



NFS behaviour is one issue I don't know but would be interested in..  
could we SERVE such files?



Julian


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: anyone had experience expanding uid_t and gid_t?

[Julian Elischer]

On 19/8/17 11:15 am, Julian Elischer wrote:

at $JOB there are clients where 32bits is starting to chafe.

Has anyone expanded them?

Other than a few offline comments I haven't heard anyone directly 
respond to this.

Does anyone have any comments on feasibility or suggestions?
NFSV3 will definitely be screwed..



This is starting to become a serious limitation in some places.

Especially large institutions with Samba active.

Samba uses a map between SIDs (session IDs) and UIDS, but it's a 
sparse map and due to various issues the mapping is not able to 
re-use numbers well.







___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

assigning priorities to swap partitions?

[Julian Elischer]

On an AZURE system there is a "local" device that is useful as swap.

It is, I believe, faster than regular network based storage, but it is 
ephemeral, and may go away during a shutdown.
It is in some machines a bit small so we'd like to add a bit more for 
safety.
But we would like the ephemeral local storage to be used first. Can 
this be done at all? I can't see anything


The last time I checked all swap was used in some balanced way. (the 
manual says so)
One solution is to have a small cron job that only creates the added 
swap partition when the first one is (say) 70% full,
but it'd be nice if there were some less hackish way. Another would be 
to occasionally wake up, and if the swap in use would all fit into the 
device we would like to be used, we do a swapoff on the other and 
force everything to be put on the fast drive..  but that sort of 
defeats the purpose as it's doing extra work..


Has anyone done any work on adding priority to swap?

Julian

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

mapping a device from device-id to /dev

[Julian Elischer]

I have the following (Azure) device (disk) id:

dev.storvsc..%pnpinfo: classid=32412632-86cb-44a2-9b5c-50d1417354f5 
deviceid=-0001-8899--

the question is:  "how can I map that do /dev/da1"..
I know that for my device it IS /dev/da1
but how can I prove it? there are so many mappings from one space to another 
I've totally lost track.
 there are acpi mappings, pnpmappngs, /dev/ mappings, PCI space mappings, 
things in sysctl, things in their own spaces, etc. etc.
In some architectures htere are fdt mappings and in some it's still pretty 
random from what I see.

Is there a document that covers all of these?



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

anyone had experience expanding uid_t and gid_t?

[Julian Elischer]

at $JOB there are clients where 32bits is starting to chafe.

Has anyone expanded them?


This is starting to become a serious limitation in some places.

Especially large institutions with Samba active.

Samba uses a map between SIDs (session IDs) and UIDS, but it's a 
sparse map and due to various issues the mapping is not able to re-use 
numbers well.





___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [Help] Linux low level data structures < - > FreeBSD low level data structures

[Julian Elischer]

On 4/6/17 7:07 pm, blubee blubeeme wrote:

Hello

Is there anyone on either of these lists that have experience with both
linux low level data structures and their equivalents on FreeBSD?

For instance the linux header file:


which includes the header file:


Then looking at that file:







You are going to have to be a lot more specific about this.
I have worked in several places where they use s shim layer to make 
Linux basic services work on freeBSD.

usually  a mix of functions, macros and inlines.
However you need to narrow down your questions a bit as the POSSIBLE 
scope of your question is too large for anyone to attempt an answer.


Remember that both systems are POSIX inspired so outside the kernel 
there are many more simlarities than one might be led to expect,

 but you need to be way more specific.
It's even possible to write kernel code to run on both, but it is 
usually domain specific.





I'll be doing a lot of work trying to find these FreeBSD equivalent of
these types of files to port some code.

Does anyone here have experience with something like this? Is there any
other projects that maps these low level data structures from
Linux <-> FreeBSD, etc?

Best,
Owen
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Time to increase MAXPHYS?

[Julian Elischer]

On 4/6/17 4:59 am, Colin Percival wrote:

On January 24, 1998, in what was later renumbered to SVN r32724, dyson@
wrote:

Add better support for larger I/O clusters, including larger physical
I/O.  The support is not mature yet, and some of the underlying implementation
needs help.  However, support does exist for IDE devices now.

and increased MAXPHYS from 64 kB to 128 kB.  Is it time to increase it again,
or do we need to wait at least two decades between changes?

This is hurting performance on some systems; in particular, EC2 "io1" disks
are optimized for 256 kB I/Os, EC2 "st1" (throughput optimized spinning rust)
disks are optimized for 1 MB I/Os, and Amazon's NFS service (EFS) recommends
using a maximum I/O size of 1 MB (and despite NFS not being *physical* I/O it
seems to still be limited by MAXPHYS).


We increase it in freebsd 8 and 10.3 on our systems,  Only good results.

sys/sys/param.h:#define MAXPHYS (1024 * 1024)   /* max raw I/O 
transfer size */


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Ssh.. can we please have HPN back?

[Julian Elischer]

On 23/5/17 12:13 am, Allan Jude wrote:

On 2017-05-22 03:50, Julian Elischer wrote:

On 22/5/17 2:20 pm, Allan Jude wrote:

On 2017-05-18 22:28, Julian Elischer wrote:

So after stripping out the HPN version of ssh from our product becasue
"it was no longer needed" we dicovered that we were premature in
doing so.
Apparently ssh still really needs HPN to get any throughput at all when
there are latencies involved.


For example, with HPN we get 13MB/sec between the Azure US west
Data center and the Azure East data center.But the standard ssh in 10.3
(with HPN stripped out) can barely manage 2MB/sec transfers.

I did ask at the time whether it was proved that the new ssh didn't
require the HPN changes,
and was assured, "no" but it would appear that the picture isn't as
clear.
tht seems silly to have to import the port when we have what would
otherwise be a
perfectly good ssh as part of hte system, and it's really annoying
having to specify
/usr/local/bin/scp  or /usr/local/bin/ssh in every script.

So can we please have the latest version of the HPN changes back in the
default system please?
It seem rather odd that the upstream openssh has had this problem for SO
LONG and not fixed it.

Julian


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to
"freebsd-current-unsubscr...@freebsd.org"

I have this stand-alone patch ready now:

https://github.com/openssh/openssh-portable/compare/master...allanjude:V_7_5_dynamic_window


In my benchmarks with 100ms of latency (from dummynet) is increases SSH
send throughput from 1 megabyte/sec to 225 megabytes/sec provided a
large enough socket buffer.

Still seeing lesser performance on the recv case, working on it.

We have two patches of our own that upstream have ignored   an option to
set NoDelay   and a pushwindow setting option
(though I'm not sure the second is operational in the current code, it
does apply with no errors...)

The NoDelay options massively speeds up the case where you have chatty
back and forth traffic (client/server) through a ssh tunnel.

Are your changes against the sources in current?  what about stable/10?


That change is against upstream's latest release, but should apply
cleanly to any version of OpenSSH from the past 10 years that does not
have the HPN patches applied already.

However, the function channel_check_window() in stable/10 is currently
the same as the latest upstream since the commit that removed HPN. The
function is actually unchanged from upstream if you go back far enough
to before we added HPN as well.

If you revert both HPN commits, the most recent change to that function
is Aug 2008, when upstream introduced the 'move the window forward every
3 packets' thing that seams to be the main cause of the window scaling
problem in the first place. I think the nodelay option might be what
mitigates that, as it causes rapid back and forth and never allows the
window to grow to even the 2mb allowed by SSH since version 4.7



great

I'll look at applying it at $JOB and let you know what happens .. 
assuming I can get the patch downloaded and applied.



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The futur of the roff toolchain

[Julian Elischer]

On 21/5/17 8:57 pm, Baptiste Daroussin wrote:

Hi all,


[...]


No the problem left is documentations available in share/doc.

I would like to push them elsewhere. Those documents are mostly useful for
historical reason (hence we want to keep them) but not really for daily use of
modern FreeBSD.
Another issue with those documentation, they are installed as text/ascii version
in base, which makes most of them not really readable (as the documents has not
be written for a ascii/text target but more for a PDF/html view - using pic(1)
for example)


I was surprised that they stayed in src when we split it up.
And really they should be supplemented by every FreeBSD based paper 
presented at a conference :-)




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Ssh.. can we please have HPN back?

[Julian Elischer]

On 22/5/17 2:20 pm, Allan Jude wrote:

On 2017-05-18 22:28, Julian Elischer wrote:

So after stripping out the HPN version of ssh from our product becasue
"it was no longer needed" we dicovered that we were premature in doing so.
Apparently ssh still really needs HPN to get any throughput at all when
there are latencies involved.


For example, with HPN we get 13MB/sec between the Azure US west
Data center and the Azure East data center.But the standard ssh in 10.3
(with HPN stripped out) can barely manage 2MB/sec transfers.

I did ask at the time whether it was proved that the new ssh didn't
require the HPN changes,
and was assured, "no" but it would appear that the picture isn't as clear.
tht seems silly to have to import the port when we have what would
otherwise be a
perfectly good ssh as part of hte system, and it's really annoying
having to specify
/usr/local/bin/scp  or /usr/local/bin/ssh in every script.

So can we please have the latest version of the HPN changes back in the
default system please?
It seem rather odd that the upstream openssh has had this problem for SO
LONG and not fixed it.

Julian


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

I have this stand-alone patch ready now:

https://github.com/openssh/openssh-portable/compare/master...allanjude:V_7_5_dynamic_window

In my benchmarks with 100ms of latency (from dummynet) is increases SSH
send throughput from 1 megabyte/sec to 225 megabytes/sec provided a
large enough socket buffer.

Still seeing lesser performance on the recv case, working on it.


We have two patches of our own that upstream have ignored   an option 
to set NoDelay   and a pushwindow setting option
(though I'm not sure the second is operational in the current code, it 
does apply with no errors...)


The NoDelay options massively speeds up the case where you have chatty 
back and forth traffic (client/server) through a ssh tunnel.


Are your changes against the sources in current?  what about stable/10?





___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Ssh.. can we please have HPN back?

[Julian Elischer]

So after stripping out the HPN version of ssh from our product becasue
"it was no longer needed" we dicovered that we were premature in doing so.
Apparently ssh still really needs HPN to get any throughput at all when
there are latencies involved.


For example, with HPN we get 13MB/sec between the Azure US west
Data center and the Azure East data center.But the standard ssh in 10.3
(with HPN stripped out) can barely manage 2MB/sec transfers.

I did ask at the time whether it was proved that the new ssh didn't 
require the HPN changes,

and was assured, "no" but it would appear that the picture isn't as clear.
tht seems silly to have to import the port when we have what would 
otherwise be a
perfectly good ssh as part of hte system, and it's really annoying 
having to specify

/usr/local/bin/scp  or /usr/local/bin/ssh in every script.

So can we please have the latest version of the HPN changes back in 
the default system please?
It seem rather odd that the upstream openssh has had this problem for 
SO LONG and not fixed it.


Julian


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: bootcode capable of booting both UFS and ZFS? (Amazon/ec2)

[Julian Elischer]

On 7/5/17 1:45 pm, Warner Losh wrote:

On Sat, May 6, 2017 at 10:03 PM, Julian Elischer <jul...@freebsd.org> wrote:

On 6/5/17 4:01 am, Toomas Soome wrote:



On 5. mai 2017, at 22:07, Julian Elischer <jul...@freebsd.org
<mailto:jul...@freebsd.org>> wrote:

Subject says it all really, is this an option at this time?

we'd like to try boot the main zfs root partition and then fall back to a
small UFS based recovery partition.. is that possible?

I know we could use grub but I'd prefer keep it in the family.





it is, sure. but there is an compromise to be made for it.

Lets start with what I have done in illumos port, as the idea there is
exactly about having as “universal” binaries as possible (just the binaries
are listed below to get the size):

-r-xr-xr-x   1 root sys   171008 apr 30 19:55 bootia32.efi
-r-xr-xr-x   1 root sys   148992 apr 30 19:55 bootx64.efi
-r--r--r--   1 root sys 1255 okt 25  2015 cdboot
-r--r--r--   1 root sys   154112 apr 30 19:55 gptzfsboot
-r-xr-xr-x   1 root sys   482293 mai  2 21:10 loader32.efi
-r-xr-xr-x   1 root sys   499218 mai  2 21:10 loader64.efi
-r--r--r--   1 root sys  512 okt 15  2015 pmbr
-r--r--r--   1 root sys   377344 mai  2 21:10 pxeboot
-r--r--r--   1 root sys   376832 mai  2 21:10 zfsloader

the loader (bios/efi) is built with full complement - zfs, ufs, dosfs,
cd9660, nfs, tftp + gzipfs. The cdboot is starting zfsloader (thats trivial
string change).

The gptzfsboot in illumos case is only built with zfs, dosfs and ufs - as
it has to support only disk based media to read out the loader. Also I am
building gptzfsboot with libstand and libi386 to get as much shared code as
possible - which has both good and bad sides, as usual;)

The gptzfsboot size means that with ufs the dedicated boot partition is
needed (freebsd-boot), with zfs the illumos port is always using the 3.5MB
boot area after first 2 labels (as there is no geli, the illumos does not
need dedicated boot partition with zfs).

As the freebsd-boot is currently created 512k, the size is not an issue.
Also using common code does allow the generic partition code to be used, so
GPT/MBR/BSD (VTOC in illumos case) labels are not problem.


So, even just with cd boot (iso), starting zfsloader (which in fbsd has
built in ufs, zfs etc), you already can get rescue capability.

Now, even with just adding ufs reader to gptzfsboot, we can use gpt +
freebsd-boot and ufs root but loading zfsloader on usb image, so it can be
used for both live/install and rescue, because zfsloader itself has support
for all file systems + partition types.

I have kept myself a bit off from freebsd gptzfsboot because of simple
reason - the older setups have smaller size for freebsd boot, and not
everyone is necessarily happy about size changes:D also in freebsd case
there is another factor called geli - it most certainly does contribute some
bits, but also needs to be properly addressed on IO call stack (as we have
seen with zfsbootcfg bits). But then again, here also the shared code can
help to reduce the complexity.

Yea, the zfsloader/loader*.efi in that listing above is actually built
with framebuffer code and compiled in 8x16 default font (lz4 compressed
ascii+boxdrawing basically - because zfs has lz4, the decompressor is always
there), and ficl 4.1, so thats a bit of difference from fbsd loader.

Also note that we can still build the smaller dedicated blocks like boot2,
just that we can not use those blocks for more universal cases and
eventually those special cases will diminish.


thanks for that..

  so, here's my exact problem I need to solve.
FreeBSD 10 (or newer) on Amazon EC2.
We need to have a plan for recovering the scenario where somethign goes
wrong (e.g. during an upgrade) and we are left with a system where the
default zpool rootfs points to a dataset that doesn't boot. It is possible
that mabe the entire pool is unbootable into multi-user..  Maybe somehow it
filled up? who knows. It's hard to predict future problems.
There is no console access at all so there is no possibility of human
intervention. So all recovery paths that start "enter single user mode
and" are unusable.

The customers who own the amazon account are not crazy about giving us the
keys to the kingdom as far as all their EC2 instances, so taking a root
drive off a 'sick' VM and grafting it onto a freebsd instance to 'repair' it
becomes a task we don't want to really have to ask them to do. They may not
have the in-house expertise to do it. confidently.

This leaves us with automatic recovery, or at least automatic methods of
getting access to that drive from the network.
Since the regular root is zfs, my gut feeling is that to deduce the chances
of confusion during recovery, I'd like the (recovery) system itself to be
running off a UFS partition, and potentially, with a memory root filesystem.
As long as it can be reached over the n

Re: bootcode capable of booting both UFS and ZFS? (Amazon/ec2)

[Julian Elischer]

On 6/5/17 4:01 am, Toomas Soome wrote:


On 5. mai 2017, at 22:07, Julian Elischer <jul...@freebsd.org 
<mailto:jul...@freebsd.org>> wrote:


Subject says it all really, is this an option at this time?

we'd like to try boot the main zfs root partition and then fall 
back to a small UFS based recovery partition.. is that possible?


I know we could use grub but I'd prefer keep it in the family.






it is, sure. but there is an compromise to be made for it.

Lets start with what I have done in illumos port, as the idea there 
is exactly about having as “universal” binaries as possible (just 
the binaries are listed below to get the size):


-r-xr-xr-x   1 root sys   171008 apr 30 19:55 bootia32.efi
-r-xr-xr-x   1 root sys   148992 apr 30 19:55 bootx64.efi
-r--r--r--   1 root sys 1255 okt 25  2015 cdboot
-r--r--r--   1 root sys   154112 apr 30 19:55 gptzfsboot
-r-xr-xr-x   1 root sys   482293 mai  2 21:10 loader32.efi
-r-xr-xr-x   1 root sys   499218 mai  2 21:10 loader64.efi
-r--r--r--   1 root sys  512 okt 15  2015 pmbr
-r--r--r--   1 root sys   377344 mai  2 21:10 pxeboot
-r--r--r--   1 root sys   376832 mai  2 21:10 zfsloader

the loader (bios/efi) is built with full complement - zfs, ufs, 
dosfs, cd9660, nfs, tftp + gzipfs. The cdboot is starting zfsloader 
(thats trivial string change).


The gptzfsboot in illumos case is only built with zfs, dosfs and ufs 
- as it has to support only disk based media to read out the loader. 
Also I am building gptzfsboot with libstand and libi386 to get as 
much shared code as possible - which has both good and bad sides, as 
usual;)


The gptzfsboot size means that with ufs the dedicated boot partition 
is needed (freebsd-boot), with zfs the illumos port is always using 
the 3.5MB boot area after first 2 labels (as there is no geli, the 
illumos does not need dedicated boot partition with zfs).


As the freebsd-boot is currently created 512k, the size is not an 
issue. Also using common code does allow the generic partition code 
to be used, so GPT/MBR/BSD (VTOC in illumos case) labels are not 
problem.



So, even just with cd boot (iso), starting zfsloader (which in fbsd 
has built in ufs, zfs etc), you already can get rescue capability.


Now, even with just adding ufs reader to gptzfsboot, we can use gpt 
+ freebsd-boot and ufs root but loading zfsloader on usb image, so 
it can be used for both live/install and rescue, because zfsloader 
itself has support for all file systems + partition types.


I have kept myself a bit off from freebsd gptzfsboot because of 
simple reason - the older setups have smaller size for freebsd boot, 
and not everyone is necessarily happy about size changes:D also in 
freebsd case there is another factor called geli - it most certainly 
does contribute some bits, but also needs to be properly addressed 
on IO call stack (as we have seen with zfsbootcfg bits). But then 
again, here also the shared code can help to reduce the complexity.


Yea, the zfsloader/loader*.efi in that listing above is actually 
built with framebuffer code and compiled in 8x16 default font (lz4 
compressed ascii+boxdrawing basically - because zfs has lz4, the 
decompressor is always there), and ficl 4.1, so thats a bit of 
difference from fbsd loader.


Also note that we can still build the smaller dedicated blocks like 
boot2, just that we can not use those blocks for more universal 
cases and eventually those special cases will diminish.


thanks for that..

 so, here's my exact problem I need to solve.
FreeBSD 10 (or newer) on Amazon EC2.
We need to have a plan for recovering the scenario where somethign 
goes wrong (e.g. during an upgrade) and we are left with a system 
where the default zpool rootfs points to a dataset that doesn't boot. 
It is possible that mabe the entire pool is unbootable into 
multi-user..  Maybe somehow it filled up? who knows. It's hard to 
predict future problems.
There is no console access at all so there is no possibility of human 
intervention. So all recovery paths that start "enter single user mode 
and" are unusable.


The customers who own the amazon account are not crazy about giving us 
the keys to the kingdom as far as all their EC2 instances, so taking a 
root drive off a 'sick' VM and grafting it onto a freebsd instance to 
'repair' it becomes a task we don't want to really have to ask them to 
do. They may not have the in-house expertise to do it. confidently.


This leaves us with automatic recovery, or at least automatic methods 
of getting access to that drive from the network.
Since the regular root is zfs, my gut feeling is that to deduce the 
chances of confusion during recovery, I'd like the (recovery) system 
itself to be running off a UFS partition, and potentially, with a 
memory root filesystem. As long as it can be reached over the network 
we can then take over.


we'd also like to have the bo

bootcode capable of booting both UFS and ZFS?

[Julian Elischer]

Subject says it all really, is this an option at this time?

we'd like to try boot the main zfs root partition and then fall back 
to a small UFS based recovery partition.. is that possible?


I know we could use grub but I'd prefer keep it in the family.



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: process killed: text file modification

[Julian Elischer]

On 13/4/17 5:45 am, Rick Macklem wrote:

I have just committed a patch to head (r316745) which should fix this.
(It includes code to handle the recent change to head to make the pageouts
  write through the buffer cache.)

It will be MFC'd and should be in 11.1.


is there any relevance of this change to stable/10?



Thanks everyone, for your help with this, rick

From: owner-freebsd-curr...@freebsd.org  on behalf 
of Rick Macklem 
Sent: Friday, March 24, 2017 4:14:45 PM
To: Konstantin Belousov
Cc: Gergely Czuczy; Dimitry Andric; Ian Lepore; FreeBSD Current
Subject: Re: process killed: text file modification

I can't do commits until I get home in mid-April.

That's why it will be waiting until then.

It should make it into stable/11 in plenty of time for 11.1.

Thanks for your help with this, rick

From: owner-freebsd-curr...@freebsd.org  on behalf 
of Konstantin Belousov 
Sent: Friday, March 24, 2017 3:01:41 AM
To: Rick Macklem
Cc: Gergely Czuczy; Dimitry Andric; Ian Lepore; FreeBSD Current
Subject: Re: process killed: text file modification

On Thu, Mar 23, 2017 at 09:39:00PM +, Rick Macklem wrote:

Try whatever you like. However, if the case that failed before doesn't fail now,
I'd call the test a success.

Thanks, rick
ps: It looks like Kostik is going to work on converting the NFS vop_putpages() 
to
   using the buffer cache. However, if this isn't ready for head/current by 
mid-April,
   I will commit this patch to help fix things in the meantime.

I do not see a reason to wait for my work before committing your patch.
IMO, instead, it should be committed ASAP and merged into stable/11 for
upcoming 11.1.



I will do required adjustments if/when _putpages() patch progresses enough.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: effect of strip(1) on du(1)

[Julian Elischer]

On 3/3/17 8:31 am, Rodney W. Grimes wrote:

On Fri, Mar 3, 2017 at 2:04 AM, Peter Jeremy  wrote:

On 2017-Mar-02 22:29:46 +0300, Subbsd  wrote:

During some interval after strip call, du will show 512B for any file.
If execute du(1) after strip(1) without delay, this behavior is reproduced 100%:

What filesystem are you using?  strip(1) rewrites the target file and du(1)
reports the number of blocks reported by stat(2).  It seems that you are
hitting a situation where the file metadata isn't immediately updated.

--
Peter Jeremy


Got it. My filesystem is ZFS. Looks like when ZFS open and write data
to file, we get wrong number of blocks during a small interval after
writing. Thanks for pointing this out!

Even if that is the case file system cache effects should NOT be
visible to a userland process.   This is NOT as if your running
2 different processing beating on a file.  Your test cases are
serialially syncronous shell invoked commands seperated with
&& the results should be exact and predictable.

When strip returns the operation from the userland perspecive
is completed and any and all processeses started after that
should have the view of the completed strip command.

This IS a bug.


actually it's all in how you look at it.
Due to the way ZFS is doing the work and the metadata transitions, 
that amount of storage is actually directly attributable to that 
file's existence.

so from that perspective the du is correct.


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Fwd: Re: I/O semantics of pipe and FIFO.

[Julian Elischer]

an interesting point to discuss? is our behaviour in this test right?
   from: "austin-group mailng list (posix standard discussion)"

-- rest of email is quoted ---
On 5/3/17 5:48 am, Stephane Chazelas wrote:

2017-03-04 13:14:08 +, Danny Niu:

Hi all.

I couldn't remember where I saw it saying, that when reading
from a pipe or a FIFO, the read syscall returns the content of
at most one write call. It's a bit similar to the
message-nondiscard semantics of dear old STREAM.

Currently, I'm reading through the text to find out a bit
more, and I appreciate a bit of pointer on this.

[...]

(echo x; echo y) | (sleep 1; dd count=1 2> /dev/null)

outputs both x and y in all of Linux, FreeBSD and Solaris in my
tests.

That a read wouldn't read what's currently in the pipe would be
quite surprising.

I also wouldn't expect pipes to store the writes as individual
separate message but use one buffer.

In:

(
  dd bs=4 count=1 if=/dev/zero 2> /dev/null
  echo first through >&2
  dd bs=4 count=1 if=/dev/zero 2> /dev/null
  echo second through >&2
) | (sleep 1; dd bs=10 count=1 2> /dev/null) | wc -c

That is where the second write blocks because the pipe is full,
the reading dd still reads both writes in Linux and Solaris in
my tests (on Solaris (10 on amd64 at least), reduce to 2
instead of 4 or both writes would block).

On FreeBSD, I get only the first write (using 8000 followed by
1 for instance).

FreeBSD is also the only one of the three where

dd bs=100 count=1 if=/dev/zero | dd bs=100 count=1 | wc -c

Doesn't output 100. The others schedule both processes back
and forth during their write() and read() system call while the
pipe is being filled and emptied several times.

--
Stephane

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: CFLAGS for certain ports

[Julian Elischer]

On 2/3/17 8:58 pm, Dimitry Andric wrote:

On 2 Mar 2017, at 12:02, Mingo Rrubioer  wrote:

I would like to see how well FreeBSD does as a workstation OS in the
HPC world due to its stability and reliability, as well as LLVM/clang.
I would like to know if FreeBSD has something similar to Gentoo's
/etc/portage/make.conf file and /etc/portage/package.use/* files in
order to compile certain ports with certain compiler flags.

It doesn't, though it would certainly be nice to have something like it
at some point.  The current idiom is to put something similar to the
following in your /etc/make.conf:

.if ${.CURDIR:M/usr/ports/foo/bar}
CFLAGS+= [... flags for the foo/bar port ...]
.endif

.if ${.CURDIR:M/usr/ports/what/ever}
CFLAGS+= [... flags for the what/ever port ...]
.endif



Regarding LLVM/clang, I've been reading the documentation and found
these flags: -arch=, -march=, -mcpu=,
--target=, target-cpu . I'm not quite sure which
one would be the one to use. In case someone wants to know, my initial
play/test machine has this processor:

CPU: Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz (3600.11-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206d7  Family=0x6  Model=0x2d  Stepping=7

And I'm currently running: 11.0-RELEASE-p8.

So I imagine I should use something like CFLAGS+= -march=corei7-avx
-march=sandybridge -target-cpu. Is that correct?

Don't specify -march or -mcpu directly, but add the following line to
/etc/make.conf:

CPUTYPE?= native

This will take care of everything automatically.  See also make.conf(5).

-Dimitry

Many many 3rd party packages use a consistent set of variables fed 
into the automumble tools.
Only today I was pleasantly surprised to see the lftp port pick up my 
CFLAGS and LDFLAGS changes without me having to change anything in the 
ports themselves..


we should investigate what these defacto stndards are and document them


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: confusing KTR_SCHED traces

[Julian Elischer]

On 18/2/17 2:48 am, Andriy Gapon wrote:

First, an example, three consecutive entries for the same thread (from top to
bottom):
KTRGRAPH group:"thread", id:"zio_write_intr_3 tid 100260", state:"sleep",
attributes: prio:84, wmesg:"-", lockname:"(null)"
KTRGRAPH group:"thread", id:"zio_write_intr_3 tid 100260", state:"spinning",
attributes: lockname:"sched lock 1"
KTRGRAPH group:"thread", id:"zio_write_intr_3 tid 100260", state:"running",
attributes: none

Any automatic analysis tool including schedgraph.py will assume that the thread
ends up in the running state.  In reality, of course, the thread is in the
sleeping state.
The confusing trace is a result of logging the thread's intention to switch out
in mi_switch() before calling sched_switch().  In ULE's sched_switch() we
acquire the "TDQ_LOCK" which could be contested.  In that case the thread spins
waiting for the lock to be released.  This is reported as "spinning" and then
"running" states.

I would like to fix that, but not sure how to do that best.
One idea is to move the mi_switch() trace closer to the cpu_switch() call
similarly to DTrace sched:cpu-off and sched:cpu-on probes.


I think that is the way to fix it


Any suggestions are welcome.
Thanks!



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [IGNORE] ldd linker script /usr/lib/libc.so fail [IGNORE]

[Julian Elischer]

Tracked this down to a rogue copy of libc.so in an unexpected place 
which was being found earlier than the real one.



On 30/1/17 1:13 am, Julian Elischer wrote:

Hi

the linker script /usr/lib/libc.so fails when you are using the 
--sysroot options because it


contains absolute paths.


Does anyone know if there is a way to add the sysroot to the script?

currently teh on ein our sysroot looks like:

$ cat /usr/build/buildroot/tools/x86_FBSD1X_gcc4.2.4/usr/lib/libc.so
/* $FreeBSD$ */
GROUP ( /lib/libc.so.7 /usr/lib/libc_nonshared.a 
/usr/lib/libssp_nonshared.a )


but I'd like to do something like:

GROUP ( ${sysroot}/lib/libc.so.7 ${sysroot}/usr/lib/libc_nonshared.a 
${sysroot}/usr/lib/libssp_nonshared.a )


but don't think I can do that

from what I see below however it shouldn't be needed.

Is this a bug in our version of ld? or am I misreading it?


I quote from one such source :

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Using_ld_the_GNU_Linker/simple-commands.html 




INPUT(file, file, …), INPUT(file file …)

   The INPUT command directs the linker to include the named files in
   the link, as though they were named on the command line.

   For example, if you always want to include subr.o any time you do
   a link, but you can't be bothered to put it on every link command
   line, then you can put INPUT (subr.o) in your linker script.

   In fact, if you like, you can list all of your input files in the
   linker script, and then invoke the linker with nothing but a -T
   option.

   In case a /sysroot prefix/ is configured, and the filename starts
   with the / character, and the script being processed was located
   inside the /sysroot prefix/, the filename will be looked for in
   the /sysroot prefix/. Otherwise, the linker will try to open the
   file in the current directory. If it is not found, the linker will
   search through the archive library search path. See the
   description of -L in Section 3.1 /Command Line Options/
<https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Using_ld_the_GNU_Linker/invocation.html#OPTIONS>.


   If you use INPUT (-lfile), ld will transform the name to
   libfile.a, as with the command line argument -l.

   When you use the INPUT command in an implicit linker script, the
   files will be included in the link at the point at which the
   linker script file is included. This can affect archive searching.

GROUP(file, file, …), GROUP(file file …)

   The GROUP command is like INPUT, except that the named files
   should all be archives, and they are searched repeatedly until no
   new undefined references are created.

   =


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 
"freebsd-current-unsubscr...@freebsd.org"





___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

ldd linker script /usr/lib/libc.so fail

[Julian Elischer]

Hi

the linker script /usr/lib/libc.so fails when you are using the 
--sysroot options because it


contains absolute paths.


Does anyone know if there is a way to add the sysroot to the script?

currently teh on ein our sysroot looks like:

$ cat /usr/build/buildroot/tools/x86_FBSD1X_gcc4.2.4/usr/lib/libc.so
/* $FreeBSD$ */
GROUP ( /lib/libc.so.7 /usr/lib/libc_nonshared.a 
/usr/lib/libssp_nonshared.a )


but I'd like to do something like:

GROUP ( ${sysroot}/lib/libc.so.7 ${sysroot}/usr/lib/libc_nonshared.a 
${sysroot}/usr/lib/libssp_nonshared.a )


but don't think I can do that

from what I see below however it shouldn't be needed.

Is this a bug in our version of ld? or am I misreading it?


I quote from one such source :

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Using_ld_the_GNU_Linker/simple-commands.html


INPUT(file, file, …), INPUT(file file …)

   The INPUT command directs the linker to include the named files in
   the link, as though they were named on the command line.

   For example, if you always want to include subr.o any time you do
   a link, but you can't be bothered to put it on every link command
   line, then you can put INPUT (subr.o) in your linker script.

   In fact, if you like, you can list all of your input files in the
   linker script, and then invoke the linker with nothing but a -T
   option.

   In case a /sysroot prefix/ is configured, and the filename starts
   with the / character, and the script being processed was located
   inside the /sysroot prefix/, the filename will be looked for in
   the /sysroot prefix/. Otherwise, the linker will try to open the
   file in the current directory. If it is not found, the linker will
   search through the archive library search path. See the
   description of -L in Section 3.1 /Command Line Options/
   
.


   If you use INPUT (-lfile), ld will transform the name to
   libfile.a, as with the command line argument -l.

   When you use the INPUT command in an implicit linker script, the
   files will be included in the link at the point at which the
   linker script file is included. This can affect archive searching.

GROUP(file, file, …), GROUP(file file …)

   The GROUP command is like INPUT, except that the named files
   should all be archives, and they are searched repeatedly until no
   new undefined references are created.

   =


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: gptzfsboot grew a lot after skein support was added; need knob to control bloat

[Julian Elischer]

On 28/1/17 1:35 am, Allan Jude wrote:

On 2017-01-27 12:33, Shawn Webb wrote:

On Fri, Jan 27, 2017 at 12:30:17PM -0500, Allan Jude wrote:

On 2017-01-27 12:05, Warner Losh wrote:

On Fri, Jan 27, 2017 at 12:34 AM, Toomas Soome  wrote:

On 27. jaan 2017, at 1:40, Ngie Cooper (yaneurabeya)  
wrote:

Hi,
   I tried upgrading one of my workstations and unfortunately the 
freebsd-boot partition is too small (I follow manpage directions, exactly, and 
those seem to be too small as of 10.3-RELEASE timeframe), and I don???t have 
enough space or ability to resize the partition and make it bigger. So, I???m 
in need of a build knob to control the bloat, and/or having an alternative boot 
loader without geli/skein/crypto support compiled in. Would you be opposed to 
the work?
Thanks,
-Ngie


I do agree that since the geli knob is already there, it may do. Of course we 
also can think of additional knobs, but there is an issue - it wont help just 
to exclude some files, the additional features also do sit in the code, so the 
replacement stubs will be needed, also testing them all over will take some 
time. And the preprocessor spaghetti really is nasty thing to deal with;)

And then there is another issue (partly why I did the feature support in first 
place) - as the kernel does not block user from enabling the features, the user 
can end up facing non-bootable setup which is also not good, as user is using 
perfectly legal options, and still the whole thing is just rendered unusable???

I'm curious why you can't find the space for a bigger partition?
Almost all drives these days are partitioned with a little wasted
space, and that wasted space should be more than enough to cover us
here. Also, most drives have a swap partition that can be shrunk a
trivial amount to get space for this...

Warner


I need to do some testing to make a recipe that works for it, but the
other option is to use the ZFS bootcode area.

ZFS it self, reserves something like 3.5 mb of space in the ZFS
partition, for boot code. This is how we boot ZFS on MBR.

It should be possible to use this on GPT as well, we just don't.

In the future, maybe it'd be a good idea for the installer to leave
more space (a few MB, perhaps?) between the freebsd-boot and
freebsd-swap partitions? At least, for ZFS installs.

Thanks,


The PMBR code has a limitation for 536kb, and it all has to fit under
the 640k barrier, so the current 512kb size is plenty. The issue is some
people are upgrading from systems that were isntalled long ago, when
64kb or less was the default.

with 512K we can append a copy of FreeBSD1.0 on the end of the 
bootblock and leave us the option of bringing up a networked NFS based 
system for debugging.


maybe we could fall back to that after a crash...



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: gptzfsboot grew a lot after skein support was added; need knob to control bloat

[Julian Elischer]

On 28/1/17 4:16 am, Ngie Cooper wrote:

On Jan 27, 2017, at 09:05, Warner Losh  wrote:

...


I'm curious why you can't find the space for a bigger partition?
Almost all drives these days are partitioned with a little wasted
space, and that wasted space should be more than enough to cover us
here. Also, most drives have a swap partition that can be shrunk a
trivial amount to get space for this...

Unfortunately, in my infinite wisdom (IIRC) I put the zfs partition before the 
swap partition.

We have a similar problem at work with sys/boot unfortunately, but that's a 
side discussion for another time/place.

Thank you for the idea though -- I'll check when I get back to work.


at $JOB we are just testing a script that expands the root zfs 
partition on in-field appliances by shaving a bit off swap and 
cannibalising a small data partition we don't really use. I see we 
only left 64K for the boot part. It's big enough for us for now, but 
possibly we should fix that as well.
We have a mirror setup for system disks so we have the ability to take 
each system drive offline one at a time and rearrange it and then 
re-add the root partition to the mirror.
What are the chances a regular gpt+ZFS (no encrypt) bootblock will 
grow over 64K?





-Ngie
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: r312602: panic: Panic String: __lockmgr_args: unknown lockmgr request 0x0

[Julian Elischer]

On 22/01/2017 4:51 AM, O. Hartmann wrote:

Am Sat, 21 Jan 2017 21:13:49 +0100
Mateusz Guzik  schrieb:


On Sat, Jan 21, 2017 at 08:45:55PM +0100, O. Hartmann wrote:

The most recent CURRENT panics spontanously and crashes with the error message:

  
Panic String: __lockmgr_args: unknown lockmgr request 0x0
   

That's a braino in r312600, will be fixed shortly.


??? What is a "braino"?


it's like a typo but more in the head than in the fingers.

The sort of error you make when your wife/girlfriend/boss/kids 
distract you right in the middle of some bit of code and you forget 
that you just decided to reverse the sense of the conditional, and 
complete the rest of the code backwards.


Some of us older folks don't need an external source of confusion.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: recent change to vim defaults?

[Julian Elischer]

On 19/01/2017 11:18 AM, Julian Elischer wrote:

On 19/01/2017 1:37 AM, Adam Weinberger wrote:
On 18 Jan, 2017, at 10:35, Julian Elischer <jul...@freebsd.org> 
wrote:


On 17/01/2017 1:23 AM, Adam Weinberger wrote:
On 16 Jan, 2017, at 9:25, Baptiste Daroussin <b...@freebsd.org> 
wrote:


On Mon, Jan 16, 2017 at 12:03:08AM +0800, Julian Elischer wrote:
I noticed that suddenly vim is grabbing mouse movements, which 
makes life

really hard.

Was there a specific revision that brought in this change, and 
can it be

removed?
This change appeared in one of the last patchset of vim 7.4 and 
was one of the

"features" of the vim 8.0 release.

I do agree this is just totally painful :(

Best regards,
Bapt
One of the things that I inherited with the Vim port was the 
DEFAULT_VIMRC option (which installs 
/usr/ports/editors/vim/files/vimrc), and I haven't touched it.


I have moused disabled in all my boxes so I have no idea about 
bad mouse behaviour in Vim. If there is a bad default that is 
causing grief, let's just fix it in that default vimrc.


I'm not really understanding what the unexpected behaviour is so 
I can't make an intelligent recommendation myself, but I'll go 
with whatever you folks suggest.


# Adam

I'm in iterm on my mac.
I ssh to a freebsd machine
I use vim on a file.
I used to be able to use the mouse on my mac to copy a few lines 
into the cut buffer.. slide-shift-click etc..
now suddenly if I try highlight some code in vim to copy it vim 
drags stuff around, scrolls up and down, deletes stuff and 
generally makes a mess.

if click, instead of starting a copy zone it grabs some of the text.

basically it makes hte mouse useless.
I can;t copy and paste from a file I'm ediitng. I end up having to 
exit vim and do it in vi.
There have been a number of recommendations in this thread for you, 
Julian, including "set mouse=a" and "set mouse=v". Test some of 
them out and let me know what works for you.


actually I never saw one about mouse=a however I did see and try 
mouse=v which didn't work for me.


I have now tried mouse=a and am happy to say that that does what I 
need.

thanks!


actually no, 'set mouse='
seems to be what I want.. not sure why I thought =a worked:




# Adam




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 
"freebsd-current-unsubscr...@freebsd.org"




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: recent change to vim defaults?

[Julian Elischer]

On 19/01/2017 1:37 AM, Adam Weinberger wrote:

On 18 Jan, 2017, at 10:35, Julian Elischer <jul...@freebsd.org> wrote:

On 17/01/2017 1:23 AM, Adam Weinberger wrote:

On 16 Jan, 2017, at 9:25, Baptiste Daroussin <b...@freebsd.org> wrote:

On Mon, Jan 16, 2017 at 12:03:08AM +0800, Julian Elischer wrote:

I noticed that suddenly vim is grabbing mouse movements, which makes life
really hard.

Was there a specific revision that brought in this change, and can it be
removed?

This change appeared in one of the last patchset of vim 7.4 and was one of the
"features" of the vim 8.0 release.

I do agree this is just totally painful :(

Best regards,
Bapt

One of the things that I inherited with the Vim port was the DEFAULT_VIMRC 
option (which installs /usr/ports/editors/vim/files/vimrc), and I haven't 
touched it.

I have moused disabled in all my boxes so I have no idea about bad mouse 
behaviour in Vim. If there is a bad default that is causing grief, let's just 
fix it in that default vimrc.

I'm not really understanding what the unexpected behaviour is so I can't make 
an intelligent recommendation myself, but I'll go with whatever you folks 
suggest.

# Adam

I'm in iterm on my mac.
I ssh to a freebsd machine
I use vim on a file.
I used to be able to use the mouse on my mac to copy a few lines into the cut 
buffer.. slide-shift-click etc..
now suddenly if I try highlight some code in vim to copy it vim drags stuff 
around, scrolls up and down, deletes stuff and generally makes a mess.
if click, instead of starting a copy zone it grabs some of the text.

basically it makes hte mouse useless.
I can;t copy and paste from a file I'm ediitng. I end up having to exit vim and 
do it in vi.

There have been a number of recommendations in this thread for you, Julian, including "set 
mouse=a" and "set mouse=v". Test some of them out and let me know what works for you.


actually I never saw one about mouse=a however I did see and try 
mouse=v which didn't work for me.


I have now tried mouse=a and am happy to say that that does what I need.
thanks!



# Adam




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: recent change to vim defaults?

[Julian Elischer]

On 18/01/2017 5:03 PM, Raimund Sacherer wrote:

I have to put mouse=v to get the behavior I am used to.

Best


doesn't really work for me.

vim is still taking mouse events


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: recent change to vim defaults?

[Julian Elischer]

On 17/01/2017 1:23 AM, Adam Weinberger wrote:

On 16 Jan, 2017, at 9:25, Baptiste Daroussin <b...@freebsd.org> wrote:

On Mon, Jan 16, 2017 at 12:03:08AM +0800, Julian Elischer wrote:

I noticed that suddenly vim is grabbing mouse movements, which makes life
really hard.

Was there a specific revision that brought in this change, and can it be
removed?

This change appeared in one of the last patchset of vim 7.4 and was one of the
"features" of the vim 8.0 release.

I do agree this is just totally painful :(

Best regards,
Bapt

One of the things that I inherited with the Vim port was the DEFAULT_VIMRC 
option (which installs /usr/ports/editors/vim/files/vimrc), and I haven't 
touched it.

I have moused disabled in all my boxes so I have no idea about bad mouse 
behaviour in Vim. If there is a bad default that is causing grief, let's just 
fix it in that default vimrc.

I'm not really understanding what the unexpected behaviour is so I can't make 
an intelligent recommendation myself, but I'll go with whatever you folks 
suggest.

# Adam

I'm in iterm on my mac.
I ssh to a freebsd machine
I use vim on a file.
I used to be able to use the mouse on my mac to copy a few lines into 
the cut buffer..  slide-shift-click etc..
now suddenly if I try highlight some code in vim to copy it vim drags 
stuff around, scrolls up and down, deletes stuff and generally makes a 
mess.

if click, instead of starting a copy zone it grabs some of the text.

basically it makes hte mouse useless.
I can;t copy and paste from a file I'm ediitng. I end up having to 
exit vim and do it in vi.







___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: recent change to vim defaults?

[Julian Elischer]

On 17/01/2017 12:07 AM, ohauer wrote:

I suspect you mean the /usr/local/etc/vim/vimrc and gvimrc files.
That was the first place I've tried to overwrite it, but without 
luck (even with set mouse=) but it works in ~/.vimrc


what to put IN the file?


--
olli
--
send with broken GMX mailer client, sorry for tofu and html scrap
On 15/01/2017, 22:48 Benjamin Kaduk <ka...@mit.edu> wrote:

On Mon, Jan 16, 2017 at 12:03:08AM +0800, Julian Elischer wrote:
> I noticed that suddenly vim is grabbing mouse movements, which
makes
> life really hard.
>
> Was there a specific revision that brought in this change, and
can it
> be removed?

I remember seeing something go by during an upgrade somewhat
recently
about there now being a defaults file that gets used when a user
does
not specify a .vimrc. Unfortunately, I don't remember whether I saw
that notice on a FreeBSD machine or a Debian one, and haven't
been able
to find the notice I remember through searching some likely places.

Just to check: do you have a .vimrc file in place already?


not yet.
when I work out what to put into it I will make it.



-Ben
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to
"freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

recent change to vim defaults?

[Julian Elischer]

I noticed that suddenly vim is grabbing mouse movements, which makes 
life really hard.


Was there a specific revision that brought in this change, and can it 
be removed?


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC as timecounter makes system lag [-> jhb]

[Julian Elischer]

On 15/01/2017 10:11 AM, Jia-Shiun Li wrote:

On Fri, Jan 13, 2017 at 8:26 AM, Jia-Shiun Li  wrote:


Hi all,

since 2 or 3 weeks ago, I noticed that my old Penryn-based Intel Pentium
T4200 notebook lagged a lot. System time was running a lot slower,
sometimes even looked like it freezed. Keystroke repeat rate was slow too.

Since system time is slow, I tried to change timecounter from default TSC
to HPET. And it resumed normal immediately.



Did a binary search. Turns out it was caused by r310177 "Enable
EARLY_AP_STARTUP on amd64 and i386 kernels by default." r310175 does not
have this issue. Removing this option from kernel config also solves it.


making sure jhb notices this.

-Jia-Shiun.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

FreeBSD system profiling and tuning for 10, 11 and 12

[Julian Elischer]

Hi

I'm looking for recent information regarding profiling and tuning in 
FreeBSD.


Google has turned up some links but I think that the best leads are 
still hiding..
 for example I only found 
https://wiki.freebsd.org/NetworkPerformanceTuning recently.


(BTW Anyone who has a moment is encouraged to check if they have 
anything to add to it.)


I am sure there is better information around for profiling the kernel 
and modules,
but I am not seeing "the definitive profiler's guide" out there. I do 
know several people

have blogs that cover this sort of thing. If you have one or know of good
profiling resources we should gather them.. send them to me and I'll 
try make

sure everything is up to date, and put them together in a wiki page.

there is already some stuff there,
(see 
https://wiki.freebsd.org/NetworkPerformanceTuning?action=fullsearch=180=profiling=Text 
)

but it could do with gathering together.


Julian



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: tput failing on 10 and 12 systems?

[Julian Elischer]

On 22/11/2016 12:49 AM, Adrian Chadd wrote:

Hiya,

Just to follow through - part of tput is to decode command args, but
it then just requests it from the ncurses tputs() call.

So it could be tput, but it could also be the BSD ncurses work.


https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=214709



-adrian


On 21 November 2016 at 07:54, Julian Elischer <jul...@freebsd.org> wrote:

example on freefall (FreeBSD 12.0-CURRENT #0 r306376)

julian@freefall:tput setaf 4|od -c
000  033   [   m
003

so nothing happens, and tput returns 1.

but on a FreeBSD 8 system:

[jelischer@alpha ~]$ tput setaf 4|od -c
000  033   [   3   4   m
005

which make the text change color.


similarly all the related tput color commands I can think of fail on
10,11,12

but programs such as vim can use color just fine. So I suspect tput itself.


anyone have experience with this? seems easy to reproduce.







___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: tput failing on 10 and 12 systems?

[Julian Elischer]

On 22/11/2016 12:49 AM, Adrian Chadd wrote:

Hiya,

Just to follow through - part of tput is to decode command args, but
it then just requests it from the ncurses tputs() call.

So it could be tput, but it could also be the BSD ncurses work.

but the ncurses stuff works for things like vim



-adrian


On 21 November 2016 at 07:54, Julian Elischer <jul...@freebsd.org> wrote:

example on freefall (FreeBSD 12.0-CURRENT #0 r306376)

julian@freefall:tput setaf 4|od -c
000  033   [   m
003

so nothing happens, and tput returns 1.

but on a FreeBSD 8 system:

[jelischer@alpha ~]$ tput setaf 4|od -c
000  033   [   3   4   m
005

which make the text change color.


similarly all the related tput color commands I can think of fail on
10,11,12

but programs such as vim can use color just fine. So I suspect tput itself.


anyone have experience with this? seems easy to reproduce.







___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

tput failing on 10 and 12 systems?

[Julian Elischer]

example on freefall (FreeBSD 12.0-CURRENT #0 r306376)

julian@freefall:tput setaf 4|od -c
000  033   [   m
003

so nothing happens, and tput returns 1.

but on a FreeBSD 8 system:

[jelischer@alpha ~]$ tput setaf 4|od -c
000  033   [   3   4   m
005

which make the text change color.


similarly all the related tput color commands I can think of fail on 
10,11,12


but programs such as vim can use color just fine. So I suspect tput 
itself.



anyone have experience with this? seems easy to reproduce.







___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

problem with mpt driver. anyone seen this or similar? (10.3)

[Julian Elischer]

Does this ring any bells?
even a theory would be a big improvement.

memcpy+0xc
mpt_read_cfg_page+0xcc
mpt_cation+0x148e
xpt_action_default+0x7e
cam_periph_runccb+0x7c
passdoioctl+0x719
passioctl+0x30
devfs_ioctl_f+0x7c
kern_ioctl+0x1a8
sys_ioctl+0x11f
amd64_syscall+0x3f9
xfast_syscall+0xf7

we see a memory access fault at line 1821..

1786 int
1787 mpt_read_cfg_page(struct mpt_softc *mpt, int Action, uint32_t PageAddress,
1788   CONFIG_PAGE_HEADER *hdr, size_t len, int sleep_ok,
1789   int timeout_ms)
1790 {
1791 request_t*req;
1792 cfgparms_tparams;
1793 int   error;
1794
1795 req = mpt_get_request(mpt, sleep_ok);
1796 if (req == NULL) {
1797 mpt_prt(mpt, "mpt_read_cfg_page: Get request failed!\n");
1798 return (-1);
1799 }
1800
1801 params.Action = Action;
1802 params.PageVersion = hdr->PageVersion;
1803 params.PageLength = hdr->PageLength;
1804 params.PageNumber = hdr->PageNumber;
1805 params.PageType = hdr->PageType & MPI_CONFIG_PAGETYPE_MASK;
1806 params.PageAddress = PageAddress;
1807 error = mpt_issue_cfg_req(mpt, req, ,
1808   req->req_pbuf + MPT_RQSL(mpt),
1809   len, sleep_ok, timeout_ms);
1810 if (error != 0) {
1811 mpt_prt(mpt, "read_cfg_page(%d) timed out\n", Action);
1812 return (-1);
1813 }
1814
1815 if ((req->IOCStatus & MPI_IOCSTATUS_MASK) != 
MPI_IOCSTATUS_SUCCESS) {
1816 mpt_prt(mpt, "mpt_read_cfg_page: Config Info Status %x\n",
1817 req->IOCStatus);
1818 mpt_free_request(mpt, req);
1819 return (-1);
1820 }
1821 memcpy(hdr, ((uint8_t *)req->req_vbuf)+MPT_RQSL(mpt), len);   
<--
1822 mpt_free_request(mpt, req);
1823 return (0);
1824 }
1825
1826 int
1827 mpt_write_cfg_page(struct mpt_softc *mpt, int Action, uint32_t PageAddress,
"mpt/mpt.c" [readonly] 3146 lines --58%--

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Fwd: Re: how to reduce the size of /usr/share/i18n data?

[Julian Elischer]

On 3/11/2016 7:46 PM, Tomoaki AOKI wrote:

On Thu, 3 Nov 2016 06:46:39 -0400
Mark Heily  wrote:


On Nov 3, 2016 5:30 AM, "Kurt Jaeger"  wrote:

Hi!


So I am to take it that no-one has any idea how this stuff works and
how to stub it out?

Or had time to write about it.

--
p...@opsec.eu+49 171 3101372 4 years to

go !
Maybe you could use 'svn blame' to research who has commited those files,
and contact them directly?

Not shure but maybe WITHOUT_ICONV knob would eliminate them from build.
Beware! It should completely eliminate iconv features.

See commit message for Revision 219019 below.

   https://svnweb.freebsd.org/base/head/share/i18n/Makefile?view=log

Sorry, I don't know how to safely delete some (not all) part of
conversions to shrink the contents there, keeping limited iconv
features to work.

there are some 3rd party apps that want them so I'd rather just know 
how to make a really small subset.


there is a database that is describes the data there but I have no 
clue how to generate a new db.





___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [RFC] remove GNU rcs from FreeBSD 12

[Julian Elischer]

On 3/11/2016 10:45 AM, Pete Wright wrote:



On 11/02/2016 19:17, Julian Elischer wrote:

On 26/10/2016 2:27 AM, Eivind Nicolay Evensen wrote:

On Sun, Sep 11, 2016 at 03:38:04PM +0200, Baptiste Daroussin wrote:

hi,

For long we are planning to remove GNU rcs from base, after a 
failed attempt
before FreeBSD 10.0. Let see where we are to be able to remove it 
from FreeBSD

12.


why should we remove it?
What will replace it?  it's an integral part of many people's systems.

Is there a non gnu RCS with the same features?



surprised to hear so many people are dependent upon having rcs in 
their base system.  there are options though - for example OpenBSD 
uses OpenRCS in their base:

I don't see what is surprising about it.
It's been common "best practice" for decades to keep all your files in 
/etc under source control. RCS fits he bill well and many people have 
it in their muscle memory.




http://man.openbsd.org/rcs.1

its not strictly GNU compliant as I believe it adheres to the 
original implementation (which frankly is probably a good thing 
imho).  GNU RCS is also available via ports/pkgs as well.


If people are adamant about preserving a rcs binary in base though 
this seems like a great opportunity to step up and help 
import/support OpenRCS.


that's ok by me as long as:
1/ it can continue to read all the old data
2/ it works the same (for scripts etc)



just my two bits - hope it helps.

-pete


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 
"freebsd-current-unsubscr...@freebsd.org"




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Fwd: Re: how to reduce the size of /usr/share/i18n data?

[Julian Elischer]

So I am to take it that no-one has any idea how this stuff works and 
how to stub it out?


On 1/11/2016 7:11 PM, Julian Elischer wrote:


01.11.2016 17:53, Julian Elischer пишет:
there are a number of packages that want to link with or use that 
data, and you can't always disable it, but it's very very big 
(38MB?), especially in the context of an appliance that doesn't 
really need it at all.



If anyone has a procedure to follow to put that onto a diet, maybe 
just as a stub then I'm all ears.


+1

Introduction of such large part of base system is kind of 
catastrophe for embedded systems
that need only ASCII and may be additionally one of "good old" 8-bit 
locales.


FreeBSD 11 got pretty large and embedded-unfriendly without clear 
way to exclude such unneeded parts.



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 
"freebsd-current-unsubscr...@freebsd.org"





___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [RFC] remove GNU rcs from FreeBSD 12

[Julian Elischer]

On 26/10/2016 2:27 AM, Eivind Nicolay Evensen wrote:

On Sun, Sep 11, 2016 at 03:38:04PM +0200, Baptiste Daroussin wrote:

hi,

For long we are planning to remove GNU rcs from base, after a failed attempt
before FreeBSD 10.0. Let see where we are to be able to remove it from FreeBSD
12.


why should we remove it?
What will replace it?  it's an integral part of many people's systems.

Is there a non gnu RCS with the same features?


Whatever the outcome may be and for whatever my opinion is worth, I hope
rcs will stay in base. I don't care about the licensing. I don't
care if a switch to openrcs happens either, as long as it works.

For years, one has been able to rely upon this operating system having
certain pieces of software available. Losing that makes it a worse choice
than before.

I've already had to readd cvs to my freebsd tree since that was removed,
but if it keeps getting worse and worse and there's soon a
freebsd kernel and some random bits of freebsd userland available
through ports, there's not much reason to keep using it as an operating
system, because then it is not an operating system anymore, rather an
emulation of another "system" built around that concept.



Eivind N. E.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Fwd: Re: how to reduce the size of /usr/share/i18n data?

[Julian Elischer]

 Forwarded Message 
From:   25 2016 <>
X-Mozilla-Status:   0013
X-Mozilla-Status2:  
X-Mozilla-Keys: 
Return-Path:<eu...@grosbein.net>
Received: 	from sea.spamstick.net (sea.spamstick.net [204.109.57.196]) 
by vps1.elischer.org (8.15.2/8.15.2) with ESMTPS id uA1AxGgp045127 
(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) 
for <jul...@elischer.org>; Tue, 1 Nov 2016 03:59:20 -0700 (PDT) 
(envelope-from eu...@grosbein.net)
Received: 	from mx2.freebsd.org ([8.8.178.116]) by sea.spamstick.net 
with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.86) 
(envelope-from <eu...@grosbein.net>) id 1c1Wmg-0008PF-00 for 
jul...@elischer.org; Tue, 01 Nov 2016 06:59:16 -0400
Received: 	from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using 
TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client 
certificate requested) by mx2.freebsd.org (Postfix) with ESMTPS id 
9F2AE6694B for <jul...@elischer.org>; Tue, 1 Nov 2016 10:59:09 + 
(UTC) (envelope-from eu...@grosbein.net)
Received: 	from freefall.freebsd.org (freefall.freebsd.org 
[IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with 
ESMTP id 85EE31320 for <jul...@elischer.org>; Tue, 1 Nov 2016 10:59:09 
+ (UTC) (envelope-from eu...@grosbein.net)
Received: 	by freefall.freebsd.org (Postfix) id 847141D68; Tue, 1 Nov 
2016 10:59:09 + (UTC)

Delivered-To:   jul...@localmail.freebsd.org
Received: 	from mx1.freebsd.org (mx1.freebsd.org 
[IPv6:2001:1900:2254:206a::19:1]) by freefall.freebsd.org (Postfix) 
with ESMTP id 839321D67 for <jul...@localmail.freebsd.org>; Tue, 1 Nov 
2016 10:59:09 + (UTC) (envelope-from eu...@grosbein.net)
Received: 	from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) 
(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN 
"hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by 
mx1.freebsd.org (Postfix) with ESMTPS id 1CDD4131E; Tue, 1 Nov 2016 
10:59:08 + (UTC) (envelope-from eu...@grosbein.net)
Received: 	from eg.sd.rdtc.ru (r...@eg.sd.rdtc.ru [62.231.161.221]) by 
hz.grosbein.net (8.14.9/8.14.9) with ESMTP id uA1Ax3QX049994 
(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); 
Tue, 1 Nov 2016 11:59:04 +0100 (CET) (envelope-from eu...@grosbein.net)

X-Envelope-From:eu...@grosbein.net
X-Envelope-To:  jul...@freebsd.org
Received: 	from [10.58.0.10] (dadv@dadvw [10.58.0.10]) by 
eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id uA1AwxFL006726 
(version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 
1 Nov 2016 17:59:00 +0700 (KRAT) (envelope-from eu...@grosbein.net)

Subject:    Re: how to reduce the size of /usr/share/i18n data?
To: 	Julian Elischer <jul...@freebsd.org>, freebsd 
<freebsd-hack...@freebsd.org>

References: <7b036323-aa77-6d41-36b0-439a12a36...@freebsd.org>
From:   Eugene Grosbein <eu...@grosbein.net>
Message-ID: <58187573.7020...@grosbein.net>
Date:   Tue, 1 Nov 2016 17:58:59 +0700
User-Agent: 	Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) 
Gecko/20100101 Thunderbird/38.7.2

MIME-Version:   1.0
In-Reply-To:<7b036323-aa77-6d41-36b0-439a12a36...@freebsd.org>
Content-Type:   text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding:  8bit
X-Spam-Status: 	No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM 
autolearn=no version=3.3.2
X-Spam-Report: 	* -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 
1% * [score: 0.] * 2.6 LOCAL_FROM From my domains
X-Spam-Checker-Version: 	SpamAssassin 3.3.2 (2011-06-06) on 
hz.grosbein.net
X-Filter-ID: 
s0sct1PQhAABKnZB5plbIXW6wgLqzC+TM2pKklekeEcWCsCPLMUAyrpAA5pkQvo1v0WM/M+C7g6f 
jq7KivljIgACc3eCAL7Sz/73rBvo2zdf4seDSE2pIFs49EdaF4r0GlH4z9w5rjAiHtPWW3keAShN 
xLoqnsw2BxcNT/6ha/Mu906Tg7P5f/6ZDJb4IgmNpVaLEMBpkLjCNy1XrjsuhMD0qhcz494kqrQX 
F25rLf9/U/jsH4EXGgfUT7LIXyPe+DsKzbLMLhst3n8kX6ngqWiWloQ6ztDmkvSWg7Jumw5Pg6dw 
4lDcu51VWwE6bi+ZrE6kdjyQEObSFjdQy/q2Lhnis/L5RJfxkhIgp8XOgK0jJHKF9QPFEIASHMiN 
C2cwmeLy/f6zOMlHyzALZS3+BqGp3RSwCXW+mzuiSaXGa7JQvpJ0u5JNUTFfeWDaD3S/lnEaTdft 
T8Kul6ZpNgSqNORxWsgqcF2qRmccXjejs6+dtOiBbpDQ+zv6ENK2PSDvNSjVRJ+igYTdhxxwxq0Y 
S6Qp9bdL8oYeEB2/gTYIJTIxL7hrJSk60SF3F6RYOYr2

X-Report-Abuse-To:  s...@sea.spamstick.net
X-SpamStick-Class:  ham
X-SpamStick-Evidence:   SB/spamstick_net (0.0487503235617)
X-Recommended-Action:   accept



01.11.2016 17:53, Julian Elischer пишет:

there are a number of packages that want to link with or use that data, and you 
can't always disable it, but it's very very big (38MB?), especially in the 
context of an appliance that doesn't really need it at all.


If anyone has a procedure to follow to put that onto a diet, maybe just as a 
stub then I'm all ears.


+1

Introduction of such large part of base system is kind of catastrophe for 
embedded systems
that need only ASCII and may be additionally one of "good old" 8-bit loc

Re: zfs crash on FreeBSD 10.3

[Julian Elischer]

On 14/10/2016 6:31 PM, Trond Endrestøl wrote:

On Fri, 14 Oct 2016 00:19-0700, Julian Elischer wrote:


I attempted to add a second partition to an existing FS pool in FreeBSD 10.3
and the result was a crash..

is there anyone out there with a scratch system (10.3) (or two spare drives)
who can show me this working?

Does it look familiar to anyone?

The drive 'boot0' is being used as the root drive, but we are in single user
mode, so its' read-only at this stage.

=== cut-n-paste =

[boot -s]

[...]

Trying to mount root from zfs:p8/image1 []...
Enter full pathname of shell or RETURN for /bin/sh:
PS1="# "
#
#
# ls /dev/gpt
boot0boot1
# zpool attach -f p8 gpt/boot0 gpt/boot1

Do you really need to force zpool to attach the second partition?
What happens if you omit the -f flag?

from memory, it does nothing
I can't test it now but I know the example I saw on the internet gave 
'-f , and when I tried to leave it out,
it didn't work. I vaguely remember that it complained about the 
filesystem being in use.


I will try it again some time when I have the setup running and get 
back to you.



Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address= 0x50
fault code= supervisor read data, page not present
instruction pointer= 0x20:0x81717063
stack pointer= 0x28:0xfe0169bfc640
frame pointer= 0x28:0xfe0169bfc9a0
code segment= base 0x0, limit 0xf, type 0x1b
 = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process= 3 (txg_thread_enter)
trap number= 12
Panic:Thought about setting the watchdog to 10 Minutes
panic: page fault
cpuid = 1

KDB: stack backtrace:
  stack1 db_trace_self_wrapper+0x2a
  kdb_backtrace+0x37 vpanic+0xf7
  panic+0x67 trap_fatal+0x264
  trap_pfault+0x1c2
  trap+0x38c
  calltrap+0x8
  dsl_scan_sync+0x2f3
  spa_sync+0x328
  txg_sync_thread+0x140
  fork_exit+0x135
  fork_trampoline+0xe

KDB: enter: panic
[ thread pid 3 tid 100328 ]
Stopped at  kdb_enter+0x50: movq$0,0x6bd5cd(%rip)
db> reboot

I will add that after this, every boot hits this problem. (same stack trace).
the box is effectively bricked
(or would be if it weren't just a bhyve image)



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

zfs crash on FreeBSD 10.3

[Julian Elischer]

I attempted to add a second partition to an existing FS pool in 
FreeBSD 10.3 and the result was a crash..


is there anyone out there with a scratch system (10.3) (or two spare 
drives) who can show me this working?


Does it look familiar to anyone?

The drive 'boot0' is being used as the root drive, but we are in 
single user mode, so its' read-only at this stage.


=== cut-n-paste =

[boot -s]

[...]

Trying to mount root from zfs:p8/image1 []...
Enter full pathname of shell or RETURN for /bin/sh:
PS1="# "
#
#
# ls /dev/gpt
boot0boot1
# zpool attach -f p8 gpt/boot0 gpt/boot1

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address= 0x50
fault code= supervisor read data, page not present
instruction pointer= 0x20:0x81717063
stack pointer= 0x28:0xfe0169bfc640
frame pointer= 0x28:0xfe0169bfc9a0
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process= 3 (txg_thread_enter)
trap number= 12
Panic:Thought about setting the watchdog to 10 Minutes
panic: page fault
cpuid = 1

KDB: stack backtrace:
 stack1 db_trace_self_wrapper+0x2a
 kdb_backtrace+0x37 vpanic+0xf7
 panic+0x67 trap_fatal+0x264
 trap_pfault+0x1c2
 trap+0x38c
 calltrap+0x8
 dsl_scan_sync+0x2f3
 spa_sync+0x328
 txg_sync_thread+0x140
 fork_exit+0x135
 fork_trampoline+0xe

KDB: enter: panic
[ thread pid 3 tid 100328 ]
Stopped at  kdb_enter+0x50: movq$0,0x6bd5cd(%rip)
db> reboot

I will add that after this, every boot hits this problem. (same stack 
trace). the box is effectively bricked

(or would be if it weren't just a bhyve image)


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Maximum memory for various freebsd releases

[Julian Elischer]

Looking through the various release notes I see that one thing thing 
that is rarely mentioned is the amount of physical memory supported, 
or any tuning that may be required to run more than some amoutn 
(should it be necessary to change some table sizes etc.).


If anyone has that information I'm looking to know if it is possible 
to run 768GM of ram on an 8.0 machine and a 10.0 machine, and if not 
in the default configuration, whether there are any changes to the 
configuration that would allow this.


Julian


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Passwordless accounts vi ports!

[Julian Elischer]

On 11/08/2016 1:16 PM, Ngie Cooper wrote:

On Aug 10, 2016, at 22:05, O. Hartmann  wrote:

I just checked the security scanning outputs of FreeBSD and found this
surprising result:

[...]
Checking for passwordless accounts:
polkitd::565:565::0:0:Polkit Daemon User:/var/empty:/usr/sbin/nologin
pulse::563:563::0:0:PulseAudio System User:/nonexistent:/usr/sbin/nologin
saned::194:194::0:0:SANE Scanner Daemon:/nonexistent:/bin/sh
clamav::106:106::0:0:Clamav Antivirus:/nonexistent:/usr/sbin/nologin
bacula::910:910::0:0:Bacula Daemon:/var/db/bacula:/usr/sbin/nologin
[...]

Obviously, some ports install accounts but do not secure them as there is an
empty password.

I consider this not a feature, but a bug.

saned is the only one that might concern me because the login shell isn't 
nologin(1).


but other tools use the password database.. e.g. ftp



Cheers,
-Ngie
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: date(1) default format changed between 10.3 and 11.0-BETA3

[Julian Elischer]

On 6/08/2016 11:09 PM, Benjamin Kaduk wrote:

On Sat, 6 Aug 2016, Baptiste Daroussin wrote:


On Sat, Aug 06, 2016 at 02:15:36PM +1000, Greg 'groggy' Lehey wrote:

On Friday,  5 August 2016 at 18:56:33 +0300, Andrey A. Chernov wrote:

On 05.08.2016 18:44, Mark Martinec wrote:

On 2016-08-05 17:23, Andrey Chernov wrote:

POSIX does say that the default format should be the same
as with "+%a %b %e %H:%M:%S %Z %Y".
It also says that %a and %b are locale's abbreviated names.

It is true for _POSIX_ locale only, as I already say. en_US.* is not
POSIX or C locale.

It still violates POLA.


I really do not think that it violates POLA fiven that the behaviour you are
expecting is still available in the default configurtion that is still POSIX.

Regardless, at a new major release is precisely when it is permissible to
break POLA.
switching from short form to long form is more than a POLA..  short 
form has a specific fixed layout

and feeding a long form string into it will break things.



Set LC_TIME to C and then you are back on your behaviour (and this is the
default when you install FreeBSD).

locales should be seen as tzdata for exemple, they are a moving target
complicated to handle for every locale we do support: 78 for 11.0-RELEASE and
193 if we do count the encoding variants. locales are updated very often (for
exemple cldr unicode make a new release of the data every 8 month or so)

As I understand it, your change will improve the maintainability of
locales in FreeBSD in the future, which justifies a POLA break at the
release boundary.

-Ben
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: date(1) default format changed between 10.3 and 11.0-BETA3

[Julian Elischer]

On 5/08/2016 5:44 AM, Mark Martinec wrote:

Should I open a bug report, or has the problem been noted?
it's not clear without reading the standard whether the bug is in the 
old or new version.

have you tried other systems? In particular I'd check OSX

sh-3.2$ export LC_CTYPE="en_US.UTF-8"
sh-3.2$ export LC_TIME="en_US.UTF-8"
sh-3.2$ export LC_ALL="en_US.UTF-8"
sh-3.2$ export LC_NUMERIC="en_US.UTF-8"
sh-3.2$ date
Fri Aug  5 12:57:47 AWST 2016

if it IS a bug then yes, file a report with full reproduction steps.



  Mark


On 2016-08-04 04:32, Julian Elischer wrote:

On 4/08/2016 7:24 AM, Mark Martinec wrote:

Is it normal/expected/documented that the date(1) command in 11.0
now produces a timestamp in substantially different format
in an "en_US.UTF-8" locale (long names, commas, 12 vs. 24h hour 
time):


  Thursday, August  4, 2016 at 12:50:43 AM CEST
vs:
  Thu Aug  4 00:52:29 CEST 2016


one of those is a bug.  the formats are defined in posix I believe.



Setting LC_TIME does not help:

  $LC_TIME="C" date
  Thursday, August  4, 2016 at 01:13:37 AM CEST

although LC_ALL="C" _does_ help.


This is funny too, especially regarding commas:
  $ LC_ALL="en_GB.UTF-8" date
  Thursday  4 August 2016 at 01:16:45 CEST
  $ LC_ALL="en_US.UTF-8" date
  Thursday, August  4, 2016 at 01:16:54 AM CEST


The date(1) man page states:
  The date utility is expected to be compatible with IEEE Std 1003.2
  (“POSIX.2”).
What does POSIX.2 say about date(1) following a locale?



==
11.0-BETA3:

$ date
Thursday, August  4, 2016 at 12:50:43 AM CEST

$ uname -a
FreeBSD xxx.ijs.si 11.0-BETA3 FreeBSD 11.0-BETA3 #0 r303469: Fri 
Jul 29 02:27:28 UTC 2016 
r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64


$ locale
LANG=
LC_CTYPE="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=en_US.UTF-8

==
10.3-RELEASE-p6 :

$ date
Thu Aug  4 00:52:29 CEST 2016

$ freebsd-version
10.3-RELEASE-p6

$ uname -a
FreeBSD yyy.ijs.si 10.3-RELEASE-p4 FreeBSD 10.3-RELEASE-p4 #0: Sat 
May 28 12:23:44 UTC 2016 
r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64


$ locale
LANG=
LC_CTYPE="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=en_US.UTF-8



  Mark

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 
"freebsd-current-unsubscr...@freebsd.org"





___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Socket sendmsg() porting question

[Julian Elischer]

On 4/08/2016 1:18 AM, Lundberg, Johannes wrote:

​Hi Alan

Thanks for the reply.

Can I still use the same receiving function for sendmsg/send and tell what
kind of message is coming?
How would I tell if there is an fd attached or not?

Even if I set cmsg_level and cmsg_type it won't let me send it. The problem
is having a zero length attachment on freebsd
I can't send -1 as fd because that errors to invalid file descriptor.


On Wed, Aug 3, 2016 at 10:12 AM, Alan Somers  wrote:


On Wed, Aug 3, 2016 at 10:54 AM, Lundberg, Johannes
 wrote:

Hi

I'm porting a project to fbsd and I have problem with this part that

works

in linux but not fbsd when fd = -1.

https://github.com/Cloudef/wlc/blob/master/src/session/fd.c#L80-L108

I get "invalid argument" from sendmsg() when setting CMSG_LEN(0).

Anyone have a clue how to correctly do this on fbsd?

Thanks!

Johannes


It sounds like you're trying to send an empty cmsg.  The error may
happen because your msg_controllen field is inconsistent with your
cmsg_len field.  You're setting msg_controllen as if there were a full
cmsg, but  then cmsg_len says that there is no cmsg.  Or maybe the
error is because (just guessing) FreeBSD doesn't allow sending empty
or undefined cmsgs.  Notice that cmsg_level and cmsg_type are
undefined in the case where fd == -1.  POSIX doesn't say whether
sendmsg supports empty cmsgs, but why bother?  You could just use send
instead of sendmsg if you're not sending a file descriptor.

-Alan


I think it's a standards interpretation thing.

what data do you send WITH the message? I assume you have some in-band 
data as well.


if you have no FD, just use "sendto()."

the other end will still be able to do a recvmesg() but will discover 
no added info.



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: date(1) default format changed between 10.3 and 11.0-BETA3

[Julian Elischer]

On 4/08/2016 7:24 AM, Mark Martinec wrote:

Is it normal/expected/documented that the date(1) command in 11.0
now produces a timestamp in substantially different format
in an "en_US.UTF-8" locale (long names, commas, 12 vs. 24h hour time):

  Thursday, August  4, 2016 at 12:50:43 AM CEST
vs:
  Thu Aug  4 00:52:29 CEST 2016


one of those is a bug.  the formats are defined in posix I believe.



Setting LC_TIME does not help:

  $ LC_TIME="C" date
  Thursday, August  4, 2016 at 01:13:37 AM CEST

although LC_ALL="C" _does_ help.


This is funny too, especially regarding commas:
  $ LC_ALL="en_GB.UTF-8" date
  Thursday  4 August 2016 at 01:16:45 CEST
  $ LC_ALL="en_US.UTF-8" date
  Thursday, August  4, 2016 at 01:16:54 AM CEST


The date(1) man page states:
  The date utility is expected to be compatible with IEEE Std 1003.2
  (“POSIX.2”).
What does POSIX.2 say about date(1) following a locale?



==
11.0-BETA3:

$ date
Thursday, August  4, 2016 at 12:50:43 AM CEST

$ uname -a
FreeBSD xxx.ijs.si 11.0-BETA3 FreeBSD 11.0-BETA3 #0 r303469: Fri Jul 
29 02:27:28 UTC 2016 
r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64


$ locale
LANG=
LC_CTYPE="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=en_US.UTF-8

==
10.3-RELEASE-p6 :

$ date
Thu Aug  4 00:52:29 CEST 2016

$ freebsd-version
10.3-RELEASE-p6

$ uname -a
FreeBSD yyy.ijs.si 10.3-RELEASE-p4 FreeBSD 10.3-RELEASE-p4 #0: Sat 
May 28 12:23:44 UTC 2016 
r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64


$ locale
LANG=
LC_CTYPE="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=en_US.UTF-8



  Mark
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 
"freebsd-current-unsubscr...@freebsd.org"





___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Xen networking problems in -current with xn driver?

[Julian Elischer]

I upgraded my VPS machine to today's current, and on reboot I couldn't 
get into it by network.


A quick switch to the VNC console showed that it was up but that it 
couldn't get out.



The xn interfaces said they were UP but attempts to get out were met 
with "network is down".


if I did 'tcpdump -n -i xn0' (and xn1) hten all was fine again.

tcpdump saw packets, and in fact ipfw saw some packets coming in even 
before that but it was not possible to send.



Has anyone seen similar?

some relevant parts of the dmesg output.:


T(vga): text 80x25
XEN: Hypervisor version 3.4 detected.
CPU: Intel(R) Xeon(R) CPU   E5620  @ 2.40GHz (2400.05-MHz 
686-class CPU)

  Origin="GenuineIntel"  Id=0x206c2  Family=0x6  Model=0x2c Stepping=2
Features=0x1781fbff
Features2=0x80982201
  AMD Features=0x2010
  AMD Features2=0x1
Hypervisor: Origin = "XenVMMXenVMM"
real memory  = 536870912 (512 MB)
avail memory = 503783424 (480 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: 
WARNING: L1 data cache covers less APIC IDs than a core
0 < 1
WARNING: L2 data cache covers less APIC IDs than a core
0 < 1
WARNING: L3 data cache covers less APIC IDs than a core
0 < 1

ipfw2 (+ipv6) initialized, divert loadable, nat enabled, default to 
deny, logging disabled

xs_dev0:  on xenstore0
xenbusb_front0:  on xenstore0
xn0:  at device/vif/0 on xenbusb_front0
xn0: Ethernet address: 00:16:3e:01:99:54
xn1:  at device/vif/1 on xenbusb_front0
xn1: Ethernet address: 00:16:3e:01:9a:54
xenbusb_back0:  on xenstore0
xenballoon0:  on xenstore0
xctrl0:  on xenstore0
xn0: backend features: feature-sg feature-gso-tcp4
xn1: backend features: feature-sg feature-gso-tcp4
xbd0: 20480MB  at device/vbd/768 on xenbusb_front0
xbd0: attaching as ada0
xbd0: features: write_barrier
xbd0: synchronize cache commands enabled.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: dtrace and kernel modules

[Julian Elischer]

On 13/07/2016 1:14 AM, Mark Johnston wrote:

On Thu, Jul 07, 2016 at 12:51:52PM +0800, Julian Elischer wrote:

I'm specifically interested in the case of kernel modules that
instantiate new syscalls.

How much support do we have for that?  In the one example in our
sources of a kld with a syscall (kgssapi.ko) dtrace seems to find
regular function entrypoints but not the syscall.


root@porridge:/usr/src # dtrace -n ":kgssapi::entry {}"
dtrace: description ':kgssapi::entry ' matched 138 probes
^C

root@porridge:/usr/src # dtrace -n "syscall:kgssapi::entry {}"
dtrace: invalid probe specifier syscall:kgssapi::entry {}: probe
description syscall:kgssapi::entry does not match any probes
root@porridge:/usr/src #


Do we have plans to support dynamic syscall support?

I don't know of any plans to add support. It would be fairly
straightforward to dynamically create syscall probes using a hook or
eventhandler in syscall_register(), but getting argument type info would
be more difficult. Right now, argument types are specified by code
generated by makesyscalls.sh using the types in syscalls.master. I'm not
sure how one might obtain these types for dynamically-registered
syscalls.


yes that is the tricky part for sure.

for now function calls is probably enough because every syscall ends 
up calling a function somewhere :-)




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

dtrace and kernel modules

[Julian Elischer]

I'm specifically interested in the case of kernel modules that 
instantiate new syscalls.


How much support do we have for that?  In the one example in our 
sources of a kld with a syscall (kgssapi.ko) dtrace seems to find 
regular function entrypoints but not the syscall.



root@porridge:/usr/src # dtrace -n ":kgssapi::entry {}"
dtrace: description ':kgssapi::entry ' matched 138 probes
^C

root@porridge:/usr/src # dtrace -n "syscall:kgssapi::entry {}"
dtrace: invalid probe specifier syscall:kgssapi::entry {}: probe 
description syscall:kgssapi::entry does not match any probes

root@porridge:/usr/src #


Do we have plans to support dynamic syscall support?

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Virtualbox kernel module on 11-CURRENT

[Julian Elischer]

On 8/06/2016 5:13 AM, Kevin Oberman wrote:

On Tue, Jun 7, 2016 at 1:04 AM, Guido Falsi  wrote:


On 06/07/16 02:23, Rafael Rodrigues Nakano wrote:

Hello,

I tried installing virtualbox from packages, building it from sources,
trying the GENERIC kernel but everytime I can't start the kernel module
'vboxdrv', it says:
"KLD vboxdrv.ko: depends on kernel - not available or version mismatch.
linker_load_file: Unsupported file type"


The virtualbox module needs to be in full sync with the kernel. Most
probably the sources being used by the cluster for building packages on
head are a little different from yours, so the kernel module is not in
sync.

You will need to build the kernel module yourself to actually match your
kernel sources.

It's not really a problem or a bug, it's how it works. On head there is
no warranty about the KBI. This cannot happen on releases and stable
because the KBI is not going to change there.

--
Guido Falsi 


I don't think this is true. While shareable libraries have fixed ABIs, I
believe the KBI can change even in STABLE branches.  If a security fix
requires it, it might even change in a RELEASE. I my be wrong about this,
but I recall having to re-build the VB kmod port even withing a minor
version (i.e. STABLE).


We try hard NOT to change the KBI within a single stable branch.
we do things like add spare fields before we make a new stable branch 
to help with this.



In any case, I do strongly recommend the use of PORTS_MODULES in
/etc/src.conf to assure that the kernel modules always get re-built when
the kernel is re-built. so that the sources, the kernel, and the module are
in sync. The PORTS_MODULES are re-installed as a part of the "make
installkernel", so things are almost safe, but beware of "make
reinstallkernel" as it does not do the right thing. (See
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=201779)
--
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkober...@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

question on packaging base.

[Julian Elischer]

Given the work going on for incremental builds, together with the work 
in packaging the base, is it planned that we will be able to rebuild 
jut one package at a time?


For example will be be able to rebuild just he openssl-base (or 
whatever it is called) package without recompiling everything else, 
even if we have not previously recompiled everything else?





___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: pkg chroot issues?

[Julian Elischer]

On 30/05/2016 12:33 AM, Tim Kientzle wrote:

On May 29, 2016, at 8:55 AM, Julian Elischer <jul...@freebsd.org> wrote:

I was thinking that in order to do this properly the chrooted child should do 
all it's fetch requests etc via the non-chrooted parent, but that would have 
probably been a bit too complicated. (including add requests)

How complex would it be to perform all of the fetch requests before chroot?
I don't know but you'd have to read the DB in the chroot before 
knowing what dependencies you need to fix.




Tim





___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: pkg chroot issues?

[Julian Elischer]

On 23/05/2016 4:32 AM, b...@freebsd.org wrote:

On Sun, May 22, 2016 at 10:31:08PM +0200, b...@freebsd.org wrote:

On Sun, May 22, 2016 at 01:24:12PM -0700, Tim Kientzle wrote:

Crochet has some experimental hooks to install packages onto the system being 
built, but this seems to be hitting problems due to limitations in 'pkg -c'.  
In particular, it seems that pkg performs the chroot before it does any network 
lookups.  This is a problem if the chroot is not a complete system environment 
(which it cannot be when you're building an image for another system).

There's some further discussion on github:

   https://github.com/freebsd/crochet/issues/141

Any suggestions?


I'll reply directly to github thanks for pointing me to the ticket

Best regards,
Bapt

As people might only follow this thread and not the ticket here is what I
answered:

pkg supports an option for that which is pkg -o NAMESERVER= to avoid having to
copy resolv.conf it was broken in 1.8 but I fixed it in pkg 1.8.0 which has been
released today.

the problem with pkg -c is that it calls chroot very early. To avoid that
problem we have added pkg -r which does not perform any chroot at all therefore
having not network issue, but the ports tree are is not yet entirely aware of it
and some scripts (preinstall/postinstall) might cause some issues.
at least creating users from the script is safe in that regard. for most simple
case that should work.


As Bapt knows (I've been harassing him) pkg needs an "install all this 
over there" mode.
The answer is "-r" but it does suffer from the fact that it requires 
postinstall scripts to cooperate
by knowing to look for the appropriate environment value (I forget its 
name).


I've used three different methods to get around this..
1/ use -c and prepopulate it with a copy of /rescue, and copying all 
the pkg files I need in first into /pkg, which I delete afterwards.
I'd like clarification in the Man page about setting a good $PATH in 
the chrooted part of pkg currently I just copy /rescue into /bin but 
pkg seems to sometimes use full paths for things so that meant also 
/usr/bin and usr/sbin. (or maybe that was an install script.).
(in my application I could just link all 4 locations to be the same 
place).


2/ use   PKG_DBDIR=$(TARGET_DIR)/var/db/pkg pkg add --relocate 
$(TARGET_DIR) $(PKGFILE)

Almost the same as -r from my perspective.

3/ because I need to work as non-root, and pkg doesn't have a -u and 
-g option like chroot does, there are times when I need to do:


sudo chroot -u $ME -g $MYGROUP sh -c "INSTALL_AS_USER pkg add -f -M 
$(PKGFILE)"


instead of using pkg -c.

I worked up a patch for -u and -g but lost it.. it was only about 20 
lines including the usage() change.  I'm sure bapt could redo it much 
better.

using -u automatically enabled install_as_user.

I was thinking that in order to do this properly the chrooted child 
should do all it's fetch requests etc via the non-chrooted parent, but 
that would have probably been a bit too complicated. (including add 
requests)







Best regards,
Bapt



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

documentation looking for a home. (kernel modules and Vimage)

[Julian Elischer]

Sorry for double posting.. this email addresses two separate groups of 
people.


Some time ago I wrote the following.

http://p4web.freebsd.org/@md=d=//depot/projects/vimage/=//depot/projects/vimage/porting_to_vimage.txt=s0G@//depot/projects/vimage/porting_to_vimage.txt?ac=64=18

It really should get split into bits and put into the handbook or 
something..


It includes a section that is all about the module load inits that is 
not specific to vimage, and there is also  a good description of 
vimage in the that the handbook could use somewhere, maybe in the 
developer's hand book.



Firstly are there any doc people that can take a look at it and see if 
it would fit in somewhere?  (Do we have an editor in chief?)  and also 
maybe some kernel people who can go over the part that describes how 
modules are loaded and check that it is still up to date.. I last 
changed it in 2010 so it could be in need of updating.


If you know about kerle module loading please take a look and see if 
anything stands out to you.. It is in the second half of the doc.


Also anyone lookign at vimage might find it worth a look.


Julian


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"