Re: FreeBSD head -r341836 amd64->aarch64 cross-build of -r484783 ports via poudriere: devel/qt5-testlib hung-up during "Checking for POSIX monotonic clock"

[Mark Millard]

[I attached to the hung-up process with gdb and looked around a little.]

On 2018-Dec-19, at 13:58, Mark Millard  wrote:

> [Looks like a race or some such for devel/qt5-testlib: retry of 
> poudreire-devel
> did not hang. The other hang-up seems to be repeating and I give some 
> details.]
> 
> On 2018-Dec-19, at 12:20, Mark Millard  wrote:
> 
>> FYI: Based on FreeBSD head -r341836 (host and target) and ports -r484783 . 
>> This
>> was a rebuild based on going from perl5.26 to perl5.28 without updating the 
>> ports
>> tree and from system clang 6 for the prior FreeBSD-head context used to 
>> clang 7
>> this time. (I'm not attributing causes here.) poudriere was using 
>> amd64-native
>> tools for speeding up the cross-build.
>> 
>> # grep -r =perl5= /etc/ ~/src.configs/ /usr/local/etc/
>> /etc/make.conf:DEFAULT_VERSIONS+=perl5=5.28 gcc=8
>> /usr/local/etc/poudriere.d/make.conf:DEFAULT_VERSIONS+=perl5=5.28 gcc=8
>> 
>> There was also a "print/texinfo:configure/runaway" but I've not looked into
>> it at all yet and it may be a while before I do. The other ports attempted
>> built fine as far as I can tell so far.
>> 
>> 
>> The devel/qt5-testlib failure looks like:
>> 
>> [00:00:13] Building 123 packages using 28 builders
>> . . .
>> [00:49:30] [10] [00:00:00] Building devel/qt5-testlib | qt5-testlib-5.11.2
>> . . .
>> [07:31:31] [10] [06:42:01] Saved devel/qt5-testlib | qt5-testlib-5.11.2 
>> wrkdir to: 
>> /usr/local/poudriere/data/wrkdirs/FBSDFSSDjailCortexA57-default/default/qt5-testlib-5.11.2.tar
>> [07:31:32] [10] [06:42:02] Finished devel/qt5-testlib | qt5-testlib-5.11.2: 
>> Failed: configure/runaway
>> 
>> With logs/errors/qt5-testlib-5.11.2.log showing:
>> 
>> Checking for POSIX monotonic clock... 
>> + cd 
>> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
>>  && 
>> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/bin/qmake
>>  "CONFIG -= qt debug_and_release app_bundle lib_bundle" "CONFIG += shared 
>> warn_off console single_arch" 
>> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
>> + cd 
>> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
>>  && MAKEFLAGS= make
>> =>> Killing runaway build after 21600 seconds with no output
>> =>> Cleaning up wrkdir
>> ===>  Cleaning for qt5-testlib-5.11.2
>> Killed
>> build of devel/qt5-testlib | qt5-testlib-5.11.2 ended at Wed Dec 19 06:45:42 
>> PST 2018
>> build time: 06:41:46
>> !!! build failure encountered !!!
>> 
>> 
>> # less 
>> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.log
>> . . .
>> test config.qtbase_corelib.libraries.librt succeeded
>> executing config test clock-monotonic
>> + cd 
>> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
>>  && 
>> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/bin/qmake
>>  "CONFIG -= qt debug_and_release app_bundle lib_bundle" "CONFIG += shared 
>> warn_off console single_arch" 
>> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
>> + cd 
>> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
>>  && MAKEFLAGS= make
>> 
>> 
>> Some supporting details of context:
>> 
>> # uname -apKU
>> FreeBSD FBSDFSSD 13.0-CURRENT FreeBSD 13.0-CURRENT #5 r341836M: Tue Dec 11 
>> 16:37:42 PST 2018 
>> markmi@FBSDFSSD:/usr/obj/amd64_clang/amd64.amd64/usr/src/amd64.amd64/sys/GENERIC-NODBG
>>   amd64 amd64 135 135
>> 
>> # svnlite info /usr/ports/ | grep "Re[plv]"
>> Relative URL: ^/head
>> Repository Root: svn://svn.freebsd.org/ports
>> Repository UUID: 35697150-7ecd-e111-bb59-0022644237b5
>> Revision: 484783
>> Last Changed Rev: 484783
>> 
> 
> I started poudriere up again with just the 2 needing to be rebuilt (plus
> what depends on the 2). devel/qt5-testlib quickly completed just fine:
> 
> [00:02:16] [02] [00:00:00] Building devel/qt5-testlib | qt5-testlib-5.11.2
> [00:04:54] [02] [00:02:38] Finished devel/qt5-testlib | qt5-testlib-5.11.2: 
> Success
> 
> 
> In the prior build that had the hang-ups I looked and dor print/texinfo :
> 
> /wrkdirs/usr/ports/print/texinfo/work/texinfo-6.5/config.log shows for its
> hang-up:
> 
> . . .
> configure:6639: checking for alloca
> configure:6676: /nxb-bin/usr/bin/cc -o conftest -O2 -pipe -mcpu=cortex-a57  
> -DLIBICONV_PLUG -g -fno-strict-aliasing  -mcpu=cortex-a57 -DLIBICONV_PLUG 
> -D_THREAD_SAFE   conftest.c  >&5
> configure:6676: $? = 0
> configure:6684: result: yes
> configure:6794: checking for C/C++ restrict keyword
> configure:6821: /nxb-bin/usr/bin/cc -c -O2 -pipe -mcpu=cortex-a57  
> -DLIBICONV_PLUG -g -fno-strict-aliasing  -mcpu=cortex-a57 -DLIBICONV_PLUG 
> -D_THREAD_SAFE conftest.c >&5
> configure:6821: $? = 0
> configure:6829: result: __restrict
> configure:6844: checking whether 

Re: The future of ZFS in FreeBSD

[Matthew Macy]

On Wed, Dec 19, 2018 at 15:11 Steven Hartland 
wrote:

> Sorry been off for a few weeks so must have missed that, please do prod me
> on again if you don’t see any response to anything not just this. Like many
> others I get so may emails across so many lists it’s more than likely I
> just missed it.
>
> That said would you say that with the right support we can make progress
> on the this prior to the port? I have to ask as the alternative version has
> been on the cusp for many years now so it’s feels more like a distant
> memory than something that may happen, no disrespect to anyone involved, as
> I know all too well how hard it can be to get something like this over the
> line, especially when people have competing priorities.
>

I am hoping that it's sufficiently important to FreeBSD ZFS developers that
they'll give the PR the attention it needs so that it can be merged before
summer. My understanding is that it's mostly suffered from neglect. TRIM is
most important to FreeBSD and it already had its own implementation.

https://github.com/zfsonlinux/zfs/pull/5925

I forwarded you the private communication again as well.

-M


>
> On Wed, 19 Dec 2018 at 22:52, Matthew Macy  wrote:
>
>>
>>
>> On Wed, Dec 19, 2018 at 14:47 Steven Hartland 
>> wrote:
>>
>>> Thanks for the write up most appreciated. One of the more meaty
>>> differences is that FreeBSD ZFS still has the only merged and production
>>> ready TRIM support so my question would be are their any plans to address
>>> this before creating the new port as going back to a world without TRIM
>>> support wouldn’t be something I’d look forward to.
>>>
>>
>> Well, then please follow up on the request I CC'd you on a week ago
>> asking that you engage on the deadlist based TRIM  PR. That's a better
>> forum for discussing these details than lamenting in public lists.
>>
>> Thanks.
>>
>> -M
>>
>>
>>
>>
>>> On Wed, 19 Dec 2018 at 06:51, Matthew Macy  wrote:
>>>
 The sources for FreeBSD's ZFS support are currently taken directly
 from Illumos with local ifdefs to support the peculiarities of FreeBSD
 where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
 has regularly pulled changes from Illumos and tried to push back any
 bug fixes and new features done in the context of FreeBSD. In the past
 few years the vast majority of new development in ZFS has taken place
 in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
 that they will be moving to ZoL
 https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
 that there will be little to no net new development of Illumos. While
 working through the git history of ZoL I have also discovered that
 many races and locking bugs have been fixed in ZoL and never made it
 back to Illumos and thus FreeBSD. This state of affairs has led to a
 general agreement among the stakeholders that I have spoken to that it
 makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
 has graciously encouraged me to add FreeBSD support directly to ZoL
 https://github.com/zfsonfreebsd/ZoF so that we might all have a single
 shared code base.

 A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
 Before it can be committed some additional functionality needs to be
 added to the FreeBSD opencrypto framework. These can be found at
 https://reviews.freebsd.org/D18520

 This port will provide FreeBSD users with multi modifier protection,
 project quotas, encrypted datasets, allocation classes, vectorized
 raidz, vectorized checksums, and various command line improvements.

 Before ZoF can be merged back in to ZoL several steps need to be taken:
 - Integrate FreeBSD support into ZoL CI
 - Have most of the ZFS test suite passing
 - Complete additional QA testing at iX

 We at iX Systems need to port ZoL's EC2 CI scripts to work with
 FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes.
 Being integrated in to their CI will mean that, among other things,
 most integration issues will be caught before a PR is merged upstream
 rather than many months later when it is MFVed into FreeBSD. I’m
 hoping to submit the PR to ZoL some time in January.

 This port will make it easy for end users on a range of releases to
 run the latest version of ZFS. Nonetheless, transitioning away from an
 Illumos based ZFS is not likely to be entirely seamless. The
 stakeholders I’ve spoken to all agree that this is the best path
 forward but some degree of effort needs to be made to accommodate
 downstream consumers. The current plan is to import ZoF and unhook the
 older Illumos based sources from the build on April 15th or two months
 after iX systems QA deems ZoF stable - which ever comes later. The
 Illumos based sources will be removed some time later - but well
 before 13. This will 

Re: The future of ZFS in FreeBSD

[Steven Hartland]

Sorry been off for a few weeks so must have missed that, please do prod me
on again if you don’t see any response to anything not just this. Like many
others I get so may emails across so many lists it’s more than likely I
just missed it.

That said would you say that with the right support we can make progress on
the this prior to the port? I have to ask as the alternative version has
been on the cusp for many years now so it’s feels more like a distant
memory than something that may happen, no disrespect to anyone involved, as
I know all too well how hard it can be to get something like this over the
line, especially when people have competing priorities.


On Wed, 19 Dec 2018 at 22:52, Matthew Macy  wrote:

>
>
> On Wed, Dec 19, 2018 at 14:47 Steven Hartland 
> wrote:
>
>> Thanks for the write up most appreciated. One of the more meaty
>> differences is that FreeBSD ZFS still has the only merged and production
>> ready TRIM support so my question would be are their any plans to address
>> this before creating the new port as going back to a world without TRIM
>> support wouldn’t be something I’d look forward to.
>>
>
> Well, then please follow up on the request I CC'd you on a week ago asking
> that you engage on the deadlist based TRIM  PR. That's a better forum for
> discussing these details than lamenting in public lists.
>
> Thanks.
>
> -M
>
>
>
>
>> On Wed, 19 Dec 2018 at 06:51, Matthew Macy  wrote:
>>
>>> The sources for FreeBSD's ZFS support are currently taken directly
>>> from Illumos with local ifdefs to support the peculiarities of FreeBSD
>>> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
>>> has regularly pulled changes from Illumos and tried to push back any
>>> bug fixes and new features done in the context of FreeBSD. In the past
>>> few years the vast majority of new development in ZFS has taken place
>>> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
>>> that they will be moving to ZoL
>>> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
>>> that there will be little to no net new development of Illumos. While
>>> working through the git history of ZoL I have also discovered that
>>> many races and locking bugs have been fixed in ZoL and never made it
>>> back to Illumos and thus FreeBSD. This state of affairs has led to a
>>> general agreement among the stakeholders that I have spoken to that it
>>> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
>>> has graciously encouraged me to add FreeBSD support directly to ZoL
>>> https://github.com/zfsonfreebsd/ZoF so that we might all have a single
>>> shared code base.
>>>
>>> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
>>> Before it can be committed some additional functionality needs to be
>>> added to the FreeBSD opencrypto framework. These can be found at
>>> https://reviews.freebsd.org/D18520
>>>
>>> This port will provide FreeBSD users with multi modifier protection,
>>> project quotas, encrypted datasets, allocation classes, vectorized
>>> raidz, vectorized checksums, and various command line improvements.
>>>
>>> Before ZoF can be merged back in to ZoL several steps need to be taken:
>>> - Integrate FreeBSD support into ZoL CI
>>> - Have most of the ZFS test suite passing
>>> - Complete additional QA testing at iX
>>>
>>> We at iX Systems need to port ZoL's EC2 CI scripts to work with
>>> FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes.
>>> Being integrated in to their CI will mean that, among other things,
>>> most integration issues will be caught before a PR is merged upstream
>>> rather than many months later when it is MFVed into FreeBSD. I’m
>>> hoping to submit the PR to ZoL some time in January.
>>>
>>> This port will make it easy for end users on a range of releases to
>>> run the latest version of ZFS. Nonetheless, transitioning away from an
>>> Illumos based ZFS is not likely to be entirely seamless. The
>>> stakeholders I’ve spoken to all agree that this is the best path
>>> forward but some degree of effort needs to be made to accommodate
>>> downstream consumers. The current plan is to import ZoF and unhook the
>>> older Illumos based sources from the build on April 15th or two months
>>> after iX systems QA deems ZoF stable - which ever comes later. The
>>> Illumos based sources will be removed some time later - but well
>>> before 13. This will give users a 3 month period during which both the
>>> port and legacy Illumos based ZFS will be available to users. Pools
>>> should interoperate between ZoF and legacy provided the user does not
>>> enable any features available only in ZoF. We will try to accommodate
>>> any downstream consumers in the event that they need that date pushed
>>> back. We ask that any downstream consumers who are particularly
>>> sensitive to changes start testing the port when it is formally
>>> announced and report back any issues they have. I will do my best to
>>> 

Re: The future of ZFS in FreeBSD

[Matthew Macy]

On Wed, Dec 19, 2018 at 14:47 Steven Hartland 
wrote:

> Thanks for the write up most appreciated. One of the more meaty
> differences is that FreeBSD ZFS still has the only merged and production
> ready TRIM support so my question would be are their any plans to address
> this before creating the new port as going back to a world without TRIM
> support wouldn’t be something I’d look forward to.
>

Well, then please follow up on the request I CC'd you on a week ago asking
that you engage on the deadlist based TRIM  PR. That's a better forum for
discussing these details than lamenting in public lists.

Thanks.

-M




> On Wed, 19 Dec 2018 at 06:51, Matthew Macy  wrote:
>
>> The sources for FreeBSD's ZFS support are currently taken directly
>> from Illumos with local ifdefs to support the peculiarities of FreeBSD
>> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
>> has regularly pulled changes from Illumos and tried to push back any
>> bug fixes and new features done in the context of FreeBSD. In the past
>> few years the vast majority of new development in ZFS has taken place
>> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
>> that they will be moving to ZoL
>> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
>> that there will be little to no net new development of Illumos. While
>> working through the git history of ZoL I have also discovered that
>> many races and locking bugs have been fixed in ZoL and never made it
>> back to Illumos and thus FreeBSD. This state of affairs has led to a
>> general agreement among the stakeholders that I have spoken to that it
>> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
>> has graciously encouraged me to add FreeBSD support directly to ZoL
>> https://github.com/zfsonfreebsd/ZoF so that we might all have a single
>> shared code base.
>>
>> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
>> Before it can be committed some additional functionality needs to be
>> added to the FreeBSD opencrypto framework. These can be found at
>> https://reviews.freebsd.org/D18520
>>
>> This port will provide FreeBSD users with multi modifier protection,
>> project quotas, encrypted datasets, allocation classes, vectorized
>> raidz, vectorized checksums, and various command line improvements.
>>
>> Before ZoF can be merged back in to ZoL several steps need to be taken:
>> - Integrate FreeBSD support into ZoL CI
>> - Have most of the ZFS test suite passing
>> - Complete additional QA testing at iX
>>
>> We at iX Systems need to port ZoL's EC2 CI scripts to work with
>> FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes.
>> Being integrated in to their CI will mean that, among other things,
>> most integration issues will be caught before a PR is merged upstream
>> rather than many months later when it is MFVed into FreeBSD. I’m
>> hoping to submit the PR to ZoL some time in January.
>>
>> This port will make it easy for end users on a range of releases to
>> run the latest version of ZFS. Nonetheless, transitioning away from an
>> Illumos based ZFS is not likely to be entirely seamless. The
>> stakeholders I’ve spoken to all agree that this is the best path
>> forward but some degree of effort needs to be made to accommodate
>> downstream consumers. The current plan is to import ZoF and unhook the
>> older Illumos based sources from the build on April 15th or two months
>> after iX systems QA deems ZoF stable - which ever comes later. The
>> Illumos based sources will be removed some time later - but well
>> before 13. This will give users a 3 month period during which both the
>> port and legacy Illumos based ZFS will be available to users. Pools
>> should interoperate between ZoF and legacy provided the user does not
>> enable any features available only in ZoF. We will try to accommodate
>> any downstream consumers in the event that they need that date pushed
>> back. We ask that any downstream consumers who are particularly
>> sensitive to changes start testing the port when it is formally
>> announced and report back any issues they have. I will do my best to
>> ensure that this message is communicated to all those who it may
>> concern. However, I can’t ensure that everyone reads these lists. That
>> is the responsibility of -CURRENT users.
>>
>> -M
>>
> ___
>> freebsd...@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to "freebsd-fs-unsubscr...@freebsd.org"
>>
>
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Steven Hartland]

Thanks for the write up most appreciated. One of the more meaty differences
is that FreeBSD ZFS still has the only merged and production ready TRIM
support so my question would be are their any plans to address this before
creating the new port as going back to a world without TRIM support
wouldn’t be something I’d look forward to.

On Wed, 19 Dec 2018 at 06:51, Matthew Macy  wrote:

> The sources for FreeBSD's ZFS support are currently taken directly
> from Illumos with local ifdefs to support the peculiarities of FreeBSD
> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
> has regularly pulled changes from Illumos and tried to push back any
> bug fixes and new features done in the context of FreeBSD. In the past
> few years the vast majority of new development in ZFS has taken place
> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
> that they will be moving to ZoL
> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
> that there will be little to no net new development of Illumos. While
> working through the git history of ZoL I have also discovered that
> many races and locking bugs have been fixed in ZoL and never made it
> back to Illumos and thus FreeBSD. This state of affairs has led to a
> general agreement among the stakeholders that I have spoken to that it
> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
> has graciously encouraged me to add FreeBSD support directly to ZoL
> https://github.com/zfsonfreebsd/ZoF so that we might all have a single
> shared code base.
>
> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
> Before it can be committed some additional functionality needs to be
> added to the FreeBSD opencrypto framework. These can be found at
> https://reviews.freebsd.org/D18520
>
> This port will provide FreeBSD users with multi modifier protection,
> project quotas, encrypted datasets, allocation classes, vectorized
> raidz, vectorized checksums, and various command line improvements.
>
> Before ZoF can be merged back in to ZoL several steps need to be taken:
> - Integrate FreeBSD support into ZoL CI
> - Have most of the ZFS test suite passing
> - Complete additional QA testing at iX
>
> We at iX Systems need to port ZoL's EC2 CI scripts to work with
> FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes.
> Being integrated in to their CI will mean that, among other things,
> most integration issues will be caught before a PR is merged upstream
> rather than many months later when it is MFVed into FreeBSD. I’m
> hoping to submit the PR to ZoL some time in January.
>
> This port will make it easy for end users on a range of releases to
> run the latest version of ZFS. Nonetheless, transitioning away from an
> Illumos based ZFS is not likely to be entirely seamless. The
> stakeholders I’ve spoken to all agree that this is the best path
> forward but some degree of effort needs to be made to accommodate
> downstream consumers. The current plan is to import ZoF and unhook the
> older Illumos based sources from the build on April 15th or two months
> after iX systems QA deems ZoF stable - which ever comes later. The
> Illumos based sources will be removed some time later - but well
> before 13. This will give users a 3 month period during which both the
> port and legacy Illumos based ZFS will be available to users. Pools
> should interoperate between ZoF and legacy provided the user does not
> enable any features available only in ZoF. We will try to accommodate
> any downstream consumers in the event that they need that date pushed
> back. We ask that any downstream consumers who are particularly
> sensitive to changes start testing the port when it is formally
> announced and report back any issues they have. I will do my best to
> ensure that this message is communicated to all those who it may
> concern. However, I can’t ensure that everyone reads these lists. That
> is the responsibility of -CURRENT users.
>
> -M
> ___
> freebsd...@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscr...@freebsd.org"
>
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FreeBSD head -r341836 amd64->aarch64 cross-build of -r484783 ports via poudriere: devel/qt5-testlib hung-up during "Checking for POSIX monotonic clock"

[Mark Millard]

[Looks like a race or some such for devel/qt5-testlib: retry of poudreire-devel
did not hang. The other hang-up seems to be repeating and I give some details.]

On 2018-Dec-19, at 12:20, Mark Millard  wrote:

> FYI: Based on FreeBSD head -r341836 (host and target) and ports -r484783 . 
> This
> was a rebuild based on going from perl5.26 to perl5.28 without updating the 
> ports
> tree and from system clang 6 for the prior FreeBSD-head context used to clang 
> 7
> this time. (I'm not attributing causes here.) poudriere was using amd64-native
> tools for speeding up the cross-build.
> 
> # grep -r =perl5= /etc/ ~/src.configs/ /usr/local/etc/
> /etc/make.conf:DEFAULT_VERSIONS+=perl5=5.28 gcc=8
> /usr/local/etc/poudriere.d/make.conf:DEFAULT_VERSIONS+=perl5=5.28 gcc=8
> 
> There was also a "print/texinfo:configure/runaway" but I've not looked into
> it at all yet and it may be a while before I do. The other ports attempted
> built fine as far as I can tell so far.
> 
> 
> The devel/qt5-testlib failure looks like:
> 
> [00:00:13] Building 123 packages using 28 builders
> . . .
> [00:49:30] [10] [00:00:00] Building devel/qt5-testlib | qt5-testlib-5.11.2
> . . .
> [07:31:31] [10] [06:42:01] Saved devel/qt5-testlib | qt5-testlib-5.11.2 
> wrkdir to: 
> /usr/local/poudriere/data/wrkdirs/FBSDFSSDjailCortexA57-default/default/qt5-testlib-5.11.2.tar
> [07:31:32] [10] [06:42:02] Finished devel/qt5-testlib | qt5-testlib-5.11.2: 
> Failed: configure/runaway
> 
> With logs/errors/qt5-testlib-5.11.2.log showing:
> 
> Checking for POSIX monotonic clock... 
> + cd 
> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
>  && 
> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/bin/qmake
>  "CONFIG -= qt debug_and_release app_bundle lib_bundle" "CONFIG += shared 
> warn_off console single_arch" 
> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
> + cd 
> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
>  && MAKEFLAGS= make
> =>> Killing runaway build after 21600 seconds with no output
> =>> Cleaning up wrkdir
> ===>  Cleaning for qt5-testlib-5.11.2
> Killed
> build of devel/qt5-testlib | qt5-testlib-5.11.2 ended at Wed Dec 19 06:45:42 
> PST 2018
> build time: 06:41:46
> !!! build failure encountered !!!
> 
> 
> # less 
> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.log
> . . .
> test config.qtbase_corelib.libraries.librt succeeded
> executing config test clock-monotonic
> + cd 
> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
>  && 
> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/bin/qmake
>  "CONFIG -= qt debug_and_release app_bundle lib_bundle" "CONFIG += shared 
> warn_off console single_arch" 
> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
> + cd 
> /wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
>  && MAKEFLAGS= make
> 
> 
> Some supporting details of context:
> 
> # uname -apKU
> FreeBSD FBSDFSSD 13.0-CURRENT FreeBSD 13.0-CURRENT #5 r341836M: Tue Dec 11 
> 16:37:42 PST 2018 
> markmi@FBSDFSSD:/usr/obj/amd64_clang/amd64.amd64/usr/src/amd64.amd64/sys/GENERIC-NODBG
>   amd64 amd64 135 135
> 
> # svnlite info /usr/ports/ | grep "Re[plv]"
> Relative URL: ^/head
> Repository Root: svn://svn.freebsd.org/ports
> Repository UUID: 35697150-7ecd-e111-bb59-0022644237b5
> Revision: 484783
> Last Changed Rev: 484783
> 

I started poudriere up again with just the 2 needing to be rebuilt (plus
what depends on the 2). devel/qt5-testlib quickly completed just fine:

[00:02:16] [02] [00:00:00] Building devel/qt5-testlib | qt5-testlib-5.11.2
[00:04:54] [02] [00:02:38] Finished devel/qt5-testlib | qt5-testlib-5.11.2: 
Success


In the prior build that had the hang-ups I looked and dor print/texinfo :

/wrkdirs/usr/ports/print/texinfo/work/texinfo-6.5/config.log shows for its
hang-up:

. . .
configure:6639: checking for alloca
configure:6676: /nxb-bin/usr/bin/cc -o conftest -O2 -pipe -mcpu=cortex-a57  
-DLIBICONV_PLUG -g -fno-strict-aliasing  -mcpu=cortex-a57 -DLIBICONV_PLUG 
-D_THREAD_SAFE   conftest.c  >&5
configure:6676: $? = 0
configure:6684: result: yes
configure:6794: checking for C/C++ restrict keyword
configure:6821: /nxb-bin/usr/bin/cc -c -O2 -pipe -mcpu=cortex-a57  
-DLIBICONV_PLUG -g -fno-strict-aliasing  -mcpu=cortex-a57 -DLIBICONV_PLUG 
-D_THREAD_SAFE conftest.c >&5
configure:6821: $? = 0
configure:6829: result: __restrict
configure:6844: checking whether // is distinct from /


In the poudriere re-run print/texinfo seems to be not progressing:

root   879130.0  0.0  12920  3668  0  I13:29   0:00.06 | |  
 `-- sh: poudriere[FBSDFSSDjailCortexA57-default][01]: build_pkg 
(texinfo-6.5_1,1) (sh)
root   88869  

FreeBSD head -r341836 amd64->aarch64 cross-build of -r484783 ports via poudriere: devel/qt5-testlib hung-up during "Checking for POSIX monotonic clock"

[Mark Millard]

FYI: Based on FreeBSD head -r341836 (host and target) and ports -r484783 . This
was a rebuild based on going from perl5.26 to perl5.28 without updating the 
ports
tree and from system clang 6 for the prior FreeBSD-head context used to clang 7
this time. (I'm not attributing causes here.) poudriere was using amd64-native
tools for speeding up the cross-build.

# grep -r =perl5= /etc/ ~/src.configs/ /usr/local/etc/
/etc/make.conf:DEFAULT_VERSIONS+=perl5=5.28 gcc=8
/usr/local/etc/poudriere.d/make.conf:DEFAULT_VERSIONS+=perl5=5.28 gcc=8

There was also a "print/texinfo:configure/runaway" but I've not looked into
it at all yet and it may be a while before I do. The other ports attempted
built fine as far as I can tell so far.


The devel/qt5-testlib failure looks like:

[00:00:13] Building 123 packages using 28 builders
. . .
[00:49:30] [10] [00:00:00] Building devel/qt5-testlib | qt5-testlib-5.11.2
. . .
[07:31:31] [10] [06:42:01] Saved devel/qt5-testlib | qt5-testlib-5.11.2 wrkdir 
to: 
/usr/local/poudriere/data/wrkdirs/FBSDFSSDjailCortexA57-default/default/qt5-testlib-5.11.2.tar
[07:31:32] [10] [06:42:02] Finished devel/qt5-testlib | qt5-testlib-5.11.2: 
Failed: configure/runaway

With logs/errors/qt5-testlib-5.11.2.log showing:

Checking for POSIX monotonic clock... 
+ cd 
/wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
 && 
/wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/bin/qmake
 "CONFIG -= qt debug_and_release app_bundle lib_bundle" "CONFIG += shared 
warn_off console single_arch" 
/wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
+ cd 
/wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
 && MAKEFLAGS= make
=>> Killing runaway build after 21600 seconds with no output
=>> Cleaning up wrkdir
===>  Cleaning for qt5-testlib-5.11.2
Killed
build of devel/qt5-testlib | qt5-testlib-5.11.2 ended at Wed Dec 19 06:45:42 
PST 2018
build time: 06:41:46
!!! build failure encountered !!!


# less 
/wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.log
. . .
test config.qtbase_corelib.libraries.librt succeeded
executing config test clock-monotonic
+ cd 
/wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
 && 
/wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/bin/qmake
 "CONFIG -= qt debug_and_release app_bundle lib_bundle" "CONFIG += shared 
warn_off console single_arch" 
/wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
+ cd 
/wrkdirs/usr/ports/devel/qt5-testlib/work/qtbase-everywhere-src-5.11.2/config.tests/clock-monotonic
 && MAKEFLAGS= make


Some supporting details of context:

# uname -apKU
FreeBSD FBSDFSSD 13.0-CURRENT FreeBSD 13.0-CURRENT #5 r341836M: Tue Dec 11 
16:37:42 PST 2018 
markmi@FBSDFSSD:/usr/obj/amd64_clang/amd64.amd64/usr/src/amd64.amd64/sys/GENERIC-NODBG
  amd64 amd64 135 135

# svnlite info /usr/ports/ | grep "Re[plv]"
Relative URL: ^/head
Repository Root: svn://svn.freebsd.org/ports
Repository UUID: 35697150-7ecd-e111-bb59-0022644237b5
Revision: 484783
Last Changed Rev: 484783


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Composite PCI devices in FreeBSD (mfd in Linux)

[Ian Lepore]

On Wed, 2018-12-19 at 14:35 -0500, Anthony Jenkins wrote:
> On 12/19/18 10:41 AM, Anthony Jenkins wrote:
> > 
> > [snip]
> > 
> > I'm not feeling too confident about the condition of the FreeBSD
> > ig4 
> > driver; the PCI attach code was calling pci_alloc_msi() wrong,
> > passing 
> > a pointer to the rid (0) instead of a pointer to a count variable,
> > and 
> > not passing bus_alloc_resource_any() an IRQ rid > 0 if it has an
> > MSI.  
> > I'd be happy(er) if ig4 created a /dev/iic0 node - I figured
> > iicbus(4) 
> > took care of all that...
> > 
> > https://github.com/ScoobiFreeBSD/freebsd-intel-lpss
> > 
> Found it!  I didn't declare ig4_iic to include the ig4_lpss as a 
> sub-device.  Now it at least /looks/ like I'm getting I2C devices
> found 
> on both my DesignWare I2C busses.
> 
> diff --git a/sys/dev/ichiic/ig4_iic.c b/sys/dev/ichiic/ig4_iic.c
> index 6bbe417..34c1adb 100644
> --- a/sys/dev/ichiic/ig4_iic.c
> +++ b/sys/dev/ichiic/ig4_iic.c
> @@ -802,3 +802,4 @@ ig4iic_dump(ig4iic_softc_t *sc)
> 
>   DRIVER_MODULE(iicbus, ig4iic_acpi, iicbus_driver, iicbus_devclass, 
> NULL, NULL);
>   DRIVER_MODULE(iicbus, ig4iic_pci, iicbus_driver, iicbus_devclass, 
> NULL, NULL);
> +DRIVER_MODULE(iicbus, ig4iic_lpss, iicbus_driver, iicbus_devclass, 
> NULL, NULL);

That new DRIVER_MODULE() statement should be in your new driver, not in
ig4_iic.c. Those other two statements should also be moved into their
corresponding source code files.

At least, that's the precedent followed by all the i2c controller
drivers except ig4_iic: the DRIVER_MODULE() statement that establishes
the connection between iicbus and the controller lives right alongside
the DRIVER_MODULE() statement that establishes the connection between
the controller and the bus it sits on. See, for example:

 sys/dev/glxiic/glxiic.c
 sys/dev/iicbus/iicoc.c

-- Ian

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Composite PCI devices in FreeBSD (mfd in Linux)

[Anthony Jenkins]

On 12/19/18 10:41 AM, Anthony Jenkins wrote:

[snip]

I'm not feeling too confident about the condition of the FreeBSD ig4 
driver; the PCI attach code was calling pci_alloc_msi() wrong, passing 
a pointer to the rid (0) instead of a pointer to a count variable, and 
not passing bus_alloc_resource_any() an IRQ rid > 0 if it has an MSI.  
I'd be happy(er) if ig4 created a /dev/iic0 node - I figured iicbus(4) 
took care of all that...


https://github.com/ScoobiFreeBSD/freebsd-intel-lpss

Found it!  I didn't declare ig4_iic to include the ig4_lpss as a 
sub-device.  Now it at least /looks/ like I'm getting I2C devices found 
on both my DesignWare I2C busses.


diff --git a/sys/dev/ichiic/ig4_iic.c b/sys/dev/ichiic/ig4_iic.c
index 6bbe417..34c1adb 100644
--- a/sys/dev/ichiic/ig4_iic.c
+++ b/sys/dev/ichiic/ig4_iic.c
@@ -802,3 +802,4 @@ ig4iic_dump(ig4iic_softc_t *sc)

 DRIVER_MODULE(iicbus, ig4iic_acpi, iicbus_driver, iicbus_devclass, 
NULL, NULL);
 DRIVER_MODULE(iicbus, ig4iic_pci, iicbus_driver, iicbus_devclass, 
NULL, NULL);
+DRIVER_MODULE(iicbus, ig4iic_lpss, iicbus_driver, iicbus_devclass, 
NULL, NULL);


ajenkins@ajenkins-delllaptop ~/Projects/freebsd-intel-lpss (master)
$ ls /dev/ii*
/dev/iic0  /dev/iic1

ajenkins@ajenkins-delllaptop ~/Projects/freebsd-intel-lpss (master)
$ for i2cbus in iic0 iic1; do sudo i2c -s -f "/dev/${i2cbus}"; done
Hardware may not support START/STOP scanning; trying less-reliable read 
method.

Scanning I2C devices on /dev/iic0: 0a
Hardware may not support START/STOP scanning; trying less-reliable read 
method.

Scanning I2C devices on /dev/iic1: 2c

Anthony
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Pete Wright]

On 12/19/18 10:32 AM, Allan Jude wrote:


The biggest thing to remember is that this is still OpenZFS, and still
run by the same developers as it has been. We are just commonizing on
the repo that has the most features integrated into it.


thanks for the clarification on this, as i was a bit confused as well. :)

One question - is there a plan to rename things to not be linux 
specific, especially since it seems that this effort will result in the 
original goal of the OpenZFS repro (if I am reading things correctly)?  
Or is that not going to happen because ZoL has the most commercial backing?


It's probably a mute point - but I do take pride in the fact that 
FreeBSD has had support ZFS for so long and has allowed me to do things 
for my customers that I can't do on Linux...so I guess what I'm saying 
is I hope we don't loose credit for the hard work we've put into getting 
ZFS mainstream over the years :)


Thanks!
-pete


--
Pete Wright
p...@nomadlogic.org
@nomadlogicLA

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Matthew Ahrens]

On Wed, Dec 19, 2018 at 8:35 AM Shawn Webb 
wrote:

> I'm curious what this means for OpenZFS.
>

OpenZFS will continue to be an umbrella project for coordinating all work
on open-source ZFS.  The primary activities of OpenZFS are the annual OpenZFS
Developer Summit
 and the
monthly OpenZFS Leadership Meetings

.
There is also an OpenZFS GitHub repo, which has identical code to illumos.
This is used to streamline the process of getting changes into illumos
(removing some of the burden of testing and the illumos RTI process).  This
repo will continue to exist as long as there's need for it and folks able
to keep the automation running.  The naming of this GitHub project was
perhaps unfortunate, since it is specific to illumos and not
platform-agnostic.

 I was under the impression that OpenZFS was the upstream for all the ZFS
> implementations (sans
> Oracle).


Being "upstream" is more a matter of degree than an absolute ordering.  In
the past, most ZFS features have originated in illumos, and those changes
have been ported to the other platforms (notably FreeBSD and ZoL).  Looking
forward, we see more features originating in ZoL.  The OpenZFS community
(including FreeBSD) is working to put in place technical processes to
ensure that new ZFS features are available on all platforms.  The original
post in this thread was a proposal for how to do that.

--matt
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Allan Jude]

On 2018-12-19 11:30, Shawn Webb wrote:
> On Tue, Dec 18, 2018 at 10:49:48PM -0800, Matthew Macy wrote:
>> The sources for FreeBSD's ZFS support are currently taken directly
>> from Illumos with local ifdefs to support the peculiarities of FreeBSD
>> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
>> has regularly pulled changes from Illumos and tried to push back any
>> bug fixes and new features done in the context of FreeBSD. In the past
>> few years the vast majority of new development in ZFS has taken place
>> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
>> that they will be moving to ZoL
>> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
>> that there will be little to no net new development of Illumos. While
>> working through the git history of ZoL I have also discovered that
>> many races and locking bugs have been fixed in ZoL and never made it
>> back to Illumos and thus FreeBSD. This state of affairs has led to a
>> general agreement among the stakeholders that I have spoken to that it
>> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
>> has graciously encouraged me to add FreeBSD support directly to ZoL
>> https://github.com/zfsonfreebsd/ZoF so that we might all have a single
>> shared code base.
>>
>> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
>> Before it can be committed some additional functionality needs to be
>> added to the FreeBSD opencrypto framework. These can be found at
>> https://reviews.freebsd.org/D18520
>>
>> This port will provide FreeBSD users with multi modifier protection,
>> project quotas, encrypted datasets, allocation classes, vectorized
>> raidz, vectorized checksums, and various command line improvements.
>>
>> Before ZoF can be merged back in to ZoL several steps need to be taken:
>> - Integrate FreeBSD support into ZoL CI
>> - Have most of the ZFS test suite passing
>> - Complete additional QA testing at iX
>>
>> We at iX Systems need to port ZoL's EC2 CI scripts to work with
>> FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes.
>> Being integrated in to their CI will mean that, among other things,
>> most integration issues will be caught before a PR is merged upstream
>> rather than many months later when it is MFVed into FreeBSD. I???m
>> hoping to submit the PR to ZoL some time in January.
>>
>> This port will make it easy for end users on a range of releases to
>> run the latest version of ZFS. Nonetheless, transitioning away from an
>> Illumos based ZFS is not likely to be entirely seamless. The
>> stakeholders I???ve spoken to all agree that this is the best path
>> forward but some degree of effort needs to be made to accommodate
>> downstream consumers. The current plan is to import ZoF and unhook the
>> older Illumos based sources from the build on April 15th or two months
>> after iX systems QA deems ZoF stable - which ever comes later. The
>> Illumos based sources will be removed some time later - but well
>> before 13. This will give users a 3 month period during which both the
>> port and legacy Illumos based ZFS will be available to users. Pools
>> should interoperate between ZoF and legacy provided the user does not
>> enable any features available only in ZoF. We will try to accommodate
>> any downstream consumers in the event that they need that date pushed
>> back. We ask that any downstream consumers who are particularly
>> sensitive to changes start testing the port when it is formally
>> announced and report back any issues they have. I will do my best to
>> ensure that this message is communicated to all those who it may
>> concern. However, I can???t ensure that everyone reads these lists. That
>> is the responsibility of -CURRENT users.
> 
> Hey Matt,
> 
> Thank you for your detailed and informative post. It really helps
> downstream consumers of FreeBSD.
> 
> I'm curious what this means for OpenZFS. I was under the impression
> that OpenZFS was the upstream for all the ZFS implementations (sans
> Oracle).
> 
> Thanks,
> 

The original goal of the OpenZFS project was to have a single OS
agnostic repo where all of the common code lives, to try to make it
easier to have all of the implementations be relatively in sync with
each other, and have feature parity and import compatibility.

For the last 5 years, this goal is been out of reach, because no one
could justify the effort and expense of maintaining the 'one true repo'
when no one would actually be using or benefiting from that repo directly.

Today, the OpenZFS repo is just a fork of the illumos-gate repo, but
where pull requests are accepted, and where previous Delphix employees
would deal with the process of trying to upstream patches to illumos.
This process has not worked well recently, as things have gotten stuck
waiting for 'merge advocates' in illumos.

The ZoF effort will move us closer to the goal of a common repo. The
general idea will be to have 'OS 

Re: The future of ZFS in FreeBSD

[Andrew Turner]

> On 19 Dec 2018, at 08:35, Matthew Macy  wrote:
> 
> On Tue, Dec 18, 2018 at 11:49 PM Enji Cooper  > wrote:
>> 
>> Hello Matthew,
>> 
>> I appreciate the long write up, as someone who still uses FreeBSD ZFS on my 
>> NAS box and knowing some of the history with ZFS on *Solaris, etc. Something 
>> like this was bound to happen with post the Oracle buyout.
>> 
>>> On Dec 18, 2018, at 10:49 PM, Matthew Macy  wrote:
>>> 
>>> The sources for FreeBSD's ZFS support are currently taken directly
>>> from Illumos with local ifdefs to support the peculiarities of FreeBSD
>>> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
>>> has regularly pulled changes from Illumos and tried to push back any
>>> bug fixes and new features done in the context of FreeBSD. In the past
>>> few years the vast majority of new development in ZFS has taken place
>>> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
>>> that they will be moving to ZoL
>>> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
>>> that there will be little to no net new development of Illumos. While
>>> working through the git history of ZoL I have also discovered that
>>> many races and locking bugs have been fixed in ZoL and never made it
>>> back to Illumos and thus FreeBSD. This state of affairs has led to a
>>> general agreement among the stakeholders that I have spoken to that it
>>> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
>>> has graciously encouraged me to add FreeBSD support directly to ZoL
>>> https://github.com/zfsonfreebsd/ZoF so that we might all have a single
>>> shared code base.
>>> 
>>> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
>>> Before it can be committed some additional functionality needs to be
>>> added to the FreeBSD opencrypto framework. These can be found at
>>> https://reviews.freebsd.org/D18520
>>> 
>>> This port will provide FreeBSD users with multi modifier protection,
>>> project quotas, encrypted datasets, allocation classes, vectorized
>>> raidz, vectorized checksums, and various command line improvements.
>>> 
>>> Before ZoF can be merged back in to ZoL several steps need to be taken:
>>> - Integrate FreeBSD support into ZoL CI
>>> - Have most of the ZFS test suite passing
>>> - Complete additional QA testing at iX
>> 
>> Can you please describe the testing process that will be employed to verify 
>> the sanity of the ZoL on FreeBSD port? Should other large companies who use 
>> ZFS on FreeBSD (Panzura?) chime in and the ZFS on FreeBSD community (as a 
>> whole) collaborate to better suss out issues with the ZoL port?
> 
> The ZFS test suite itself provides ~80% coverage
> https://codecov.io/gh/zfsonlinux/zfs/branch/master 
>  - FreeBSD currently
> lacks equivalent gcov support, but presumably it would provide
> comparable coverage here. Andrew Turner has some form of kernel gcov
> support that he uses with syzkaller. However, I believe that it isn't
> sufficient for this purpose.

The code I have is to trace the kernel part of a single thread, e.g. what 
happens in the kernel when you make a system call. It can trace function either 
function entry or places in the code with a conditional statement, e.g. an if, 
while, or switch statement. If this is enough for your case a tool could be 
written to track coverage from the tests.

I expect to update the review soon. I’m working on a man page, but have been 
too busy with work and other projects recently to finish writing it.

Andrew

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Shawn Webb]

On Tue, Dec 18, 2018 at 10:49:48PM -0800, Matthew Macy wrote:
> The sources for FreeBSD's ZFS support are currently taken directly
> from Illumos with local ifdefs to support the peculiarities of FreeBSD
> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
> has regularly pulled changes from Illumos and tried to push back any
> bug fixes and new features done in the context of FreeBSD. In the past
> few years the vast majority of new development in ZFS has taken place
> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
> that they will be moving to ZoL
> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
> that there will be little to no net new development of Illumos. While
> working through the git history of ZoL I have also discovered that
> many races and locking bugs have been fixed in ZoL and never made it
> back to Illumos and thus FreeBSD. This state of affairs has led to a
> general agreement among the stakeholders that I have spoken to that it
> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
> has graciously encouraged me to add FreeBSD support directly to ZoL
> https://github.com/zfsonfreebsd/ZoF so that we might all have a single
> shared code base.
> 
> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
> Before it can be committed some additional functionality needs to be
> added to the FreeBSD opencrypto framework. These can be found at
> https://reviews.freebsd.org/D18520
> 
> This port will provide FreeBSD users with multi modifier protection,
> project quotas, encrypted datasets, allocation classes, vectorized
> raidz, vectorized checksums, and various command line improvements.
> 
> Before ZoF can be merged back in to ZoL several steps need to be taken:
> - Integrate FreeBSD support into ZoL CI
> - Have most of the ZFS test suite passing
> - Complete additional QA testing at iX
> 
> We at iX Systems need to port ZoL's EC2 CI scripts to work with
> FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes.
> Being integrated in to their CI will mean that, among other things,
> most integration issues will be caught before a PR is merged upstream
> rather than many months later when it is MFVed into FreeBSD. I???m
> hoping to submit the PR to ZoL some time in January.
> 
> This port will make it easy for end users on a range of releases to
> run the latest version of ZFS. Nonetheless, transitioning away from an
> Illumos based ZFS is not likely to be entirely seamless. The
> stakeholders I???ve spoken to all agree that this is the best path
> forward but some degree of effort needs to be made to accommodate
> downstream consumers. The current plan is to import ZoF and unhook the
> older Illumos based sources from the build on April 15th or two months
> after iX systems QA deems ZoF stable - which ever comes later. The
> Illumos based sources will be removed some time later - but well
> before 13. This will give users a 3 month period during which both the
> port and legacy Illumos based ZFS will be available to users. Pools
> should interoperate between ZoF and legacy provided the user does not
> enable any features available only in ZoF. We will try to accommodate
> any downstream consumers in the event that they need that date pushed
> back. We ask that any downstream consumers who are particularly
> sensitive to changes start testing the port when it is formally
> announced and report back any issues they have. I will do my best to
> ensure that this message is communicated to all those who it may
> concern. However, I can???t ensure that everyone reads these lists. That
> is the responsibility of -CURRENT users.

Hey Matt,

Thank you for your detailed and informative post. It really helps
downstream consumers of FreeBSD.

I'm curious what this means for OpenZFS. I was under the impression
that OpenZFS was the upstream for all the ZFS implementations (sans
Oracle).

Thanks,

-- 
Shawn Webb
Cofounder and Security Engineer
HardenedBSD

Tor-ified Signal:+1 443-546-8752
Tor+XMPP+OTR:latt...@is.a.hacker.sx
GPG Key ID:  0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89  3D9E 6A84 658F 5245 6EEE


signature.asc
Description: PGP signature

Re: Composite PCI devices in FreeBSD (mfd in Linux)

[Anthony Jenkins]

On 12/10/18 3:57 PM, John Baldwin wrote:

On 12/10/18 12:19 PM, Ian Lepore wrote:

On Mon, 2018-12-10 at 14:42 -0500, Anthony Jenkins wrote:

On 12/10/18 1:26 PM, John Baldwin wrote:

On 12/10/18 9:00 AM, Anthony Jenkins wrote:

Hi all,

I'm trying to port an Intel PCI I2C controller from Linux to
FreeBSD.
Linux represents this device as an MFD (multi-function device),
meaning
it has these "sub-devices" that can be handed off to other
drivers to
actually attach devices to the system.  The Linux "super" PCI
device is
the intel-lpss-pci.c, and the "sub" device is i2c-designware-
platdrv.c,
which represents the DesignWare driver's "platform" attachment to
the
Linux system.  FreeBSD also has a DesignWare I2C controller
driver,
ig4(4), but it only has PCI and ACPI bus attachment
implementations.

I have a port of the Linux intel-lpss driver to FreeBSD, but now
I'm
trying to figure out the best way to give FreeBSD's ig4(4) driver
access
to my lpss(4) device.  I'm thinking I could add an ig4_lpss.c
describing
the "attachment" of an ig4(4) to an lpss(4).  Its probe() method
would
scan the "lpss" devclass for devices, and its attach() method
would
attach itself as a child to the lpss device and "grab" the
portion of
PCI memory and the IRQ that the lpss PCI device got.

Is this the "FreeBSD Way (TM)" of handling this type of device?
If not,
can you recommend an existing FreeBSD driver I can model my code
after?
If my approach is acceptable, how do I fully describe the ig4(4)
device's attachment to the system?  Is simply making it a child
of
lpss(4) sufficient?  It's "kind of" a PCI device (it is
controlled via
access to a PCI memory region and an IRQ), but it's a sub-device
of an
actual PCI device (lpss(4)) attached to PCI.
How would my ig4_lpss attachment get information from the lpss(4)
driver
about what it probed?

There are some existing PCI drivers that act as "virtual" busses
that attach
child devices.  For example, vga_pci.c can have drm, agp, and
acpi_video
child devices.  There are also some SMBus drivers that are also
PCI-ISA
bridges and thus create separate child devices.

Yeah I was hoping to avoid using video PCI devices as a model, as
complex as they've gotten recently.  I'll check out its bus glue
logic.


For a virtual bus like this, you need to figure out how your child
devices
will be enumerated.  A simple way is to let child devices use an
identify
routine that looks at each parent device and decides if a child
device
for that driver makes sense.  It can then add a child device in the
identify routine.

Really an lpss parent PCI parent device can only have the following:

   * one of {I2C, UART, SPI} controller
   * optionally an IDMA64 controller

so I was thinking a child ig4(4) device would attach to lpss iff

   * the lpss device detected an I2C controller
   * no other ig4 device is already attached

I haven't fiddled with identify() yet, will look at that tonight.


If this is just another "bus" an ig4 instance can attach to, I'd think
the recipe would be to add another DRIVER_MODULE() to ig4_iic.c naming
ig4_lpss as the parent. Then add a new ig4_lpss.c modeled after the
existing pci and acpi attachment code, its DRIVER_MODULE() would name
lpss as parent, and its probe routine would return BUS_PROBE_NOWILDCARD
(attach only if specifically added by the parent).

Then there would be a new lpss driver that does the resource managment
stuff mentioned above, and if it detects configuration for I2C it would
do a device_add_child(lpssdev, "ig4_lpss", -1) followed by
bus_generic_attach(). There'd be no need for identify() in the child in
that case, I think.

But take jhb's word over mine on any of this stuff, he's been around
since the days when these mechanisms were all invented, whereas I tend
to cut and paste that bus and driver attachment stuff in semi-ignorance
when I'm working on drivers.

Doing the device_add_child in the parent driver's attach routine is also
fine instead of using an identify routine.  It's mostly a matter of which
driver should be in charge of adding the child device (e.g. would you want
lpss self-contained or should the parent driver know about it explicitly).

If you have an existing ig4 driver you are going to reuse, then you will
need to ensure you fake up the resource stuff so that the existing code
works perhaps, though that depends on where the bus_alloc_resource calls
occur.  If they are in shared code you have to fake more.  If they are in
the bus-specific attach routines, then you can just put lpss specific
logic in the lpss-specific attach routine.


To handle things like resources, you want to have
bus_*_resource methods that let your child device use the normal
bus_*
functions to allocate resources.  At the simplest end you don't
need to
permit any sharing of BARs among multiple children so you can just
proxy
the requests in the "real" PCI driver.  (vga_pci.c does this)  If
you need
the BARs to be shared you have a couple of options such as just
using a

Re: possible POLA violation for NFS server to make it Linux compatible

[Cy Schubert]

In message 
, Conrad Meyer writes:
> On Tue, Dec 18, 2018 at 5:07 PM Rick Macklem  wrote:
> > The change to make the FreeBSD NFSv4 server use vfs.nfsd.nfs_privport is tr
> ivial
> > and I think being compatible with Linux is important (I see it as the defac
> to
> > standard NFS implementation these days).
> >
> > What do others think I should do?
>
> Hi Rick,
>
> I think we should go ahead and honor nfs_privport.

Agreed and an UPDATING entry.


-- 
Cheers,
Cy Schubert 
FreeBSD UNIX: Web:  http://www.FreeBSD.org

The need of the many outweighs the greed of the few.


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Matthew Macy]

On Tue, Dec 18, 2018 at 11:49 PM Enji Cooper  wrote:
>
> Hello Matthew,
>
> I appreciate the long write up, as someone who still uses FreeBSD ZFS on my 
> NAS box and knowing some of the history with ZFS on *Solaris, etc. Something 
> like this was bound to happen with post the Oracle buyout.
>
> > On Dec 18, 2018, at 10:49 PM, Matthew Macy  wrote:
> >
> > The sources for FreeBSD's ZFS support are currently taken directly
> > from Illumos with local ifdefs to support the peculiarities of FreeBSD
> > where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
> > has regularly pulled changes from Illumos and tried to push back any
> > bug fixes and new features done in the context of FreeBSD. In the past
> > few years the vast majority of new development in ZFS has taken place
> > in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
> > that they will be moving to ZoL
> > https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
> > that there will be little to no net new development of Illumos. While
> > working through the git history of ZoL I have also discovered that
> > many races and locking bugs have been fixed in ZoL and never made it
> > back to Illumos and thus FreeBSD. This state of affairs has led to a
> > general agreement among the stakeholders that I have spoken to that it
> > makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
> > has graciously encouraged me to add FreeBSD support directly to ZoL
> > https://github.com/zfsonfreebsd/ZoF so that we might all have a single
> > shared code base.
> >
> > A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
> > Before it can be committed some additional functionality needs to be
> > added to the FreeBSD opencrypto framework. These can be found at
> > https://reviews.freebsd.org/D18520
> >
> > This port will provide FreeBSD users with multi modifier protection,
> > project quotas, encrypted datasets, allocation classes, vectorized
> > raidz, vectorized checksums, and various command line improvements.
> >
> > Before ZoF can be merged back in to ZoL several steps need to be taken:
> > - Integrate FreeBSD support into ZoL CI
> > - Have most of the ZFS test suite passing
> > - Complete additional QA testing at iX
>
> Can you please describe the testing process that will be employed to verify 
> the sanity of the ZoL on FreeBSD port? Should other large companies who use 
> ZFS on FreeBSD (Panzura?) chime in and the ZFS on FreeBSD community (as a 
> whole) collaborate to better suss out issues with the ZoL port?

The ZFS test suite itself provides ~80% coverage
https://codecov.io/gh/zfsonlinux/zfs/branch/master - FreeBSD currently
lacks equivalent gcov support, but presumably it would provide
comparable coverage here. Andrew Turner has some form of kernel gcov
support that he uses with syzkaller. However, I believe that it isn't
sufficient for this purpose. ZoL requires that coverage only increase
over time. If a change would decrease total coverage new tests have to
be included with the change. There is a command called ztest which
runs in user space that quickly covers a large percentage of the code
base where it doesn't integrate directly with geom or VFS (disk access
is emulated with files, kernel threads are emulated with pthreads,
etc). ztest has been broken in HEAD for the past 2 years (it triggers
an assertion failure), whereas it is not broken in ZoF. Joe Maloney is
the head of QA at iX and can likely give you more detail on their
approach to testing. This e-mail was an implicit request to Panzura to
do whatever testing that they can.

>
> > We at iX Systems need to port ZoL's EC2 CI scripts to work with
> > FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes.
> > Being integrated in to their CI will mean that, among other things,
> > most integration issues will be caught before a PR is merged upstream
> > rather than many months later when it is MFVed into FreeBSD. I’m
> > hoping to submit the PR to ZoL some time in January.
>
> Could you please better describe this, in particular provide pointers to any 
> scripts that might be executed with the ZoL port?

The ZFS test suite is in ZoL under tests, it makes some Linux specific
assumptions about paths to binaries like ksh etc and it appears to
require some additional configuration options. I started fixing things
before I decided to punt and ask that a porter at iX fix it so that I
could focus on areas that make more sense for me to spend my time on.
If it doesn't get handled I will of course go back to it. The CI
scripts are not something I've looked at. My intention is again that
the people who would be responsible for keeping them up at iX would
close the loop. It's close to Christmas time right now and folks are
inevitably preoccupied with family matters. I sent this mail in
advance of resolving these details to give any outside stakeholders a
heads up as far out as possible. "Shadow" stakeholders coming out of