Re: Slow to upload thousands of tiny files over cpiod

2017-11-03 Thread Dor Laor 
On Fri, Nov 3, 2017 at 8:14 PM, Waldek Kozaczuk 
wrote:

> Miha,
>
> It looks like you are right. When I disabled printing of files names in
> cpiod it made some difference when using QEMU and huge difference when
> using VirtualBox on OSX. With QEMU on average I was able to cut down the
> time from ~ 3 minutes to ~2 minutes and from ~10 minutes to ~ 10 SECONDS
> with VirtualBox when uploading 13,000 tiny files. Please note i have not
> run any tests on Linux yet which I am planning to do next.
>
> Does this mean that in general printing to standard out (over serial
> console?) is slow and should be avoided? And it is extra slow on some
>

Yes, serial console is slow, every character triggers an interrupt and exit
to the hypervisor userspace if not more than one.


> hypervisors/emulators like VirtualBox? Does this have anything with this
> recent patch - https://github.com/cloudius-systems/osv/commit/
> a543b9e6dd8776c43e070ec1b677f2fe1dcbb72e?
>
> Regards,
> Waldek
>
>
> On Friday, November 3, 2017 at 3:40:32 AM UTC-4, miha.ple...@gmail.com
> wrote:
>>
>> Hi Waldek,
>>
>> I think the slowness comes because cpiod reports each file on the stdout.
>> Could you try to comment-out the logging in cpiod and try if it works
>> faster?
>> So if you open two terminals and in the first one run:
>>
>> ```
>> $ capstan run demo --execute "/tools/cpiod.so --prefix /zfs/zfs" -f
>> 1:1
>> ```
>>
>> Then in another terminal run:
>>
>> ```
>> $ capstan package compose-remote localhost
>> ```
>>
>> Capstan will start uploading. But if you look at the first terminal, then
>> you will notice that each file is printed to console, e.g.:
>>
>> ```
>> $ capstan run demo --execute "/tools/cpiod.so --prefix /zfs/zfs" -f
>> 1:1
>> Command line will be set based on --run parameter
>> Created instance: demo
>> Setting cmdline: /tools/cpiod.so --prefix /zfs/zfs
>> OSv v0.24-448-g829bf76
>> eth0: 192.168.122.15
>> Waiting for connection from host...
>>
>> Adding /tools/cpiod.so...  # <-
>> --- PROBLEMATIC OUTPUT, SHOULD
>> BE SUPPRESSED
>> Adding /bin...
>> Adding /bin/libnode-4.4.5.so...
>> Adding /run/interpreter...
>> Adding /run/sleep...
>> Adding /server.js...
>> Adding /usr...
>> Adding /zpool.so...
>> Adding /etc...
>> Link /node to /bin/libnode-4.4.5.so ...
>> Adding /run/burek...
>> Adding /run/node...
>> Adding /tmp...
>> Adding /run...
>> Adding /tools...
>> Adding /libzfs.so...
>> Adding /zfs.so...
>> Adding /run/hello...
>> Adding /tools/mkfs.so...
>> Adding /etc/fstab...
>> Link /etc/mnttab to /proc/mounts ...
>> Adding /greeting.js...
>> Adding /libenviron.so...
>> Adding /libuutil.so...
>> Adding /bin/sleep.so...
>> Adding /proc...
>> Adding /bin/cd.so...
>> Adding /etc/hosts...
>> Adding /usr/lib...
>> Adding /dev...
>> Adding /usr/lib/libgcc_s.so.1...
>> ...
>> ```
>>
>> Please let me know if suppressing this log speeds up the process, I never
>> had time to play with this myself, but I'm very curious :)
>>
>>
>>
>> Best Regards,
>> Miha
>>
>>
>>
>>
>> Dne četrtek, 02. november 2017 21.16.37 UTC+1 je oseba Waldek Kozaczuk
>> napisala:
>>>
>>> I think this might be a well known and undestood issue by capstan team
>>> but I wanted to uderstand why it happens for my own sake.
>>>
>>> In my scenario I am usually able to upload 300 of files totaling 60MB in
>>> 10-12 seconds (Java app) using capstan which I believe employs same
>>> mechanism as OSv build script (over a socket to cpiod). On other hand when
>>> I upload 13,000 small JS files (node app) totalling around 100MB it takes
>>> at least 3 minutes (when I use VirtualBox to run OSv instance I am
>>> uploading to it takes over 10 minutes). Both capstan and ODv I am uploading
>>> to run on the same machine.
>>>
>>> Why is the upload of small files at least 10 times slower than with
>>> bigger files? Is it because ZFS is not most efficient handling small files?
>>> Is it possibly of how the upload is implemented on capstan side?
>>>
>>> Any ideas how to speed it up? Would somehow compressing all files in one
>>> archive and uploading to cpiod as one tarball and then decompressing help?
>>>
>>> Regards,
>>> Waldek
>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Slow to upload thousands of tiny files over cpiod

2017-11-03 Thread Waldek Kozaczuk 
Miha,

It looks like you are right. When I disabled printing of files names in 
cpiod it made some difference when using QEMU and huge difference when 
using VirtualBox on OSX. With QEMU on average I was able to cut down the 
time from ~ 3 minutes to ~2 minutes and from ~10 minutes to ~ 10 SECONDS 
with VirtualBox when uploading 13,000 tiny files. Please note i have not 
run any tests on Linux yet which I am planning to do next.

Does this mean that in general printing to standard out (over serial 
console?) is slow and should be avoided? And it is extra slow on some 
hypervisors/emulators like VirtualBox? Does this have anything with this 
recent patch 
- 
https://github.com/cloudius-systems/osv/commit/a543b9e6dd8776c43e070ec1b677f2fe1dcbb72e?

Regards,
Waldek 

On Friday, November 3, 2017 at 3:40:32 AM UTC-4, miha.ple...@gmail.com 
wrote:
>
> Hi Waldek,
>
> I think the slowness comes because cpiod reports each file on the stdout. 
> Could you try to comment-out the logging in cpiod and try if it works 
> faster?
> So if you open two terminals and in the first one run:
>
> ```
> $ capstan run demo --execute "/tools/cpiod.so --prefix /zfs/zfs" -f 
> 1:1
> ```
>
> Then in another terminal run:
>
> ```
> $ capstan package compose-remote localhost
> ```
>
> Capstan will start uploading. But if you look at the first terminal, then 
> you will notice that each file is printed to console, e.g.:
>
> ```
> $ capstan run demo --execute "/tools/cpiod.so --prefix /zfs/zfs" -f 
> 1:1
> Command line will be set based on --run parameter
> Created instance: demo
> Setting cmdline: /tools/cpiod.so --prefix /zfs/zfs
> OSv v0.24-448-g829bf76
> eth0: 192.168.122.15
> Waiting for connection from host...
>
> Adding /tools/cpiod.so...  # 
> < 
> PROBLEMATIC OUTPUT, SHOULD BE SUPPRESSED
> Adding /bin...
> Adding /bin/libnode-4.4.5.so...
> Adding /run/interpreter...
> Adding /run/sleep...
> Adding /server.js...
> Adding /usr...
> Adding /zpool.so...
> Adding /etc...
> Link /node to /bin/libnode-4.4.5.so ...
> Adding /run/burek...
> Adding /run/node...
> Adding /tmp...
> Adding /run...
> Adding /tools...
> Adding /libzfs.so...
> Adding /zfs.so...
> Adding /run/hello...
> Adding /tools/mkfs.so...
> Adding /etc/fstab...
> Link /etc/mnttab to /proc/mounts ...
> Adding /greeting.js...
> Adding /libenviron.so...
> Adding /libuutil.so...
> Adding /bin/sleep.so...
> Adding /proc...
> Adding /bin/cd.so...
> Adding /etc/hosts...
> Adding /usr/lib...
> Adding /dev...
> Adding /usr/lib/libgcc_s.so.1...
> ...
> ```
>
> Please let me know if suppressing this log speeds up the process, I never 
> had time to play with this myself, but I'm very curious :)
>
>
>
> Best Regards,
> Miha
>
>
>
>
> Dne četrtek, 02. november 2017 21.16.37 UTC+1 je oseba Waldek Kozaczuk 
> napisala:
>>
>> I think this might be a well known and undestood issue by capstan team 
>> but I wanted to uderstand why it happens for my own sake. 
>>
>> In my scenario I am usually able to upload 300 of files totaling 60MB in 
>> 10-12 seconds (Java app) using capstan which I believe employs same 
>> mechanism as OSv build script (over a socket to cpiod). On other hand when 
>> I upload 13,000 small JS files (node app) totalling around 100MB it takes 
>> at least 3 minutes (when I use VirtualBox to run OSv instance I am 
>> uploading to it takes over 10 minutes). Both capstan and ODv I am uploading 
>> to run on the same machine. 
>>
>> Why is the upload of small files at least 10 times slower than with 
>> bigger files? Is it because ZFS is not most efficient handling small files? 
>> Is it possibly of how the upload is implemented on capstan side?
>>
>> Any ideas how to speed it up? Would somehow compressing all files in one 
>> archive and uploading to cpiod as one tarball and then decompressing help?
>>
>> Regards,
>> Waldek
>>
>>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Slow to upload thousands of tiny files over cpiod

2017-11-03 Thread miha . plesko . xlab 
Hi Waldek,

I think the slowness comes because cpiod reports each file on the stdout. 
Could you try to comment-out the logging in cpiod and try if it works 
faster?
So if you open two terminals and in the first one run:

```
$ capstan run demo --execute "/tools/cpiod.so --prefix /zfs/zfs" -f 
1:1
```

Then in another terminal run:

```
$ capstan package compose-remote localhost
```

Capstan will start uploading. But if you look at the first terminal, then 
you will notice that each file is printed to console, e.g.:

```
$ capstan run demo --execute "/tools/cpiod.so --prefix /zfs/zfs" -f 
1:1
Command line will be set based on --run parameter
Created instance: demo
Setting cmdline: /tools/cpiod.so --prefix /zfs/zfs
OSv v0.24-448-g829bf76
eth0: 192.168.122.15
Waiting for connection from host...

Adding /tools/cpiod.so...  # 
< 
PROBLEMATIC OUTPUT, SHOULD BE SUPPRESSED
Adding /bin...
Adding /bin/libnode-4.4.5.so...
Adding /run/interpreter...
Adding /run/sleep...
Adding /server.js...
Adding /usr...
Adding /zpool.so...
Adding /etc...
Link /node to /bin/libnode-4.4.5.so ...
Adding /run/burek...
Adding /run/node...
Adding /tmp...
Adding /run...
Adding /tools...
Adding /libzfs.so...
Adding /zfs.so...
Adding /run/hello...
Adding /tools/mkfs.so...
Adding /etc/fstab...
Link /etc/mnttab to /proc/mounts ...
Adding /greeting.js...
Adding /libenviron.so...
Adding /libuutil.so...
Adding /bin/sleep.so...
Adding /proc...
Adding /bin/cd.so...
Adding /etc/hosts...
Adding /usr/lib...
Adding /dev...
Adding /usr/lib/libgcc_s.so.1...
...
```

Please let me know if suppressing this log speeds up the process, I never 
had time to play with this myself, but I'm very curious :)



Best Regards,
Miha




Dne četrtek, 02. november 2017 21.16.37 UTC+1 je oseba Waldek Kozaczuk 
napisala:
>
> I think this might be a well known and undestood issue by capstan team but 
> I wanted to uderstand why it happens for my own sake. 
>
> In my scenario I am usually able to upload 300 of files totaling 60MB in 
> 10-12 seconds (Java app) using capstan which I believe employs same 
> mechanism as OSv build script (over a socket to cpiod). On other hand when 
> I upload 13,000 small JS files (node app) totalling around 100MB it takes 
> at least 3 minutes (when I use VirtualBox to run OSv instance I am 
> uploading to it takes over 10 minutes). Both capstan and ODv I am uploading 
> to run on the same machine. 
>
> Why is the upload of small files at least 10 times slower than with bigger 
> files? Is it because ZFS is not most efficient handling small files? Is it 
> possibly of how the upload is implemented on capstan side?
>
> Any ideas how to speed it up? Would somehow compressing all files in one 
> archive and uploading to cpiod as one tarball and then decompressing help?
>
> Regards,
> Waldek
>
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.