Re: [Wireshark-dev] Why does the extcap stuff take so long to start up?

[Michał Łabędzki]

> At the moment there is a timeout of 4s to allow androiddump to connect to ADB.

I should be fixed right now and timeout should be 10ms (aka
non-blocking socket on Windows and  on Linux/FreeBSD). If it is
no 10ms (no ADB daemon started), then there is only a need to do truly
non-blocking socket also for Linux/etc. (follow Windows solution).
Current solution works (for me) on Linux (Ubuntu and LFS).

The question is why there is significant delay for Guy, but not for
me. I use Linux+cmake (out of source) + run Wireshark(s) from, build
directory.

Dario, could you share your results for:
$ adb kill-server
$ time run/extcap/androiddump --extcap-interfaces


My results:
real0m0.013s
user0m0.000s
sys 0m0.001s

> about .26 seconds for randpktcdump

It seems that your OS is much more slower than me in case of loading
programs. What is it?


"strace -tt -v  ./run/tshark -D" may show there reason for it.
Is that true that g_dir_open() is call many times for all extcap
captures? It may be slow on some file[/]systems.

Maybe the problem is extcap architecture: "small" binaries that must
be run. We can (?) change it to
libraries (plugins) + one generic small application that use "plugin".
Then it is possible to link or dlopen libraries by Wireshark, but
still can run it as standalone apps. I think that can save significant
loading time and multiplatform compatibility.

In other words:
./genericExtcapTextUI --load=randpktdump --extcap-interfaces
or
./wireshark # it dlopen("libextcap_randpkt.so") etc.


2017-03-27 22:33 GMT+02:00 Guy Harris :
> On Mar 27, 2017, at 1:14 PM, Guy Harris  wrote:
>
>> Currently, with that fix, I get results like
>>
>> $ time ./tshark -r /tmp/nothing.pcap
>>
>> real0m1.407s
>> user0m0.312s
>> sys 0m0.676s
>>
>> with the extcap directory in place and results like
>>
>> $ time ./tshark -r /tmp/nothing.pcap
>>
>> real0m0.334s
>> user0m0.182s
>> sys 0m0.146s
>>
>> with the extcap directory moved out of the way, so the extcap executables 
>> are taking some time to run, but it's better than wasting time trying to run 
>> androiddump.c or Makefile.am.
>
> And, if I move various extcap executables out of the way:
>
> $ time ./tshark -r /tmp/nothing.pcap# all executables
>
> real0m1.484s
> user0m0.313s
> sys 0m0.720s
>
> $ time ./tshark -r /tmp/nothing.pcap# after removing androiddump
>
> real0m1.179s
> user0m0.287s
> sys 0m0.588s
>
> $ time ./tshark -r /tmp/nothing.pcap# after removing ciscodump
>
> real0m0.950s
> user0m0.254s
> sys 0m0.491s
>
> $ time ./tshark -r /tmp/nothing.pcap# after removing randpktcdump
>
> real0m0.688s
> user0m0.228s
> sys 0m0.334s
>
> $ time ./tshark -r /tmp/nothing.pcap# after removing sshdump
>
> real0m0.493s
> user0m0.198s
> sys 0m0.235s
>
> $ time ./tshark -r /tmp/nothing.pcap# after removing udpdump
>
> real0m0.335s
> user0m0.183s
> sys 0m0.145s
>
> So that's about .3 seconds for androiddump, about .23 seconds for ciscodump, 
> about .26 seconds for randpktcdump, about .19 seconds for sshdump, and about 
> .16 seconds for usbdump.
>
> So none of them are individually out of the ordinary, but about 1.5 to 2.5 
> seconds per program, with 5 programs, adds up.
> ___
> Sent via:Wireshark-dev mailing list 
> Archives:https://www.wireshark.org/lists/wireshark-dev
> Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
>  mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe



-- 
Michał Łabędzki
___
Sent via:Wireshark-dev mailing list 
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe

Re: [Wireshark-dev] Why does the extcap stuff take so long to start up?

[Dario Lombardo]

The problem is only when compiling with autotools. With cmake the bin
extcap dir is clean and it doesn't affect startup time. Btw the check you
added is wise and should be done anyway in case unwanted artifacts are
there. The problem with extcaps is that androiddump is the only tool that
enumerates interfaces at start time. Others have static interfaces and
start very quickly. Me, Michal and others have discussed a lot on this
topic but we didn't find a final answer. At the moment there is a timeout
of 4s to allow androiddump to connect to ADB. If ADB is not running, we
wait until the timeout, and it's pretty annoying if you don't have adb and
will never have, imho.

A faster solution would be to check if the process adb is running and if
it's not, just skip the attempt. But we didn't find a suitable way to do so
on unix & windows. A possible solution could be to run this check on
systems that support the choosen method (like kill described here
http://stackoverflow.com/questions/6898337/determine-programmatically-if-a-program-is-running),
and skip it on windows, leaving the things as they are right now. Michal
what do you think?

On Mon, Mar 27, 2017 at 10:14 PM, Guy Harris  wrote:

> On Mar 26, 2017, at 11:30 PM, Michał Łabędzki <
> michal.tomasz.labed...@gmail.com> wrote:
>
> > Could you check if any of extcap binaries generates that delay?
>
> A lot of the delay comes from extcap *non*-binaries; the code that scans
> the extcap directory tried running *everything* it finds there, except for
> "." and "..", and, if you're running from the build directory, that means
> executing somewhat heavyweight {fork/vfork}-exec calls on a bunch of source
> files, object files, .deps directories, Makefiles, etc..
>
> bd3196b094ae46fa4396edbb406d68056cba6974 fixed that.
>
> Currently, with that fix, I get results like
>
> $ time ./tshark -r /tmp/nothing.pcap
>
> real0m1.407s
> user0m0.312s
> sys 0m0.676s
>
> with the extcap directory in place and results like
>
> $ time ./tshark -r /tmp/nothing.pcap
>
> real0m0.334s
> user0m0.182s
> sys 0m0.146s
>
> with the extcap directory moved out of the way, so the extcap executables
> are taking some time to run, but it's better than wasting time trying to
> run androiddump.c or Makefile.am.
> 
> ___
> Sent via:Wireshark-dev mailing list 
> Archives:https://www.wireshark.org/lists/wireshark-dev
> Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
>  mailto:wireshark-dev-requ...@wireshark.org?subject=
> unsubscribe
___
Sent via:Wireshark-dev mailing list 
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe

Re: [Wireshark-dev] Why does the extcap stuff take so long to start up?

[Guy Harris]

On Mar 27, 2017, at 1:14 PM, Guy Harris  wrote:

> Currently, with that fix, I get results like
> 
> $ time ./tshark -r /tmp/nothing.pcap 
> 
> real0m1.407s
> user0m0.312s
> sys 0m0.676s
> 
> with the extcap directory in place and results like
> 
> $ time ./tshark -r /tmp/nothing.pcap 
> 
> real0m0.334s
> user0m0.182s
> sys 0m0.146s
> 
> with the extcap directory moved out of the way, so the extcap executables are 
> taking some time to run, but it's better than wasting time trying to run 
> androiddump.c or Makefile.am.

And, if I move various extcap executables out of the way:

$ time ./tshark -r /tmp/nothing.pcap# all executables

real0m1.484s
user0m0.313s
sys 0m0.720s

$ time ./tshark -r /tmp/nothing.pcap# after removing androiddump

real0m1.179s
user0m0.287s
sys 0m0.588s

$ time ./tshark -r /tmp/nothing.pcap# after removing ciscodump

real0m0.950s
user0m0.254s
sys 0m0.491s

$ time ./tshark -r /tmp/nothing.pcap# after removing randpktcdump

real0m0.688s
user0m0.228s
sys 0m0.334s

$ time ./tshark -r /tmp/nothing.pcap# after removing sshdump

real0m0.493s
user0m0.198s
sys 0m0.235s

$ time ./tshark -r /tmp/nothing.pcap# after removing udpdump

real0m0.335s
user0m0.183s
sys 0m0.145s

So that's about .3 seconds for androiddump, about .23 seconds for ciscodump, 
about .26 seconds for randpktcdump, about .19 seconds for sshdump, and about 
.16 seconds for usbdump.

So none of them are individually out of the ordinary, but about 1.5 to 2.5 
seconds per program, with 5 programs, adds up.
___
Sent via:Wireshark-dev mailing list 
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe

Re: [Wireshark-dev] Why does the extcap stuff take so long to start up?

[Guy Harris]

On Mar 26, 2017, at 11:30 PM, Michał Łabędzki 
 wrote:

> Could you check if any of extcap binaries generates that delay?

A lot of the delay comes from extcap *non*-binaries; the code that scans the 
extcap directory tried running *everything* it finds there, except for "." and 
"..", and, if you're running from the build directory, that means executing 
somewhat heavyweight {fork/vfork}-exec calls on a bunch of source files, object 
files, .deps directories, Makefiles, etc..

bd3196b094ae46fa4396edbb406d68056cba6974 fixed that.

Currently, with that fix, I get results like

$ time ./tshark -r /tmp/nothing.pcap 

real0m1.407s
user0m0.312s
sys 0m0.676s

with the extcap directory in place and results like

$ time ./tshark -r /tmp/nothing.pcap 

real0m0.334s
user0m0.182s
sys 0m0.146s

with the extcap directory moved out of the way, so the extcap executables are 
taking some time to run, but it's better than wasting time trying to run 
androiddump.c or Makefile.am.
___
Sent via:Wireshark-dev mailing list 
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe

Re: [Wireshark-dev] Why does the extcap stuff take so long to start up?

[Michał Łabędzki]

Hello Guy,

Could you check if any of extcap binaries generates that delay? (check
one by one) On "my" Ubuntu 12.04:

my_extcap[4] = {androiddump, randpktdump, udpdump, my custom extcap};

  $ time ./run/tshark -D # returns 21 interfaces
  real0m0.188s
  user0m0.115s
  sys 0m0.061s


  $ time ./run/tshark -D # with connected phone + androiddump (+7 interfaces)
  real0m0.464s
  user0m0.115s
  sys 0m0.077s

~200ms for androiddump is very ice (expected), not a bug (if ~4s it is a bug).


  $ time ./run/tshark -D # extcap dir renamed (removed) - 13 interfaces now
  real0m0.141s
  user0m0.106s
  sys 0m0.030s

Summary: no problem on my platform.

PS. Similar results when opening small capture file.

It seems that initializing interfaces takes significant time, but my
question is: why does it  be done if user only try to open
capture file? Could we do not initialize nor extcap neither libpcap
interfaces in that case? or do it in background...

2017-03-27 1:05 GMT+02:00 Guy Harris :
> $ time ./tshark -r /tmp/nothing.pcap# reading an empty pcap file
> # with all the extcap binaries present
> real0m4.089s
> user0m0.694s
> sys 0m2.637s
>
> $ time ./tshark -r /tmp/nothing.pcap# reading the same empty pcap file
> # with the extcap binaries removed
> real0m0.540s
> user0m0.310s
> sys 0m0.220s
>
> So it took *four seconds* of real time, and over *two seconds* of system-mode 
> CPU time, to run, I guess, extcap_register_preferences(), and all the stuff 
> it runs.
>
> Does it need to take that long?
>
> If it does need to take that long, do we need to do that if we're not 
> capturing?
> ___
> Sent via:Wireshark-dev mailing list 
> Archives:https://www.wireshark.org/lists/wireshark-dev
> Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
>  mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe



-- 
Michał Łabędzki
___
Sent via:Wireshark-dev mailing list 
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe