Hi Julian,
well as I told you, I got on board of the project shortly before IBM shifted
their priorities. Unfortunately I didn't have the chance to dig into all the
details before that happened and other dropped out.
I wouldn't be expecting much input from the rest. So I guess we would have to
dig through stuff like this on our own.
But I am really willing to do this as I don't know of a similar project and I
really want to help revive Edgent again.
Chris
Am 22.09.18, 08:13 schrieb "Julian Feinauer" <j.feina...@pragmaticminds.de>:
Hey all,
just a short appendum.
I profiled through the code some more and found out the following:
* Edgent (ETIAO Runtime) creates (and deletes) one Thread for each
scheduled invocation of a Polling Source
* This thread is not only used to poll but also to do all subsequent
processing of the next pipeline steps (Oplets)
* This leads to creations of hundres of Threads / min which imposes
significant pressure on GC (I looked at the SensorAnalyticsApplication where GC
was triggered on my laptop each 30s and about 300MB in Thread Objets were
collected (about 1000 instances)
Furthermore this has the following consequences:
* No backpressure can be used with the current setup slow sinks have to
throw away messages(?)
* No guarantee of order as sheduleAtFixedRate is used and the runnable
that’s scheduled processes the complete pipeline for the fetched element, so
if, e.g., some intense operations is only performed on some objects (or simple
filtering) this could lead to races where elements that were pulled later enter
the sink earlier
I will try to provide a concrete code example where these situations are
demonstrated.
It would be interesting for me to get an idea why this implementation was
chosen or what you think are the advantages of this approach. I think Edgent
has some very nice features and can solve some of the common problems for
"edge" analytics in an elegant way but I'm not that sure with the ETIAO Runtime
for the reasons stated above.
But perhaps I am also wrong with some of my conclusions as they are solely
from profiling and debugging through the code so please correct me if I am.
Best
Julian
Am 22.09.18, 10:07 schrieb "Julian Feinauer" <j.feina...@pragmaticminds.de>:
Hi Edgenteers,
after chris does not stop to convince me that Edgent is THE THING to
use for IoT I started once again a deeper look into its internals.
I do not yet fully understand the core of Edgent and its runtime or how
it physically runs the users operation at the end.
But what I observed is that it uses a pretty high number of threads and
what’s way worse it consumes many threads, i.e., it starts and stops threads
all the time (about 1 per second in several Demo Applications (Water / Sensor
Analytics Application).
And this is also not dependent on the Provider (Direct vs Development).
Can anyone here give me some explanations on how the Threading Model of
Edgents runtime is and why it is done that way as for me, it seems a bit
wasteful with regards to resources and we want to be as low footprint in our
applications as possible.
Thank you!
Julian
