Hey all,
just a short appendum.
I profiled through the code some more and found out the following:
* Edgent (ETIAO Runtime) creates (and deletes) one Thread for each scheduled
invocation of a Polling Source
* This thread is not only used to poll but also to do all subsequent processing
of the next pipeline steps (Oplets)
* This leads to creations of hundres of Threads / min which imposes significant
pressure on GC (I looked at the SensorAnalyticsApplication where GC was
triggered on my laptop each 30s and about 300MB in Thread Objets were collected
(about 1000 instances)
Furthermore this has the following consequences:
* No backpressure can be used with the current setup slow sinks have to throw
away messages(?)
* No guarantee of order as sheduleAtFixedRate is used and the runnable that’s
scheduled processes the complete pipeline for the fetched element, so if, e.g.,
some intense operations is only performed on some objects (or simple filtering)
this could lead to races where elements that were pulled later enter the sink
earlier
I will try to provide a concrete code example where these situations are
demonstrated.
It would be interesting for me to get an idea why this implementation was
chosen or what you think are the advantages of this approach. I think Edgent
has some very nice features and can solve some of the common problems for
"edge" analytics in an elegant way but I'm not that sure with the ETIAO Runtime
for the reasons stated above.
But perhaps I am also wrong with some of my conclusions as they are solely from
profiling and debugging through the code so please correct me if I am.
Best
Julian
Am 22.09.18, 10:07 schrieb "Julian Feinauer" <j.feina...@pragmaticminds.de>:
Hi Edgenteers,
after chris does not stop to convince me that Edgent is THE THING to use
for IoT I started once again a deeper look into its internals.
I do not yet fully understand the core of Edgent and its runtime or how it
physically runs the users operation at the end.
But what I observed is that it uses a pretty high number of threads and
what’s way worse it consumes many threads, i.e., it starts and stops threads
all the time (about 1 per second in several Demo Applications (Water / Sensor
Analytics Application).
And this is also not dependent on the Provider (Direct vs Development).
Can anyone here give me some explanations on how the Threading Model of
Edgents runtime is and why it is done that way as for me, it seems a bit
wasteful with regards to resources and we want to be as low footprint in our
applications as possible.
Thank you!
Julian
