Github user bhupeshchawda commented on the issue:
https://github.com/apache/incubator-samoa/pull/55
@nicolas-kourtellis Please find my responses below:
1) The slow execution is a deliberate (although temporary) configuration
done in samoa-apex by limiting the number of tuples in an application window.
This has to do with the way iteration works in Apex, which is tightly coupled
to windowing. In case we don't limit the number of tuples, the tuples in a
particular window keep on increasing due to the additional tuples that are fed
on the iteration loop back stream. If the number of iterations is large enough,
the amount of time taken to process a window of data increases beyond normal
behaviour and the operator is killed by the Apex app master. I am working on
identifying some workaround either to eliminate this limit, or to optimally set
this limit.
2) The execution in local mode of Apex is highly asynchronous with all
operators in the topology running in different threads. The local mode of
Samoa, on the other hand seems to be synchronous; i.e. the next tuple is
processed only when the first one has been processed completely by all
operators. I also tried to check executing the local mode of Storm, which also
produces different results every time it is run for the same input file.
3) I think this is due to the same reason in (2)
4) Yes, these changes are necessary for Apex to function correctly. Apex
relies on Kryo serialization (without any fall back on Java serialization) and
hence is necessary for classes to have a default constructor. I think it will
be better to have them as part of this PR. May be I can split them into a
different commit if that helps?
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
