Hi Cesar,
I agree with you that this is a critical issue because we have the same
requirement as you in terms of availability for a project that is
already in production.
We got into some troubles (and we currently are because the issue is not
solved) with things related to transactions and database connections and
statements and we fixed some issues. By using a java profiler we found
two cases where the system was clearly producing memory leaks and these
was properly fixed.
https://issues.apache.org/jira/browse/HOP-3700
https://issues.apache.org/jira/browse/HOP-3724
Even with these fixes the memory is continuously raising up therefore
now we are currently investigating each transform that has to do with
databases that is used in our pipelines. We already solved some issues
that are related to statements not closed
https://issues.apache.org/jira/browse/HOP-3760
https://issues.apache.org/jira/browse/HOP-3786
https://issues.apache.org/jira/browse/HOP-3785
A last one has to do with PipelineExecutor and the way it is managing
pipeline execution. For this, we've neither opened any JIRA (even if it
has already been solved and is under stress test) nor pushed any PR yet.
For the moment I don't have any other to say about this problem. Anyway,
currently, the memory is continuously raising up so we are continuing
our investigation, that we hope to finish very soon (to not manually
restart everything every night!). I will keep you updated.
Best
Sergio
Il 23/02/2022 14:41, Cesar Garcia ha scritto:
Hello Sergio,
You may notice the memory consumption problem when integrating the Apache
PLC4X project.
have you had any progress in this regard? How are you approaching the
problem to see if I can support you in this regard?
It is important to solve this since we are talking about 24/7@365
applications.
Thank you for any comment,
Kind regards.
El mar, 22 feb 2022 a las 18:42, Sergio Ramazzina (SERASOFT) (<
[email protected]>) escribió:
Hi Hans,
as you know we are hardly working to fix the memory leak issues we are
getting from our long running processes running in production. We think
we are going very well. During th enext few ours I will open a few JIRA
issues to push some other PR to fix the lates bugs we found lastly.
I ask you if we can postpone the cutoff date to mid of March (March 15th
is fine I think) to be able to catch and close the latest things. The
battle is not win yet but I think anyone will benefit from this hard
work to make our Hop even stronger.
Let me know what you think.
Best
Sergio
Il 22/02/2022 19:38, Hans Van Akelyen ha scritto:
Hello Hoppers,
The development on version 1.2.0 started on the 15th of January and we
have
made some great progress, I would like to thank everyone that has been
creating tickets and solving them.
Currently we are at 65 tickets solved, including mostly bug fixes and
some
nice new features.
For those interested, a full list on what has happened on 1.2.0 can be
found in Jira [1]
I think we have come to a point where we can start thinking about the
next
release, I would like to suggest the end of the month (28th February) as
the final day where we will be merging pull requests and start working on
the vote then.
As it is really hard to maintain two separate branches and the EOL of
java
8 has come into sight (no worries it will still receive security fixes) I
would like to suggest going for Hop 2.0 after this release. Hop 2.0 will
be
our first full java 11 release and include some breaking API changes.
Love to hear your thoughts!
Cheers,
Hans
[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12324323&version=12351230
