[jira] [Commented] (OOZIE-3669) Fix purge process for bundles to prevent orphan coordinators
[
https://issues.apache.org/jira/browse/OOZIE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17633252#comment-17633252
]
Janos Makai commented on OOZIE-3669:
Thank you [~dionusos] :)
> Fix purge process for bundles to prevent orphan coordinators
>
>
> Key: OOZIE-3669
> URL: https://issues.apache.org/jira/browse/OOZIE-3669
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.2.1
>Reporter: Janos Makai
>Assignee: Janos Makai
>Priority: Major
> Fix For: 5.3.0
>
> Attachments: OOZIE-3669-001.patch, OOZIE-3669-002.patch,
> OOZIE-3669-003.patch
>
>
> The Oozie purge process for bundles is creating orphan coordinators. When
> purging bundle jobs and bundle actions, it does not always purge coordinator
> jobs, etc. This causes orphaned coordinators, meaning neither they nor their
> children will ever be purged due to the purge logic.
>
>
>
> When purging bundles, it first compiles a list of any coordinators which are
> not ready to purge [0]. It checks the coord list for status and
> coordOlderThan. If the no child coordinator meets these criteria, it adds it
> to the coordsToPurge list.
> Being added to the list does not guarantee that the coordinator will be
> purged however. The processCoordinators method also has logic to check if the
> children workflows are older than wfOlderThan [1]. If a purge command is
> started where wfOlderThan is much higher than coordOlderThan (for example the
> default values are 30 days for workflows and 7 days for coordinators), then
> the bundle will be purged, but the coordinator will not.
> Since the bundle is now purged, the child coordinator will never be purged
> because only parentless coordinators will be checked, since coordinators with
> parents will only be purged when the bundle is purged
> [0]
> {code:java}
> PurgeXCommand
> 380 long numChildrenNotReady = jpaService.execute(
> 381 new CoordJobsCountNotForPurgeFromParentIdJPAExecutor(coordOlderThan,
> bundleId));
> CoordinatorJobBean
> 192 @NamedQuery(name = "GET_COORD_COUNT_WITH_PARENT_ID_NOT_READY_FOR_PURGE",
> query = "select count(w) from CoordinatorJobBean"
> 193 + " w where w.bundleId = :parentId and (w.statusStr NOT IN ('SUCCEEDED',
> 'FAILED', 'KILLED', 'DONEWITHERROR') "
> 194 + "OR w.lastModifiedTimestamp >= :lastModTime)"),
> {code}
>
> [1]
> {code:java}
> PurgeXCommand
> 343 List workflowChildren = fetchTerminatedWorkflow(wfjBeanList);
> 344
> private boolean isWorkflowPurgeable(WorkflowJobBean wfjBean, long
> wfOlderThanMS) {
> 308 final Date wfEndTime = wfjBean.getEndTime();
> 309 final boolean isFinished = wfjBean.inTerminalState();
> 310 if (isFinished && wfEndTime != null && wfEndTime.getTime() <
> wfOlderThanMS)
> { 311 return true; 312 }
> 313 else {
> 314 final Date lastModificationTime = wfjBean.getLastModifiedTime();
> 315 if (isFinished && lastModificationTime != null &&
> lastModificationTime.getTime() < wfOlderThanMS)
> { 316 return true; 317 }
> 318 }
> 319 return false;
> 345 // if all workflow are ready to purge add them and add the coordinator
> and their actions
> 346 if(workflowChildren.size() == wfjBeanList.size()) {
> 347 LOG.debug("Purging coordinator " + coordId);
> 348 wfsToPurge.addAll(workflowChildren);
> 349 coordsToPurge.add(coordId);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OOZIE-3669) Fix purge process for bundles to prevent orphan coordinators
[
https://issues.apache.org/jira/browse/OOZIE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17632635#comment-17632635
]
ASF subversion and git services commented on OOZIE-3669:
Commit 5931b95f777b209c463dc74de88d245ab645263d in oozie's branch
refs/heads/master from Denes Bodo
[ https://gitbox.apache.org/repos/asf?p=oozie.git;h=5931b95f7 ]
OOZIE-3669 Fix purge process for bundles to prevent orphan coordinators (jmakai
via dionusos)
> Fix purge process for bundles to prevent orphan coordinators
>
>
> Key: OOZIE-3669
> URL: https://issues.apache.org/jira/browse/OOZIE-3669
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.2.1
>Reporter: Janos Makai
>Assignee: Janos Makai
>Priority: Major
> Attachments: OOZIE-3669-001.patch, OOZIE-3669-002.patch,
> OOZIE-3669-003.patch
>
>
> The Oozie purge process for bundles is creating orphan coordinators. When
> purging bundle jobs and bundle actions, it does not always purge coordinator
> jobs, etc. This causes orphaned coordinators, meaning neither they nor their
> children will ever be purged due to the purge logic.
>
>
>
> When purging bundles, it first compiles a list of any coordinators which are
> not ready to purge [0]. It checks the coord list for status and
> coordOlderThan. If the no child coordinator meets these criteria, it adds it
> to the coordsToPurge list.
> Being added to the list does not guarantee that the coordinator will be
> purged however. The processCoordinators method also has logic to check if the
> children workflows are older than wfOlderThan [1]. If a purge command is
> started where wfOlderThan is much higher than coordOlderThan (for example the
> default values are 30 days for workflows and 7 days for coordinators), then
> the bundle will be purged, but the coordinator will not.
> Since the bundle is now purged, the child coordinator will never be purged
> because only parentless coordinators will be checked, since coordinators with
> parents will only be purged when the bundle is purged
> [0]
> {code:java}
> PurgeXCommand
> 380 long numChildrenNotReady = jpaService.execute(
> 381 new CoordJobsCountNotForPurgeFromParentIdJPAExecutor(coordOlderThan,
> bundleId));
> CoordinatorJobBean
> 192 @NamedQuery(name = "GET_COORD_COUNT_WITH_PARENT_ID_NOT_READY_FOR_PURGE",
> query = "select count(w) from CoordinatorJobBean"
> 193 + " w where w.bundleId = :parentId and (w.statusStr NOT IN ('SUCCEEDED',
> 'FAILED', 'KILLED', 'DONEWITHERROR') "
> 194 + "OR w.lastModifiedTimestamp >= :lastModTime)"),
> {code}
>
> [1]
> {code:java}
> PurgeXCommand
> 343 List workflowChildren = fetchTerminatedWorkflow(wfjBeanList);
> 344
> private boolean isWorkflowPurgeable(WorkflowJobBean wfjBean, long
> wfOlderThanMS) {
> 308 final Date wfEndTime = wfjBean.getEndTime();
> 309 final boolean isFinished = wfjBean.inTerminalState();
> 310 if (isFinished && wfEndTime != null && wfEndTime.getTime() <
> wfOlderThanMS)
> { 311 return true; 312 }
> 313 else {
> 314 final Date lastModificationTime = wfjBean.getLastModifiedTime();
> 315 if (isFinished && lastModificationTime != null &&
> lastModificationTime.getTime() < wfOlderThanMS)
> { 316 return true; 317 }
> 318 }
> 319 return false;
> 345 // if all workflow are ready to purge add them and add the coordinator
> and their actions
> 346 if(workflowChildren.size() == wfjBeanList.size()) {
> 347 LOG.debug("Purging coordinator " + coordId);
> 348 wfsToPurge.addAll(workflowChildren);
> 349 coordsToPurge.add(coordId);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OOZIE-3669) Fix purge process for bundles to prevent orphan coordinators
[
https://issues.apache.org/jira/browse/OOZIE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17632501#comment-17632501
]
Hadoop QA commented on OOZIE-3669:
--
Testing JIRA OOZIE-3669
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [5] new bugs found below threshold in total that
must be fixed.
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:red}-1{color} There are [5] new bugs found below threshold in
[core] that must be fixed.
.You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
.The most important SpotBugs errors are:
.At BulkJPAExecutor.java:[line 206]: This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection
.At BulkJPAExecutor.java:[line 176]: At BulkJPAExecutor.java:[line 175]
.At BulkJPAExecutor.java:[line 205]: At BulkJPAExecutor.java:[line 199]
.This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 206]
.At BulkJPAExecutor.java:[line 111]: At BulkJPAExecutor.java:[line 127]
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:orange}0{color} There are [4] new bugs found in [server] that would
be nice to have fixed.
.You can find the SpotBugs diff here: server/findbugs-new.html
.{color:green}+1{color} There are no new bugs found in [docs].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3226
.{color:orange}Tests failed at first run:{color}
TestCoordActionsKillXCommand#testActionKillCommandActionNumbers
TestCoordActionsKillXCommand#testActionKillCommandDate
TestBlockingInputStream#testFastWritingBlockingInputStream
TestPriorityDelayQueue#testPeek
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}+1 MODERNIZER{color}
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://ci-hadoop.apache.org/job/PreCommit-OOZIE-Build/101/
> Fix purge process for bundles to prevent orphan coordinators
>
>
> Key: OOZIE-3669
> URL: https://issues.apache.org/jira/browse/OOZIE-3669
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions:
[jira] [Commented] (OOZIE-3669) Fix purge process for bundles to prevent orphan coordinators
[
https://issues.apache.org/jira/browse/OOZIE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17632456#comment-17632456
]
Hadoop QA commented on OOZIE-3669:
--
PreCommit-OOZIE-Build started
> Fix purge process for bundles to prevent orphan coordinators
>
>
> Key: OOZIE-3669
> URL: https://issues.apache.org/jira/browse/OOZIE-3669
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.2.1
>Reporter: Janos Makai
>Assignee: Janos Makai
>Priority: Major
> Attachments: OOZIE-3669-001.patch, OOZIE-3669-002.patch,
> OOZIE-3669-003.patch
>
>
> The Oozie purge process for bundles is creating orphan coordinators. When
> purging bundle jobs and bundle actions, it does not always purge coordinator
> jobs, etc. This causes orphaned coordinators, meaning neither they nor their
> children will ever be purged due to the purge logic.
>
>
>
> When purging bundles, it first compiles a list of any coordinators which are
> not ready to purge [0]. It checks the coord list for status and
> coordOlderThan. If the no child coordinator meets these criteria, it adds it
> to the coordsToPurge list.
> Being added to the list does not guarantee that the coordinator will be
> purged however. The processCoordinators method also has logic to check if the
> children workflows are older than wfOlderThan [1]. If a purge command is
> started where wfOlderThan is much higher than coordOlderThan (for example the
> default values are 30 days for workflows and 7 days for coordinators), then
> the bundle will be purged, but the coordinator will not.
> Since the bundle is now purged, the child coordinator will never be purged
> because only parentless coordinators will be checked, since coordinators with
> parents will only be purged when the bundle is purged
> [0]
> {code:java}
> PurgeXCommand
> 380 long numChildrenNotReady = jpaService.execute(
> 381 new CoordJobsCountNotForPurgeFromParentIdJPAExecutor(coordOlderThan,
> bundleId));
> CoordinatorJobBean
> 192 @NamedQuery(name = "GET_COORD_COUNT_WITH_PARENT_ID_NOT_READY_FOR_PURGE",
> query = "select count(w) from CoordinatorJobBean"
> 193 + " w where w.bundleId = :parentId and (w.statusStr NOT IN ('SUCCEEDED',
> 'FAILED', 'KILLED', 'DONEWITHERROR') "
> 194 + "OR w.lastModifiedTimestamp >= :lastModTime)"),
> {code}
>
> [1]
> {code:java}
> PurgeXCommand
> 343 List workflowChildren = fetchTerminatedWorkflow(wfjBeanList);
> 344
> private boolean isWorkflowPurgeable(WorkflowJobBean wfjBean, long
> wfOlderThanMS) {
> 308 final Date wfEndTime = wfjBean.getEndTime();
> 309 final boolean isFinished = wfjBean.inTerminalState();
> 310 if (isFinished && wfEndTime != null && wfEndTime.getTime() <
> wfOlderThanMS)
> { 311 return true; 312 }
> 313 else {
> 314 final Date lastModificationTime = wfjBean.getLastModifiedTime();
> 315 if (isFinished && lastModificationTime != null &&
> lastModificationTime.getTime() < wfOlderThanMS)
> { 316 return true; 317 }
> 318 }
> 319 return false;
> 345 // if all workflow are ready to purge add them and add the coordinator
> and their actions
> 346 if(workflowChildren.size() == wfjBeanList.size()) {
> 347 LOG.debug("Purging coordinator " + coordId);
> 348 wfsToPurge.addAll(workflowChildren);
> 349 coordsToPurge.add(coordId);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OOZIE-3669) Fix purge process for bundles to prevent orphan coordinators
[
https://issues.apache.org/jira/browse/OOZIE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17632216#comment-17632216
]
Hadoop QA commented on OOZIE-3669:
--
Testing JIRA OOZIE-3669
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [5] new bugs found below threshold in total that
must be fixed.
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:orange}0{color} There are [4] new bugs found in [server] that would
be nice to have fixed.
.You can find the SpotBugs diff here: server/findbugs-new.html
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:red}-1{color} There are [5] new bugs found below threshold in
[core] that must be fixed.
.You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
.The most important SpotBugs errors are:
.At BulkJPAExecutor.java:[line 206]: This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection
.At BulkJPAExecutor.java:[line 176]: At BulkJPAExecutor.java:[line 175]
.At BulkJPAExecutor.java:[line 205]: At BulkJPAExecutor.java:[line 199]
.This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 206]
.At BulkJPAExecutor.java:[line 111]: At BulkJPAExecutor.java:[line 127]
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3224
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}+1 MODERNIZER{color}
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://ci-hadoop.apache.org/job/PreCommit-OOZIE-Build/98/
> Fix purge process for bundles to prevent orphan coordinators
>
>
> Key: OOZIE-3669
> URL: https://issues.apache.org/jira/browse/OOZIE-3669
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.2.1
>Reporter: Janos Makai
>Assignee: Janos Makai
>Priority: Major
> Attachments: OOZIE-3669-001.patch, OOZIE-3669-002.patch
>
>
> The Oozie purge process for bundles is creating orphan coordinators. When
> purging bundle jobs and bundle actions, it does not always purge coordinator
> j
[jira] [Commented] (OOZIE-3669) Fix purge process for bundles to prevent orphan coordinators
[
https://issues.apache.org/jira/browse/OOZIE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17632141#comment-17632141
]
Hadoop QA commented on OOZIE-3669:
--
PreCommit-OOZIE-Build started
> Fix purge process for bundles to prevent orphan coordinators
>
>
> Key: OOZIE-3669
> URL: https://issues.apache.org/jira/browse/OOZIE-3669
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.2.1
>Reporter: Janos Makai
>Assignee: Janos Makai
>Priority: Major
> Attachments: OOZIE-3669-001.patch, OOZIE-3669-002.patch
>
>
> The Oozie purge process for bundles is creating orphan coordinators. When
> purging bundle jobs and bundle actions, it does not always purge coordinator
> jobs, etc. This causes orphaned coordinators, meaning neither they nor their
> children will ever be purged due to the purge logic.
>
>
>
> When purging bundles, it first compiles a list of any coordinators which are
> not ready to purge [0]. It checks the coord list for status and
> coordOlderThan. If the no child coordinator meets these criteria, it adds it
> to the coordsToPurge list.
> Being added to the list does not guarantee that the coordinator will be
> purged however. The processCoordinators method also has logic to check if the
> children workflows are older than wfOlderThan [1]. If a purge command is
> started where wfOlderThan is much higher than coordOlderThan (for example the
> default values are 30 days for workflows and 7 days for coordinators), then
> the bundle will be purged, but the coordinator will not.
> Since the bundle is now purged, the child coordinator will never be purged
> because only parentless coordinators will be checked, since coordinators with
> parents will only be purged when the bundle is purged
> [0]
> {code:java}
> PurgeXCommand
> 380 long numChildrenNotReady = jpaService.execute(
> 381 new CoordJobsCountNotForPurgeFromParentIdJPAExecutor(coordOlderThan,
> bundleId));
> CoordinatorJobBean
> 192 @NamedQuery(name = "GET_COORD_COUNT_WITH_PARENT_ID_NOT_READY_FOR_PURGE",
> query = "select count(w) from CoordinatorJobBean"
> 193 + " w where w.bundleId = :parentId and (w.statusStr NOT IN ('SUCCEEDED',
> 'FAILED', 'KILLED', 'DONEWITHERROR') "
> 194 + "OR w.lastModifiedTimestamp >= :lastModTime)"),
> {code}
>
> [1]
> {code:java}
> PurgeXCommand
> 343 List workflowChildren = fetchTerminatedWorkflow(wfjBeanList);
> 344
> private boolean isWorkflowPurgeable(WorkflowJobBean wfjBean, long
> wfOlderThanMS) {
> 308 final Date wfEndTime = wfjBean.getEndTime();
> 309 final boolean isFinished = wfjBean.inTerminalState();
> 310 if (isFinished && wfEndTime != null && wfEndTime.getTime() <
> wfOlderThanMS)
> { 311 return true; 312 }
> 313 else {
> 314 final Date lastModificationTime = wfjBean.getLastModifiedTime();
> 315 if (isFinished && lastModificationTime != null &&
> lastModificationTime.getTime() < wfOlderThanMS)
> { 316 return true; 317 }
> 318 }
> 319 return false;
> 345 // if all workflow are ready to purge add them and add the coordinator
> and their actions
> 346 if(workflowChildren.size() == wfjBeanList.size()) {
> 347 LOG.debug("Purging coordinator " + coordId);
> 348 wfsToPurge.addAll(workflowChildren);
> 349 coordsToPurge.add(coordId);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OOZIE-3669) Fix purge process for bundles to prevent orphan coordinators
[
https://issues.apache.org/jira/browse/OOZIE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17632138#comment-17632138
]
Janos Makai commented on OOZIE-3669:
Sure [~dionusos], I've just uploaded the fixed patch.
> Fix purge process for bundles to prevent orphan coordinators
>
>
> Key: OOZIE-3669
> URL: https://issues.apache.org/jira/browse/OOZIE-3669
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.2.1
>Reporter: Janos Makai
>Assignee: Janos Makai
>Priority: Major
> Attachments: OOZIE-3669-001.patch, OOZIE-3669-002.patch
>
>
> The Oozie purge process for bundles is creating orphan coordinators. When
> purging bundle jobs and bundle actions, it does not always purge coordinator
> jobs, etc. This causes orphaned coordinators, meaning neither they nor their
> children will ever be purged due to the purge logic.
>
>
>
> When purging bundles, it first compiles a list of any coordinators which are
> not ready to purge [0]. It checks the coord list for status and
> coordOlderThan. If the no child coordinator meets these criteria, it adds it
> to the coordsToPurge list.
> Being added to the list does not guarantee that the coordinator will be
> purged however. The processCoordinators method also has logic to check if the
> children workflows are older than wfOlderThan [1]. If a purge command is
> started where wfOlderThan is much higher than coordOlderThan (for example the
> default values are 30 days for workflows and 7 days for coordinators), then
> the bundle will be purged, but the coordinator will not.
> Since the bundle is now purged, the child coordinator will never be purged
> because only parentless coordinators will be checked, since coordinators with
> parents will only be purged when the bundle is purged
> [0]
> {code:java}
> PurgeXCommand
> 380 long numChildrenNotReady = jpaService.execute(
> 381 new CoordJobsCountNotForPurgeFromParentIdJPAExecutor(coordOlderThan,
> bundleId));
> CoordinatorJobBean
> 192 @NamedQuery(name = "GET_COORD_COUNT_WITH_PARENT_ID_NOT_READY_FOR_PURGE",
> query = "select count(w) from CoordinatorJobBean"
> 193 + " w where w.bundleId = :parentId and (w.statusStr NOT IN ('SUCCEEDED',
> 'FAILED', 'KILLED', 'DONEWITHERROR') "
> 194 + "OR w.lastModifiedTimestamp >= :lastModTime)"),
> {code}
>
> [1]
> {code:java}
> PurgeXCommand
> 343 List workflowChildren = fetchTerminatedWorkflow(wfjBeanList);
> 344
> private boolean isWorkflowPurgeable(WorkflowJobBean wfjBean, long
> wfOlderThanMS) {
> 308 final Date wfEndTime = wfjBean.getEndTime();
> 309 final boolean isFinished = wfjBean.inTerminalState();
> 310 if (isFinished && wfEndTime != null && wfEndTime.getTime() <
> wfOlderThanMS)
> { 311 return true; 312 }
> 313 else {
> 314 final Date lastModificationTime = wfjBean.getLastModifiedTime();
> 315 if (isFinished && lastModificationTime != null &&
> lastModificationTime.getTime() < wfOlderThanMS)
> { 316 return true; 317 }
> 318 }
> 319 return false;
> 345 // if all workflow are ready to purge add them and add the coordinator
> and their actions
> 346 if(workflowChildren.size() == wfjBeanList.size()) {
> 347 LOG.debug("Purging coordinator " + coordId);
> 348 wfsToPurge.addAll(workflowChildren);
> 349 coordsToPurge.add(coordId);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OOZIE-3669) Fix purge process for bundles to prevent orphan coordinators
[
https://issues.apache.org/jira/browse/OOZIE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17631875#comment-17631875
]
Dénes Bodó commented on OOZIE-3669:
---
[~jmakai] Thanks for the patch. Can you please fix the too long line? Thank you.
> Fix purge process for bundles to prevent orphan coordinators
>
>
> Key: OOZIE-3669
> URL: https://issues.apache.org/jira/browse/OOZIE-3669
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.2.1
>Reporter: Janos Makai
>Assignee: Janos Makai
>Priority: Major
> Attachments: OOZIE-3669-001.patch
>
>
> The Oozie purge process for bundles is creating orphan coordinators. When
> purging bundle jobs and bundle actions, it does not always purge coordinator
> jobs, etc. This causes orphaned coordinators, meaning neither they nor their
> children will ever be purged due to the purge logic.
>
>
>
> When purging bundles, it first compiles a list of any coordinators which are
> not ready to purge [0]. It checks the coord list for status and
> coordOlderThan. If the no child coordinator meets these criteria, it adds it
> to the coordsToPurge list.
> Being added to the list does not guarantee that the coordinator will be
> purged however. The processCoordinators method also has logic to check if the
> children workflows are older than wfOlderThan [1]. If a purge command is
> started where wfOlderThan is much higher than coordOlderThan (for example the
> default values are 30 days for workflows and 7 days for coordinators), then
> the bundle will be purged, but the coordinator will not.
> Since the bundle is now purged, the child coordinator will never be purged
> because only parentless coordinators will be checked, since coordinators with
> parents will only be purged when the bundle is purged
> [0]
> {code:java}
> PurgeXCommand
> 380 long numChildrenNotReady = jpaService.execute(
> 381 new CoordJobsCountNotForPurgeFromParentIdJPAExecutor(coordOlderThan,
> bundleId));
> CoordinatorJobBean
> 192 @NamedQuery(name = "GET_COORD_COUNT_WITH_PARENT_ID_NOT_READY_FOR_PURGE",
> query = "select count(w) from CoordinatorJobBean"
> 193 + " w where w.bundleId = :parentId and (w.statusStr NOT IN ('SUCCEEDED',
> 'FAILED', 'KILLED', 'DONEWITHERROR') "
> 194 + "OR w.lastModifiedTimestamp >= :lastModTime)"),
> {code}
>
> [1]
> {code:java}
> PurgeXCommand
> 343 List workflowChildren = fetchTerminatedWorkflow(wfjBeanList);
> 344
> private boolean isWorkflowPurgeable(WorkflowJobBean wfjBean, long
> wfOlderThanMS) {
> 308 final Date wfEndTime = wfjBean.getEndTime();
> 309 final boolean isFinished = wfjBean.inTerminalState();
> 310 if (isFinished && wfEndTime != null && wfEndTime.getTime() <
> wfOlderThanMS)
> { 311 return true; 312 }
> 313 else {
> 314 final Date lastModificationTime = wfjBean.getLastModifiedTime();
> 315 if (isFinished && lastModificationTime != null &&
> lastModificationTime.getTime() < wfOlderThanMS)
> { 316 return true; 317 }
> 318 }
> 319 return false;
> 345 // if all workflow are ready to purge add them and add the coordinator
> and their actions
> 346 if(workflowChildren.size() == wfjBeanList.size()) {
> 347 LOG.debug("Purging coordinator " + coordId);
> 348 wfsToPurge.addAll(workflowChildren);
> 349 coordsToPurge.add(coordId);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OOZIE-3669) Fix purge process for bundles to prevent orphan coordinators
[
https://issues.apache.org/jira/browse/OOZIE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17631869#comment-17631869
]
Hadoop QA commented on OOZIE-3669:
--
Testing JIRA OOZIE-3669
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:red}-1{color} the patch contains 7 line(s) longer than 132
characters
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [5] new bugs found below threshold in total that
must be fixed.
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:orange}0{color} There are [4] new bugs found in [server] that would
be nice to have fixed.
.You can find the SpotBugs diff here: server/findbugs-new.html
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:red}-1{color} There are [5] new bugs found below threshold in
[core] that must be fixed.
.You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
.The most important SpotBugs errors are:
.At BulkJPAExecutor.java:[line 206]: This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection
.At BulkJPAExecutor.java:[line 176]: At BulkJPAExecutor.java:[line 175]
.At BulkJPAExecutor.java:[line 205]: At BulkJPAExecutor.java:[line 199]
.This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 206]
.At BulkJPAExecutor.java:[line 111]: At BulkJPAExecutor.java:[line 127]
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3226
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}+1 MODERNIZER{color}
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://ci-hadoop.apache.org/job/PreCommit-OOZIE-Build/97/
> Fix purge process for bundles to prevent orphan coordinators
>
>
> Key: OOZIE-3669
> URL: https://issues.apache.org/jira/browse/OOZIE-3669
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.2.1
>Reporter: Janos Makai
>Assignee: Janos Makai
>Priority: Major
> Attachments: OOZIE-3669-001.patch
>
>
> The Oozie purge process for bundles is creating orphan coordinators. When
> purging bundle jobs and bundle actions, it does not always purge coordinator
> jobs, etc. This causes orp
[jira] [Commented] (OOZIE-3669) Fix purge process for bundles to prevent orphan coordinators
[
https://issues.apache.org/jira/browse/OOZIE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17631796#comment-17631796
]
Hadoop QA commented on OOZIE-3669:
--
PreCommit-OOZIE-Build started
> Fix purge process for bundles to prevent orphan coordinators
>
>
> Key: OOZIE-3669
> URL: https://issues.apache.org/jira/browse/OOZIE-3669
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.2.1
>Reporter: Janos Makai
>Assignee: Janos Makai
>Priority: Major
> Attachments: OOZIE-3669-001.patch
>
>
> The Oozie purge process for bundles is creating orphan coordinators. When
> purging bundle jobs and bundle actions, it does not always purge coordinator
> jobs, etc. This causes orphaned coordinators, meaning neither they nor their
> children will ever be purged due to the purge logic.
>
>
>
> When purging bundles, it first compiles a list of any coordinators which are
> not ready to purge [0]. It checks the coord list for status and
> coordOlderThan. If the no child coordinator meets these criteria, it adds it
> to the coordsToPurge list.
> Being added to the list does not guarantee that the coordinator will be
> purged however. The processCoordinators method also has logic to check if the
> children workflows are older than wfOlderThan [1]. If a purge command is
> started where wfOlderThan is much higher than coordOlderThan (for example the
> default values are 30 days for workflows and 7 days for coordinators), then
> the bundle will be purged, but the coordinator will not.
> Since the bundle is now purged, the child coordinator will never be purged
> because only parentless coordinators will be checked, since coordinators with
> parents will only be purged when the bundle is purged
> [0]
> {code:java}
> PurgeXCommand
> 380 long numChildrenNotReady = jpaService.execute(
> 381 new CoordJobsCountNotForPurgeFromParentIdJPAExecutor(coordOlderThan,
> bundleId));
> CoordinatorJobBean
> 192 @NamedQuery(name = "GET_COORD_COUNT_WITH_PARENT_ID_NOT_READY_FOR_PURGE",
> query = "select count(w) from CoordinatorJobBean"
> 193 + " w where w.bundleId = :parentId and (w.statusStr NOT IN ('SUCCEEDED',
> 'FAILED', 'KILLED', 'DONEWITHERROR') "
> 194 + "OR w.lastModifiedTimestamp >= :lastModTime)"),
> {code}
>
> [1]
> {code:java}
> PurgeXCommand
> 343 List workflowChildren = fetchTerminatedWorkflow(wfjBeanList);
> 344
> private boolean isWorkflowPurgeable(WorkflowJobBean wfjBean, long
> wfOlderThanMS) {
> 308 final Date wfEndTime = wfjBean.getEndTime();
> 309 final boolean isFinished = wfjBean.inTerminalState();
> 310 if (isFinished && wfEndTime != null && wfEndTime.getTime() <
> wfOlderThanMS)
> { 311 return true; 312 }
> 313 else {
> 314 final Date lastModificationTime = wfjBean.getLastModifiedTime();
> 315 if (isFinished && lastModificationTime != null &&
> lastModificationTime.getTime() < wfOlderThanMS)
> { 316 return true; 317 }
> 318 }
> 319 return false;
> 345 // if all workflow are ready to purge add them and add the coordinator
> and their actions
> 346 if(workflowChildren.size() == wfjBeanList.size()) {
> 347 LOG.debug("Purging coordinator " + coordId);
> 348 wfsToPurge.addAll(workflowChildren);
> 349 coordsToPurge.add(coordId);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
