[jira] [Updated] (OOZIE-2644) Skip queuing Notification Commands when there's nothing to notify
[
https://issues.apache.org/jira/browse/OOZIE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Azrael Seoeun updated OOZIE-2644:
-
Attachment: OOZIE-2644.2.patch
Fix test failures.
> Skip queuing Notification Commands when there's nothing to notify
> -
>
> Key: OOZIE-2644
> URL: https://issues.apache.org/jira/browse/OOZIE-2644
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: 3.1.3
>Reporter: Robert Kanter
>Assignee: Azrael Seoeun
> Fix For: 5.0.0
>
> Attachments: OOZIE-2644.1.patch, OOZIE-2644.2.patch
>
>
> When you use the
> [Workflow|https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a5_Workflow_Notifications]
> or
> [Coordinator|https://oozie.apache.org/docs/4.2.0/CoordinatorFunctionalSpec.html#a15._Coordinator_Notifications]
> Notification features, Oozie can end up queuing up a lot of
> {{WorkflowNotificationXCommand}} and {{CoordActionNotificationXCommand}}.
> This happens even if there's no notification configured on the job (which I
> imagine is most of the time); in this case, the {{execute}} method simply
> does nothing. This is wasteful and clogs the queue up.
> We should change the code so that it doesn't queue up one of these Commands
> unless there's actually a URL to notify.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OOZIE-2644) Skip queuing Notification Commands when there's nothing to notify
[
https://issues.apache.org/jira/browse/OOZIE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Azrael Seoeun updated OOZIE-2644:
-
Attachment: OOZIE-2644.1.patch
The patch is addressed comments.
As of now, CallableQueueService increase one more count in queued group for
instrumentation if CompositeCallable queued. So, if CompositeCallable which
contain 2 callables is queued, it add 3 to queued count. I am not sure it is
intended or not. If it is not intended, I can create separate Jira issue.
And this patch fix the NullPointerException when CoordActionNotification
executed from CoordActionKillXCommand. This is caused the runConf of actionBean
is not set. I can create separate Jira issue if needed.
CoordActionNotificationXCommand.java
{noformat}
Configuration conf;
try {
conf = new XConfiguration(new
StringReader(actionBean.getRunConf()));
}
catch (IOException e1) {
LOG.warn("Configuration parse error. read from DB :" +
actionBean.getRunConf());
throw new CommandException(ErrorCode.E1005, e1.getMessage(), e1);
}
{noformat}
> Skip queuing Notification Commands when there's nothing to notify
> -
>
> Key: OOZIE-2644
> URL: https://issues.apache.org/jira/browse/OOZIE-2644
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: 3.1.3
>Reporter: Robert Kanter
>Assignee: Azrael Seoeun
> Fix For: 5.0.0
>
> Attachments: OOZIE-2644.1.patch
>
>
> When you use the
> [Workflow|https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a5_Workflow_Notifications]
> or
> [Coordinator|https://oozie.apache.org/docs/4.2.0/CoordinatorFunctionalSpec.html#a15._Coordinator_Notifications]
> Notification features, Oozie can end up queuing up a lot of
> {{WorkflowNotificationXCommand}} and {{CoordActionNotificationXCommand}}.
> This happens even if there's no notification configured on the job (which I
> imagine is most of the time); in this case, the {{execute}} method simply
> does nothing. This is wasteful and clogs the queue up.
> We should change the code so that it doesn't queue up one of these Commands
> unless there's actually a URL to notify.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (OOZIE-2644) Skip queuing Notification Commands when there's nothing to notify
[
https://issues.apache.org/jira/browse/OOZIE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Azrael Seoeun reassigned OOZIE-2644:
Assignee: Azrael Seoeun
> Skip queuing Notification Commands when there's nothing to notify
> -
>
> Key: OOZIE-2644
> URL: https://issues.apache.org/jira/browse/OOZIE-2644
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: 3.1.3
>Reporter: Robert Kanter
>Assignee: Azrael Seoeun
> Fix For: 4.4.0
>
>
> When you use the
> [Workflow|https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a5_Workflow_Notifications]
> or
> [Coordinator|https://oozie.apache.org/docs/4.2.0/CoordinatorFunctionalSpec.html#a15._Coordinator_Notifications]
> Notification features, Oozie can end up queuing up a lot of
> {{WorkflowNotificationXCommand}} and {{CoordActionNotificationXCommand}}.
> This happens even if there's no notification configured on the job (which I
> imagine is most of the time); in this case, the {{execute}} method simply
> does nothing. This is wasteful and clogs the queue up.
> We should change the code so that it doesn't queue up one of these Commands
> unless there's actually a URL to notify.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2134) Remove references to Services.get().getConf() in code
[ https://issues.apache.org/jira/browse/OOZIE-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15469282#comment-15469282 ] Azrael Seoeun commented on OOZIE-2134: -- The patch does compile fine. I wonder why the patch for testing does not compile. Please let me know if I am missing something. > Remove references to Services.get().getConf() in code > - > > Key: OOZIE-2134 > URL: https://issues.apache.org/jira/browse/OOZIE-2134 > Project: Oozie > Issue Type: Task >Reporter: Rohini Palaniswamy >Assignee: Azrael Seoeun > Labels: newbie > Attachments: OOZIE-2134.1.patch, OOZIE-2134.2.patch, > OOZIE-2134.3.patch, OOZIE-2134.4.patch > > >Make getConf() package protected and switch all references of > Services.get().getConf() to ConfigurationService. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OOZIE-2134) Remove references to Services.get().getConf() in code
[ https://issues.apache.org/jira/browse/OOZIE-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Azrael Seoeun updated OOZIE-2134: - Attachment: OOZIE-2134.4.patch * Remove ConfigrautionService#getInt(String name, int default) We should not use code default, use oozie-default.xml. (OOZIE-1890) * Add XTestCase#initNewServices(Properties configuration) Construct and initialize Services with configuration for testing. > Remove references to Services.get().getConf() in code > - > > Key: OOZIE-2134 > URL: https://issues.apache.org/jira/browse/OOZIE-2134 > Project: Oozie > Issue Type: Task >Reporter: Rohini Palaniswamy >Assignee: Azrael Seoeun > Labels: newbie > Attachments: OOZIE-2134.1.patch, OOZIE-2134.2.patch, > OOZIE-2134.3.patch, OOZIE-2134.4.patch > > >Make getConf() package protected and switch all references of > Services.get().getConf() to ConfigurationService. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2134) Remove references to Services.get().getConf() in code
[ https://issues.apache.org/jira/browse/OOZIE-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15446057#comment-15446057 ] Azrael Seoeun commented on OOZIE-2134: -- Sure, I will upload the rebased one soon. > Remove references to Services.get().getConf() in code > - > > Key: OOZIE-2134 > URL: https://issues.apache.org/jira/browse/OOZIE-2134 > Project: Oozie > Issue Type: Task >Reporter: Rohini Palaniswamy >Assignee: Azrael Seoeun > Labels: newbie > Attachments: OOZIE-2134.1.patch, OOZIE-2134.2.patch, > OOZIE-2134.3.patch > > >Make getConf() package protected and switch all references of > Services.get().getConf() to ConfigurationService. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OOZIE-2081) WorkflowJob notification to include coordinator action id
[ https://issues.apache.org/jira/browse/OOZIE-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Azrael Seoeun updated OOZIE-2081: - Attachment: OOZIE-2081.3.patch Addressed comment. > WorkflowJob notification to include coordinator action id > -- > > Key: OOZIE-2081 > URL: https://issues.apache.org/jira/browse/OOZIE-2081 > Project: Oozie > Issue Type: Improvement > Components: core >Reporter: Venkatesan Ramachandran >Assignee: Azrael Seoeun > Fix For: 4.4.0 > > Attachments: OOZIE-2081.1.patch, OOZIE-2081.2.patch, > OOZIE-2081.3.patch > > > WorkflowJob notification to include parent_id (coordinator action id) if > present. This will help identify if a given workflow job is started by a > coordinator action. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OOZIE-1922) MemoryLocksService fails if lock is acquired multiple times in same thread and released
[
https://issues.apache.org/jira/browse/OOZIE-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Azrael Seoeun updated OOZIE-1922:
-
Assignee: Purshotam Shah (was: Azrael Seoeun)
> MemoryLocksService fails if lock is acquired multiple times in same thread
> and released
> ---
>
> Key: OOZIE-1922
> URL: https://issues.apache.org/jira/browse/OOZIE-1922
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-1922-V1.patch, OOZIE-1922.1.patch,
> OOZIE-1922.2.patch, OOZIE-1922.3.patch
>
>
> ReentrantLock can be acquired multiple times in same thread. For multiple
> acquire call, ReentrantLock hold count is incremented by one.
> So if we acquire lock multiple time from same thread, all will be successful
> and hold count is increased for every call.
> But if we release lock, MemoryLocksService ignore the count and deletes the
> lock. Even if it's held by some command.
> Simple step can reproduce it.
> {code}
> service.getWriteLock("test", 5000); //writeHoldCount = 1
> MemoryLockToken lock = (MemoryLockToken)service.getWriteLock("test",
> 5000); //writeHoldCount = 2
> lock.release(); //writeHoldCount = 1
> lock = (MemoryLockToken)service.getWriteLock("test", 5000);
> //writeHoldCount = 1, it should be 2.
> {code}
> {code}
> @Override
> public void release() {
> int val = rwLock.getQueueLength();
> if (val == 0) {
> synchronized (locks) {
> locks.remove(resource);
> }
> }
> lock.unlock();
> }
> }
> {code}
> MemoryLocks should check hold count before removing lock.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2293) Oozie 4.0.1 build failed while building Catalog
[ https://issues.apache.org/jira/browse/OOZIE-2293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006737#comment-15006737 ] Azrael Seoeun commented on OOZIE-2293: -- Please backport OOZIE-2293 to branch-4.2. We can not build the latest stable version (4.2). > Oozie 4.0.1 build failed while building Catalog > --- > > Key: OOZIE-2293 > URL: https://issues.apache.org/jira/browse/OOZIE-2293 > Project: Oozie > Issue Type: Bug >Reporter: Janarthanan >Assignee: Ravi Prakash >Priority: Critical > Fix For: trunk > > Attachments: OOZIE-2293.01.patch, OOZIE-2293.02.patch > > > Building oozie 4.0.1 with maven failed in Catalog build. Please let me know > how to resolve this. > Is HCatalog required to be installed prior to oozie. > Downloading: http://repository.codehaus.org/asm/asm/maven-metadata.xml > [WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to > Codehaus repository (http://repository.codehaus.org/): Failed to transfer > file: http://repository.codehaus.org/asm/asm/maven-metadata.xml. Return code > is: 410 , ReasonPhrase:Gone. > [WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to > local.repository (file:../../local.repository/trunk): Cannot access > file:../../local.repository/trunk with type legacy using the available > connector factories: BasicRepositoryConnectorFactory > Downloading: > http://repository.codehaus.org/javax/jdo/jdo2-api/2.3-ec/jdo2-api-2.3-ec.pom > [WARNING] Failure to transfer asm:asm/maven-metadata.xml from > http://repository.codehaus.org/ was cached in the local repository, > resolution will not be reattempted until the update interval of Codehaus > repository has elapsed or updates are forced. Original error: Could not > transfer metadata asm:asm/maven-metadata.xml from/to Codehaus repository > (http://repository.codehaus.org/): Failed to transfer file: > http://repository.codehaus.org/asm/asm/maven-metadata.xml. Return code is: > 410 , ReasonPhrase:Gone. > [WARNING] Failure to transfer asm:asm/maven-metadata.xml from > file:../../local.repository/trunk was cached in the local repository, > resolution will not be reattempted until the update interval of > local.repository has elapsed or updates are forced. Original error: Could not > transfer metadata asm:asm/maven-metadata.xml from/to local.repository > (file:../../local.repository/trunk): Cannot access > file:../../local.repository/trunk with type legacy using the available > connector factories: BasicRepositoryConnectorFactory > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Apache Oozie Main .. SUCCESS [ 6.031 > s] > [INFO] Apache Oozie Client SUCCESS [ 7.637 > s] > [INFO] Apache Oozie Hadoop 1.1.1.oozie-4.0.1 .. SUCCESS [ 1.004 > s] > [INFO] Apache Oozie Hadoop Distcp 1.1.1.oozie-4.0.1 ... SUCCESS [ 0.749 > s] > [INFO] Apache Oozie Hadoop 1.1.1.oozie-4.0.1 Test . SUCCESS [ 0.975 > s] > [INFO] Apache Oozie Hadoop 2.3.0.oozie-4.0.1 .. SUCCESS [ 1.855 > s] > [INFO] Apache Oozie Hadoop 2.3.0.oozie-4.0.1 Test . SUCCESS [ 1.206 > s] > [INFO] Apache Oozie Hadoop Distcp 2.3.0.oozie-4.0.1 ... SUCCESS [ 0.416 > s] > [INFO] Apache Oozie Hadoop 0.23.5.oozie-4.0.1 . SUCCESS [ 1.594 > s] > [INFO] Apache Oozie Hadoop 0.23.5.oozie-4.0.1 Test SUCCESS [ 1.134 > s] > [INFO] Apache Oozie Hadoop Distcp 0.23.5.oozie-4.0.1 .. SUCCESS [ 0.375 > s] > [INFO] Apache Oozie Hadoop Libs ... SUCCESS [ 0.061 > s] > [INFO] Apache Oozie Hbase 0.94.2.oozie-4.0.1 .. SUCCESS [ 0.423 > s] > [INFO] Apache Oozie Hbase Libs SUCCESS [ 0.058 > s] > [INFO] Apache Oozie HCatalog 0.5.0.oozie-4.0.1 FAILURE [ 2.587 > s] > [INFO] Apache Oozie HCatalog 0.6.0.oozie-4.0.1 SKIPPED -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OOZIE-1918) ActionXCommand refactoring for code reuse
[ https://issues.apache.org/jira/browse/OOZIE-1918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Azrael Seoeun updated OOZIE-1918: - Attachment: OOZIE-1918.3.noprefix.patch [~jaydeepvishwakarma] I upload new patch. This addressed the comments and rebased. Thanks. ActionXCommand refactoring for code reuse - Key: OOZIE-1918 URL: https://issues.apache.org/jira/browse/OOZIE-1918 Project: Oozie Issue Type: Improvement Components: action Affects Versions: trunk Reporter: Azrael Seoeun Assignee: Azrael Seoeun Priority: Trivial Attachments: OOZIE-1918.1.patch, OOZIE-1918.2.patch, OOZIE-1918.3.noprefix.patch Subclasses of ActionXCommand have some duplicated code. Refactoring make it more neat. This patch contain: * Renaming CompletedActionXCommand to ActionCompletedXCommand and extending ActionXCommand * Moving some variables and methods to ActionXCommand and making it protected * More meaningful log message : actionId is duplicated so removed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OOZIE-1802) Support workflow action log
[
https://issues.apache.org/jira/browse/OOZIE-1802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Azrael Seoeun resolved OOZIE-1802.
--
Resolution: Invalid
[~rohini] No, we can do this using logfilter as you mentioned.
{noformat}
oozie job -oozie http://sembp:11000/oozie -log
000-150705001901308-oozie-seoe-W -logfilter text=shell-2
{noformat}
Thanks [~rohini], [~jaydeepvishwakarma].
Support workflow action log
---
Key: OOZIE-1802
URL: https://issues.apache.org/jira/browse/OOZIE-1802
Project: Oozie
Issue Type: Improvement
Components: action
Affects Versions: trunk
Reporter: Azrael Seoeun
Assignee: Azrael Seoeun
Priority: Minor
Attachments: OOZIE-1802.1.patch
If workflow contains lots of actions, it is hard to search the specific
action log. It provides the log of workflow action.
{code}
$ oozie job -log 002-140421113135686-oozie-seoe-W@shell-1
2014-04-21 13:05:52,968 INFO ActionStartXCommand:539 - SERVER[seair.local]
USER[seoeun] GROUP[user] TOKEN[] APP[test]
JOB[002-140421113135686-oozie-seoe-W]
ACTION[002-140421113135686-oozie-seoe-W@shell-1] Start action
[002-140421113135686-oozie-seoe-W@shell-1] with user-retry state :
userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2014-04-21 13:05:53,238 INFO ShellActionExecutor:539 - SERVER[seair.local]
USER[seoeun] GROUP[user] TOKEN[] APP[test]
JOB[002-140421113135686-oozie-seoe-W]
ACTION[002-140421113135686-oozie-seoe-W@shell-1] checking action, hadoop
job ID [job_201404211041_0005] status [RUNNING]
2014-04-21 13:05:53,241 WARN ActionStartXCommand:542 - SERVER[seair.local]
USER[seoeun] GROUP[user] TOKEN[] APP[test]
JOB[002-140421113135686-oozie-seoe-W]
ACTION[002-140421113135686-oozie-seoe-W@shell-1]
[***002-140421113135686-oozie-seoe-W@shell-1***]Action status=RUNNING
2014-04-21 13:05:53,242 WARN ActionStartXCommand:542 - SERVER[seair.local]
USER[seoeun] GROUP[user] TOKEN[] APP[test]
JOB[002-140421113135686-oozie-seoe-W]
ACTION[002-140421113135686-oozie-seoe-W@shell-1]
[***002-140421113135686-oozie-seoe-W@shell-1***]Action updated in DB!
2014-04-21 13:06:00,520 INFO CallbackServlet:539 - SERVER[seair.local]
USER[-] GROUP[-] TOKEN[-] APP[-] JOB[002-140421113135686-oozie-seoe-W]
ACTION[002-140421113135686-oozie-seoe-W@shell-1] callback for action
[002-140421113135686-oozie-seoe-W@shell-1]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OOZIE-1922) MemoryLocksService fails if lock is acquired multiple times in same thread and released
[
https://issues.apache.org/jira/browse/OOZIE-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Azrael Seoeun updated OOZIE-1922:
-
Attachment: OOZIE-1922.2.patch
Upload new patch rebased on the latest branch.
MemoryLocksService fails if lock is acquired multiple times in same thread
and released
---
Key: OOZIE-1922
URL: https://issues.apache.org/jira/browse/OOZIE-1922
Project: Oozie
Issue Type: Bug
Reporter: Purshotam Shah
Assignee: Azrael Seoeun
Attachments: OOZIE-1922.1.patch, OOZIE-1922.2.patch
ReentrantLock can be acquired multiple times in same thread. For multiple
acquire call, ReentrantLock hold count is incremented by one.
So if we acquire lock multiple time from same thread, all will be successful
and hold count is increased for every call.
But if we release lock, MemoryLocksService ignore the count and deletes the
lock. Even if it's held by some command.
Simple step can reproduce it.
{code}
service.getWriteLock(test, 5000); //writeHoldCount = 1
MemoryLockToken lock = (MemoryLockToken)service.getWriteLock(test,
5000); //writeHoldCount = 2
lock.release(); //writeHoldCount = 1
lock = (MemoryLockToken)service.getWriteLock(test, 5000);
//writeHoldCount = 1, it should be 2.
{code}
{code}
@Override
public void release() {
int val = rwLock.getQueueLength();
if (val == 0) {
synchronized (locks) {
locks.remove(resource);
}
}
lock.unlock();
}
}
{code}
MemoryLocks should check hold count before removing lock.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OOZIE-1922) MemoryLocksService fails if lock is acquired multiple times in same thread and released
[
https://issues.apache.org/jira/browse/OOZIE-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Azrael Seoeun updated OOZIE-1922:
-
Attachment: OOZIE-1922.3.patch
OOZIE-1922.2.patch included wrong file. I upload the fixed one.
MemoryLocksService fails if lock is acquired multiple times in same thread
and released
---
Key: OOZIE-1922
URL: https://issues.apache.org/jira/browse/OOZIE-1922
Project: Oozie
Issue Type: Bug
Reporter: Purshotam Shah
Assignee: Azrael Seoeun
Attachments: OOZIE-1922.1.patch, OOZIE-1922.2.patch,
OOZIE-1922.3.patch
ReentrantLock can be acquired multiple times in same thread. For multiple
acquire call, ReentrantLock hold count is incremented by one.
So if we acquire lock multiple time from same thread, all will be successful
and hold count is increased for every call.
But if we release lock, MemoryLocksService ignore the count and deletes the
lock. Even if it's held by some command.
Simple step can reproduce it.
{code}
service.getWriteLock(test, 5000); //writeHoldCount = 1
MemoryLockToken lock = (MemoryLockToken)service.getWriteLock(test,
5000); //writeHoldCount = 2
lock.release(); //writeHoldCount = 1
lock = (MemoryLockToken)service.getWriteLock(test, 5000);
//writeHoldCount = 1, it should be 2.
{code}
{code}
@Override
public void release() {
int val = rwLock.getQueueLength();
if (val == 0) {
synchronized (locks) {
locks.remove(resource);
}
}
lock.unlock();
}
}
{code}
MemoryLocks should check hold count before removing lock.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
