Re: SRM Fulfillment creation issue

2018-03-27 Thread LJ LongWing 
Yes, that's the SQL server side log.  Any application servers that are
going to be processing them should have it turned on to have a
comprehensive look.  You should also include API logging to help identify
what api calls are causing the sql to fire, and if possible, it would be
best to turn on Filter as well to identify what filters are causing the sql
to fire.anything that the Remedy server sends to the db should be
logged in these logs.  The turning on of the query hint would prevent an
update to a form from stopping queries to the form from processing.

On Tue, Mar 27, 2018 at 10:20 AM, Dinesh Kumar  wrote:

> Hi Thad,
>
> We are using Oracle Database and the table row is getting locked by update
> query. Its not getting released after that.
>
> DB team is saying there is no commit from application.
>
> Is there any way we can find what are all the SQL's issued by ARS to DB.
>
> Thanks,
> Dinesh kumar.
>
>
> On Tue, Mar 27, 2018 at 4:28 PM, Thad Esser  wrote:
>
>> Dinesh,
>>
>> Look into the Select-Query-Hint setting, which might help.  Our issue
>> manifested as ridiculously slow reports, which our DBAs said were because
>> of row locking.  This setting fixed that issue for us.  I don't recall for
>> sure, but there may have a been database setting to go along with that.  We
>> are on ARS/SRM/ITSM 8.1, with a SQL Server database.
>>
>> -Thad
>>
>> BMC Docs:
>> ==
>> https://docs.bmc.com/docs/display/public/ars81/ar.cfg+or+ar.
>> conf+options+S-Z
>> Select-Query-Hint
>>
>> Text to use in a query hint in the WITH clause of a SELECT statement when
>> queries are supplied to Microsoft SQL Server databases. This option works
>> only for queries triggered by GLE,GLEWF, and GME API calls. If this option
>> is an empty string or is not present, no WITH clause is generated. To
>> determine the appropriateness of using this feature in your environment,
>> consult your SQL Server documentation. This option is commonly used with a
>> NOLOCK setting for allowing queries to execute without being blocked by
>> simultaneous updates, thereby improving performance. For example, to allow
>> SQL Server to read data being updated and avoid blocking, use this syntax:
>>
>> Select-Query-Hint: NOLOCK
>> ==
>>
>> On Mon, Mar 26, 2018 at 10:52 AM, Dinesh Kumar  wrote:
>>
>>> Hi All,
>>>
>>> We are facing below issue in our environment which causes a higher
>>> impact and Business critical.
>>>
>>>
>>>
>>> SRM Fulfillment requests are not getting generated.
>>>
>>>
>>>
>>> The requests are getting struck in the *CAI Events* form in the Running
>>> status for all the events (SRM_OUT_RESTART_SR_SUBMIT, TMS_OUT_GET_DATA).
>>>
>>> We could see that PDT’s are also not getting connected to the Service
>>> Requests. (adding screenshot).
>>>
>>>
>>>
>>>
>>>
>>> We are getting time out errors in the plugin and ARERR logs for the
>>> certain requests.
>>>
>>>
>>>
>>> *Error-1:*
>>>
>>>
>>>
>>> 2018-02-14 09:12:55,772 ERROR [pool-2-thread-4]
>>> com.bmc.itsm.cai.filterapi.cai.worker.BaseEventWorker
>>> (BaseEventWorker.java:166) - CAI plugin failed to update error code of
>>> event:TMHAA5V0GJ3KKAPEER8HBGHK64BVRP. Failed with following exception.
>>> ERROR (94): Timeout during database query -- consider using more specific
>>> search criteria to narrow the results, and retry the operation;
>>> ATVIEUUSMS006:2000 ONC/RPC call timed out
>>>
>>>
>>>
>>> *Error-2:*
>>>
>>>
>>>
>>> Wed Feb 14 09:24:49 2018 390627 : The SQL database operation failed.
>>> (ARERR 552)
>>>
>>> Wed Feb 14 09:24:49 2018 ORA-03135: connection lost contact
>>>
>>> Process ID: 315236
>>>
>>> Session ID: 411 Serial number: 60447
>>>
>>> Wed Feb 14 09:24:49 2018 lastsql UPDATE T2457 SET
>>> C112=';100018;',C5='Remedy Application Service',C6=1518531886 WHERE C1
>>> = 'PDT5704'
>>>
>>> Wed Feb 14 09:24:53 2018 390627 : SQL database is now available (ARNOTE
>>> 592)
>>>
>>> SRM:ProcessDefinitionTemplate_Base(T2457) is getting locked and we are
>>> not able to create any request fulfillment after that.  Then we need to
>>> manually release.
>>>
>>>
>>>
>>> Has one faced this kind of issues.
>>>
>>>
>>> Thanks,
>>>
>>> Dinesh kumar.
>>>
>>> --
>>> ARSList mailing list
>>> ARSList@arslist.org
>>> https://mailman.rrr.se/cgi/listinfo/arslist
>>>
>>>
>>
>> --
>> ARSList mailing list
>> ARSList@arslist.org
>> https://mailman.rrr.se/cgi/listinfo/arslist
>>
>>
>
> --
> ARSList mailing list
> ARSList@arslist.org
> https://mailman.rrr.se/cgi/listinfo/arslist
>
>
-- 
ARSList mailing list
ARSList@arslist.org
https://mailman.rrr.se/cgi/listinfo/arslist

Re: SRM Fulfillment creation issue

2018-03-27 Thread Dinesh Kumar 
Hi Thad,

We are using Oracle Database and the table row is getting locked by update
query. Its not getting released after that.

DB team is saying there is no commit from application.

Is there any way we can find what are all the SQL's issued by ARS to DB.

Thanks,
Dinesh kumar.


On Tue, Mar 27, 2018 at 4:28 PM, Thad Esser  wrote:

> Dinesh,
>
> Look into the Select-Query-Hint setting, which might help.  Our issue
> manifested as ridiculously slow reports, which our DBAs said were because
> of row locking.  This setting fixed that issue for us.  I don't recall for
> sure, but there may have a been database setting to go along with that.  We
> are on ARS/SRM/ITSM 8.1, with a SQL Server database.
>
> -Thad
>
> BMC Docs:
> ==
> https://docs.bmc.com/docs/display/public/ars81/ar.cfg+
> or+ar.conf+options+S-Z
> Select-Query-Hint
>
> Text to use in a query hint in the WITH clause of a SELECT statement when
> queries are supplied to Microsoft SQL Server databases. This option works
> only for queries triggered by GLE,GLEWF, and GME API calls. If this option
> is an empty string or is not present, no WITH clause is generated. To
> determine the appropriateness of using this feature in your environment,
> consult your SQL Server documentation. This option is commonly used with a
> NOLOCK setting for allowing queries to execute without being blocked by
> simultaneous updates, thereby improving performance. For example, to allow
> SQL Server to read data being updated and avoid blocking, use this syntax:
>
> Select-Query-Hint: NOLOCK
> ==
>
> On Mon, Mar 26, 2018 at 10:52 AM, Dinesh Kumar  wrote:
>
>> Hi All,
>>
>> We are facing below issue in our environment which causes a higher impact
>> and Business critical.
>>
>>
>>
>> SRM Fulfillment requests are not getting generated.
>>
>>
>>
>> The requests are getting struck in the *CAI Events* form in the Running
>> status for all the events (SRM_OUT_RESTART_SR_SUBMIT, TMS_OUT_GET_DATA).
>>
>> We could see that PDT’s are also not getting connected to the Service
>> Requests. (adding screenshot).
>>
>>
>>
>>
>>
>> We are getting time out errors in the plugin and ARERR logs for the
>> certain requests.
>>
>>
>>
>> *Error-1:*
>>
>>
>>
>> 2018-02-14 09:12:55,772 ERROR [pool-2-thread-4]
>> com.bmc.itsm.cai.filterapi.cai.worker.BaseEventWorker
>> (BaseEventWorker.java:166) - CAI plugin failed to update error code of
>> event:TMHAA5V0GJ3KKAPEER8HBGHK64BVRP. Failed with following exception.
>> ERROR (94): Timeout during database query -- consider using more specific
>> search criteria to narrow the results, and retry the operation;
>> ATVIEUUSMS006:2000 ONC/RPC call timed out
>>
>>
>>
>> *Error-2:*
>>
>>
>>
>> Wed Feb 14 09:24:49 2018 390627 : The SQL database operation failed.
>> (ARERR 552)
>>
>> Wed Feb 14 09:24:49 2018 ORA-03135: connection lost contact
>>
>> Process ID: 315236
>>
>> Session ID: 411 Serial number: 60447
>>
>> Wed Feb 14 09:24:49 2018 lastsql UPDATE T2457 SET
>> C112=';100018;',C5='Remedy Application Service',C6=1518531886 WHERE C1
>> = 'PDT5704'
>>
>> Wed Feb 14 09:24:53 2018 390627 : SQL database is now available (ARNOTE
>> 592)
>>
>> SRM:ProcessDefinitionTemplate_Base(T2457) is getting locked and we are
>> not able to create any request fulfillment after that.  Then we need to
>> manually release.
>>
>>
>>
>> Has one faced this kind of issues.
>>
>>
>> Thanks,
>>
>> Dinesh kumar.
>>
>> --
>> ARSList mailing list
>> ARSList@arslist.org
>> https://mailman.rrr.se/cgi/listinfo/arslist
>>
>>
>
> --
> ARSList mailing list
> ARSList@arslist.org
> https://mailman.rrr.se/cgi/listinfo/arslist
>
>
-- 
ARSList mailing list
ARSList@arslist.org
https://mailman.rrr.se/cgi/listinfo/arslist

Re: SRM Fulfillment creation issue

2018-03-27 Thread Thad Esser 
Dinesh,

Look into the Select-Query-Hint setting, which might help.  Our issue
manifested as ridiculously slow reports, which our DBAs said were because
of row locking.  This setting fixed that issue for us.  I don't recall for
sure, but there may have a been database setting to go along with that.  We
are on ARS/SRM/ITSM 8.1, with a SQL Server database.

-Thad

BMC Docs:
==
https://docs.bmc.com/docs/display/public/ars81/ar.cfg+or+ar.conf+options+S-Z
Select-Query-Hint

Text to use in a query hint in the WITH clause of a SELECT statement when
queries are supplied to Microsoft SQL Server databases. This option works
only for queries triggered by GLE,GLEWF, and GME API calls. If this option
is an empty string or is not present, no WITH clause is generated. To
determine the appropriateness of using this feature in your environment,
consult your SQL Server documentation. This option is commonly used with a
NOLOCK setting for allowing queries to execute without being blocked by
simultaneous updates, thereby improving performance. For example, to allow
SQL Server to read data being updated and avoid blocking, use this syntax:

Select-Query-Hint: NOLOCK
==

On Mon, Mar 26, 2018 at 10:52 AM, Dinesh Kumar  wrote:

> Hi All,
>
> We are facing below issue in our environment which causes a higher impact
> and Business critical.
>
>
>
> SRM Fulfillment requests are not getting generated.
>
>
>
> The requests are getting struck in the *CAI Events* form in the Running
> status for all the events (SRM_OUT_RESTART_SR_SUBMIT, TMS_OUT_GET_DATA).
>
> We could see that PDT’s are also not getting connected to the Service
> Requests. (adding screenshot).
>
>
>
>
>
> We are getting time out errors in the plugin and ARERR logs for the
> certain requests.
>
>
>
> *Error-1:*
>
>
>
> 2018-02-14 09:12:55,772 ERROR [pool-2-thread-4] 
> com.bmc.itsm.cai.filterapi.cai.worker.BaseEventWorker
> (BaseEventWorker.java:166) - CAI plugin failed to update error code of
> event:TMHAA5V0GJ3KKAPEER8HBGHK64BVRP. Failed with following exception.
> ERROR (94): Timeout during database query -- consider using more specific
> search criteria to narrow the results, and retry the operation;
> ATVIEUUSMS006:2000 ONC/RPC call timed out
>
>
>
> *Error-2:*
>
>
>
> Wed Feb 14 09:24:49 2018 390627 : The SQL database operation failed.
> (ARERR 552)
>
> Wed Feb 14 09:24:49 2018 ORA-03135: connection lost contact
>
> Process ID: 315236
>
> Session ID: 411 Serial number: 60447
>
> Wed Feb 14 09:24:49 2018 lastsql UPDATE T2457 SET
> C112=';100018;',C5='Remedy Application Service',C6=1518531886 WHERE C1
> = 'PDT5704'
>
> Wed Feb 14 09:24:53 2018 390627 : SQL database is now available (ARNOTE
> 592)
>
> SRM:ProcessDefinitionTemplate_Base(T2457) is getting locked and we are
> not able to create any request fulfillment after that.  Then we need to
> manually release.
>
>
>
> Has one faced this kind of issues.
>
>
> Thanks,
>
> Dinesh kumar.
>
> --
> ARSList mailing list
> ARSList@arslist.org
> https://mailman.rrr.se/cgi/listinfo/arslist
>
>
-- 
ARSList mailing list
ARSList@arslist.org
https://mailman.rrr.se/cgi/listinfo/arslist

Re: SRM Fulfillment creation issue

2018-03-26 Thread LJ LongWing 
Dinesh,
I've never been too much on the application side of thingsa quick
filter log should show you what workflow is pushing to that form which
might give you a better idea of 'why' :)

On Mon, Mar 26, 2018 at 1:56 PM, Dinesh Kumar  wrote:

> We are in ARS 8.1 SP1 and SRM 8.1.
>
> Why we are updating the SRM:ProcessDefinitionTemplate_Base form for every
> service request creation ?
>
> Thanks,
> Dinesh kumar.
>
> On Mon, Mar 26, 2018 at 9:48 PM, LJ LongWing 
> wrote:
>
>> You might try looking for an arexception.log fileif you are of a
>> sufficient version and have it enabled, you might find information on how
>> some of the calls are being generated and maybe even who.
>>
>> On Mon, Mar 26, 2018 at 1:45 PM, Dinesh Kumar  wrote:
>>
>>> Hi LJ,
>>>
>>> We are not able to find that update statement in the logs, its getting
>>> captured in the AR Error logs only. Not sure from where its getting
>>> triggered .
>>>
>>> Thanks,
>>> Dinesh kumar.
>>>
>>> On Mon, Mar 26, 2018 at 9:25 PM, LJ LongWing 
>>> wrote:
>>>
 Dinesh,
 Remedy does a 'BEGIN' at the beginning of an API call and does a
 'COMMIT' at the end of an API callyou'll need to run some API/SQL logs
 on your server to help identify what processes are causing the blocking to
 occur at the DB level...

 Now, that could easily describe why #1 is occurringbut for #2, the
 db having blocking would not and should not cause the Remedy server to
 disconnect.  That is being caused at the db level, possibly by killing a
 job causing the blocking, but it's not being caused on the Remedy side...

 On Mon, Mar 26, 2018 at 12:48 PM, Dinesh Kumar 
 wrote:

> Hi LJ,
>
> DB team is saying this is boz of "TX - row lock contention".
> "enq: TX - row lock contention" waits are generally related to the
> application code being executed and do not indicate a problem with the DB
> itself.
>
> The reason to this could be as the commit is still not issued , thus
> the lock is still not released. Causing other queries to wait for the 
> lock.
>
>
> When they check the segments where most amount of waits are spent we
> could see that table T2457 takes about 95% of the time
>
> Thus all update queries running on table T2457 need to be checked by
> the application team
>
> TX lock is an application coding, design and usage problem and can
> ONLY be fixed by changing application code with more frequent and explicit
> COMMIT statements and any other minor code changes. DBA team cannot
> fix TX lock wait issues other than helping to identify the objects and
> commands causing the waits.
>
>
> Thanks,
>
> Dinesh kumar.
>
>
>
> On Mon, Mar 26, 2018 at 8:03 PM, LJ LongWing 
> wrote:
>
>> Well, for #1, you are issuing a query to the db that's not returning
>> in the 120 second timeoutfor #2 you are getting disconnected from the
>> DBboth scenarios are pointing to a DB that is either overwhelmed, or
>> under performing...I would highly recommend working with your DBA to 
>> figure
>> out what's happening at the DB level first.
>>
>> On Mon, Mar 26, 2018 at 11:52 AM, Dinesh Kumar 
>> wrote:
>>
>>> Hi All,
>>>
>>> We are facing below issue in our environment which causes a higher
>>> impact and Business critical.
>>>
>>>
>>>
>>> SRM Fulfillment requests are not getting generated.
>>>
>>>
>>>
>>> The requests are getting struck in the *CAI Events* form in the
>>> Running status for all the events (SRM_OUT_RESTART_SR_SUBMIT,
>>> TMS_OUT_GET_DATA).
>>>
>>> We could see that PDT’s are also not getting connected to the
>>> Service Requests. (adding screenshot).
>>>
>>>
>>>
>>>
>>>
>>> We are getting time out errors in the plugin and ARERR logs for the
>>> certain requests.
>>>
>>>
>>>
>>> *Error-1:*
>>>
>>>
>>>
>>> 2018-02-14 09:12:55,772 ERROR [pool-2-thread-4]
>>> com.bmc.itsm.cai.filterapi.cai.worker.BaseEventWorker
>>> (BaseEventWorker.java:166) - CAI plugin failed to update error code of
>>> event:TMHAA5V0GJ3KKAPEER8HBGHK64BVRP. Failed with following
>>> exception. ERROR (94): Timeout during database query -- consider using 
>>> more
>>> specific search criteria to narrow the results, and retry the operation;
>>> ATVIEUUSMS006:2000 ONC/RPC call timed out
>>>
>>>
>>>
>>> *Error-2:*
>>>
>>>
>>>
>>> Wed Feb 14 09:24:49 2018 390627 : The SQL database operation failed.
>>> (ARERR 552)
>>>
>>> Wed Feb 14 09:24:49 2018 ORA-03135: connection lost contact
>>>
>>> Process ID: 315236
>>>
>>>

Re: SRM Fulfillment creation issue

2018-03-26 Thread Dinesh Kumar 
We are in ARS 8.1 SP1 and SRM 8.1.

Why we are updating the SRM:ProcessDefinitionTemplate_Base form for every
service request creation ?

Thanks,
Dinesh kumar.

On Mon, Mar 26, 2018 at 9:48 PM, LJ LongWing  wrote:

> You might try looking for an arexception.log fileif you are of a
> sufficient version and have it enabled, you might find information on how
> some of the calls are being generated and maybe even who.
>
> On Mon, Mar 26, 2018 at 1:45 PM, Dinesh Kumar  wrote:
>
>> Hi LJ,
>>
>> We are not able to find that update statement in the logs, its getting
>> captured in the AR Error logs only. Not sure from where its getting
>> triggered .
>>
>> Thanks,
>> Dinesh kumar.
>>
>> On Mon, Mar 26, 2018 at 9:25 PM, LJ LongWing 
>> wrote:
>>
>>> Dinesh,
>>> Remedy does a 'BEGIN' at the beginning of an API call and does a
>>> 'COMMIT' at the end of an API callyou'll need to run some API/SQL logs
>>> on your server to help identify what processes are causing the blocking to
>>> occur at the DB level...
>>>
>>> Now, that could easily describe why #1 is occurringbut for #2, the
>>> db having blocking would not and should not cause the Remedy server to
>>> disconnect.  That is being caused at the db level, possibly by killing a
>>> job causing the blocking, but it's not being caused on the Remedy side...
>>>
>>> On Mon, Mar 26, 2018 at 12:48 PM, Dinesh Kumar 
>>> wrote:
>>>
 Hi LJ,

 DB team is saying this is boz of "TX - row lock contention".
 "enq: TX - row lock contention" waits are generally related to the
 application code being executed and do not indicate a problem with the DB
 itself.

 The reason to this could be as the commit is still not issued , thus
 the lock is still not released. Causing other queries to wait for the lock.


 When they check the segments where most amount of waits are spent we
 could see that table T2457 takes about 95% of the time

 Thus all update queries running on table T2457 need to be checked by
 the application team

 TX lock is an application coding, design and usage problem and can ONLY
 be fixed by changing application code with more frequent and explicit
 COMMIT statements and any other minor code changes. DBA team cannot
 fix TX lock wait issues other than helping to identify the objects and
 commands causing the waits.


 Thanks,

 Dinesh kumar.



 On Mon, Mar 26, 2018 at 8:03 PM, LJ LongWing 
 wrote:

> Well, for #1, you are issuing a query to the db that's not returning
> in the 120 second timeoutfor #2 you are getting disconnected from the
> DBboth scenarios are pointing to a DB that is either overwhelmed, or
> under performing...I would highly recommend working with your DBA to 
> figure
> out what's happening at the DB level first.
>
> On Mon, Mar 26, 2018 at 11:52 AM, Dinesh Kumar 
> wrote:
>
>> Hi All,
>>
>> We are facing below issue in our environment which causes a higher
>> impact and Business critical.
>>
>>
>>
>> SRM Fulfillment requests are not getting generated.
>>
>>
>>
>> The requests are getting struck in the *CAI Events* form in the
>> Running status for all the events (SRM_OUT_RESTART_SR_SUBMIT,
>> TMS_OUT_GET_DATA).
>>
>> We could see that PDT’s are also not getting connected to the Service
>> Requests. (adding screenshot).
>>
>>
>>
>>
>>
>> We are getting time out errors in the plugin and ARERR logs for the
>> certain requests.
>>
>>
>>
>> *Error-1:*
>>
>>
>>
>> 2018-02-14 09:12:55,772 ERROR [pool-2-thread-4]
>> com.bmc.itsm.cai.filterapi.cai.worker.BaseEventWorker
>> (BaseEventWorker.java:166) - CAI plugin failed to update error code of
>> event:TMHAA5V0GJ3KKAPEER8HBGHK64BVRP. Failed with following
>> exception. ERROR (94): Timeout during database query -- consider using 
>> more
>> specific search criteria to narrow the results, and retry the operation;
>> ATVIEUUSMS006:2000 ONC/RPC call timed out
>>
>>
>>
>> *Error-2:*
>>
>>
>>
>> Wed Feb 14 09:24:49 2018 390627 : The SQL database operation failed.
>> (ARERR 552)
>>
>> Wed Feb 14 09:24:49 2018 ORA-03135: connection lost contact
>>
>> Process ID: 315236
>>
>> Session ID: 411 Serial number: 60447
>>
>> Wed Feb 14 09:24:49 2018 lastsql UPDATE T2457 SET
>> C112=';100018;',C5='Remedy Application Service',C6=1518531886 WHERE 
>> C1
>> = 'PDT5704'
>>
>> Wed Feb 14 09:24:53 2018 390627 : SQL database is now available
>> (ARNOTE 592)
>>
>> SRM:ProcessDefinitionTemplate_Base(T2457) is getting locked and we
>> are not able to

Re: SRM Fulfillment creation issue

2018-03-26 Thread LJ LongWing 
You might try looking for an arexception.log fileif you are of a
sufficient version and have it enabled, you might find information on how
some of the calls are being generated and maybe even who.

On Mon, Mar 26, 2018 at 1:45 PM, Dinesh Kumar  wrote:

> Hi LJ,
>
> We are not able to find that update statement in the logs, its getting
> captured in the AR Error logs only. Not sure from where its getting
> triggered .
>
> Thanks,
> Dinesh kumar.
>
> On Mon, Mar 26, 2018 at 9:25 PM, LJ LongWing 
> wrote:
>
>> Dinesh,
>> Remedy does a 'BEGIN' at the beginning of an API call and does a 'COMMIT'
>> at the end of an API callyou'll need to run some API/SQL logs on your
>> server to help identify what processes are causing the blocking to occur at
>> the DB level...
>>
>> Now, that could easily describe why #1 is occurringbut for #2, the db
>> having blocking would not and should not cause the Remedy server to
>> disconnect.  That is being caused at the db level, possibly by killing a
>> job causing the blocking, but it's not being caused on the Remedy side...
>>
>> On Mon, Mar 26, 2018 at 12:48 PM, Dinesh Kumar  wrote:
>>
>>> Hi LJ,
>>>
>>> DB team is saying this is boz of "TX - row lock contention".
>>> "enq: TX - row lock contention" waits are generally related to the
>>> application code being executed and do not indicate a problem with the DB
>>> itself.
>>>
>>> The reason to this could be as the commit is still not issued , thus the
>>> lock is still not released. Causing other queries to wait for the lock.
>>>
>>>
>>> When they check the segments where most amount of waits are spent we
>>> could see that table T2457 takes about 95% of the time
>>>
>>> Thus all update queries running on table T2457 need to be checked by the
>>> application team
>>>
>>> TX lock is an application coding, design and usage problem and can ONLY
>>> be fixed by changing application code with more frequent and explicit
>>> COMMIT statements and any other minor code changes. DBA team cannot fix
>>> TX lock wait issues other than helping to identify the objects and commands
>>> causing the waits.
>>>
>>>
>>> Thanks,
>>>
>>> Dinesh kumar.
>>>
>>>
>>>
>>> On Mon, Mar 26, 2018 at 8:03 PM, LJ LongWing 
>>> wrote:
>>>
 Well, for #1, you are issuing a query to the db that's not returning in
 the 120 second timeoutfor #2 you are getting disconnected from the
 DBboth scenarios are pointing to a DB that is either overwhelmed, or
 under performing...I would highly recommend working with your DBA to figure
 out what's happening at the DB level first.

 On Mon, Mar 26, 2018 at 11:52 AM, Dinesh Kumar 
 wrote:

> Hi All,
>
> We are facing below issue in our environment which causes a higher
> impact and Business critical.
>
>
>
> SRM Fulfillment requests are not getting generated.
>
>
>
> The requests are getting struck in the *CAI Events* form in the
> Running status for all the events (SRM_OUT_RESTART_SR_SUBMIT,
> TMS_OUT_GET_DATA).
>
> We could see that PDT’s are also not getting connected to the Service
> Requests. (adding screenshot).
>
>
>
>
>
> We are getting time out errors in the plugin and ARERR logs for the
> certain requests.
>
>
>
> *Error-1:*
>
>
>
> 2018-02-14 09:12:55,772 ERROR [pool-2-thread-4]
> com.bmc.itsm.cai.filterapi.cai.worker.BaseEventWorker
> (BaseEventWorker.java:166) - CAI plugin failed to update error code of
> event:TMHAA5V0GJ3KKAPEER8HBGHK64BVRP. Failed with following
> exception. ERROR (94): Timeout during database query -- consider using 
> more
> specific search criteria to narrow the results, and retry the operation;
> ATVIEUUSMS006:2000 ONC/RPC call timed out
>
>
>
> *Error-2:*
>
>
>
> Wed Feb 14 09:24:49 2018 390627 : The SQL database operation failed.
> (ARERR 552)
>
> Wed Feb 14 09:24:49 2018 ORA-03135: connection lost contact
>
> Process ID: 315236
>
> Session ID: 411 Serial number: 60447
>
> Wed Feb 14 09:24:49 2018 lastsql UPDATE T2457 SET
> C112=';100018;',C5='Remedy Application Service',C6=1518531886 WHERE C1
> = 'PDT5704'
>
> Wed Feb 14 09:24:53 2018 390627 : SQL database is now available
> (ARNOTE 592)
>
> SRM:ProcessDefinitionTemplate_Base(T2457) is getting locked and we
> are not able to create any request fulfillment after that.  Then we
> need to manually release.
>
>
>
> Has one faced this kind of issues.
>
>
> Thanks,
>
> Dinesh kumar.
>
> --
> ARSList mailing list
> ARSList@arslist.org
> https://mailman.rrr.se/cgi/listinfo/arslist
>
>

 --
 ARSList mailing list
 ARSList@arslist.org

Re: SRM Fulfillment creation issue

2018-03-26 Thread Dinesh Kumar 
Hi LJ,

We are not able to find that update statement in the logs, its getting
captured in the AR Error logs only. Not sure from where its getting
triggered .

Thanks,
Dinesh kumar.

On Mon, Mar 26, 2018 at 9:25 PM, LJ LongWing  wrote:

> Dinesh,
> Remedy does a 'BEGIN' at the beginning of an API call and does a 'COMMIT'
> at the end of an API callyou'll need to run some API/SQL logs on your
> server to help identify what processes are causing the blocking to occur at
> the DB level...
>
> Now, that could easily describe why #1 is occurringbut for #2, the db
> having blocking would not and should not cause the Remedy server to
> disconnect.  That is being caused at the db level, possibly by killing a
> job causing the blocking, but it's not being caused on the Remedy side...
>
> On Mon, Mar 26, 2018 at 12:48 PM, Dinesh Kumar  wrote:
>
>> Hi LJ,
>>
>> DB team is saying this is boz of "TX - row lock contention".
>> "enq: TX - row lock contention" waits are generally related to the
>> application code being executed and do not indicate a problem with the DB
>> itself.
>>
>> The reason to this could be as the commit is still not issued , thus the
>> lock is still not released. Causing other queries to wait for the lock.
>>
>>
>> When they check the segments where most amount of waits are spent we
>> could see that table T2457 takes about 95% of the time
>>
>> Thus all update queries running on table T2457 need to be checked by the
>> application team
>>
>> TX lock is an application coding, design and usage problem and can ONLY
>> be fixed by changing application code with more frequent and explicit
>> COMMIT statements and any other minor code changes. DBA team cannot fix
>> TX lock wait issues other than helping to identify the objects and commands
>> causing the waits.
>>
>>
>> Thanks,
>>
>> Dinesh kumar.
>>
>>
>>
>> On Mon, Mar 26, 2018 at 8:03 PM, LJ LongWing 
>> wrote:
>>
>>> Well, for #1, you are issuing a query to the db that's not returning in
>>> the 120 second timeoutfor #2 you are getting disconnected from the
>>> DBboth scenarios are pointing to a DB that is either overwhelmed, or
>>> under performing...I would highly recommend working with your DBA to figure
>>> out what's happening at the DB level first.
>>>
>>> On Mon, Mar 26, 2018 at 11:52 AM, Dinesh Kumar 
>>> wrote:
>>>
 Hi All,

 We are facing below issue in our environment which causes a higher
 impact and Business critical.



 SRM Fulfillment requests are not getting generated.



 The requests are getting struck in the *CAI Events* form in the
 Running status for all the events (SRM_OUT_RESTART_SR_SUBMIT,
 TMS_OUT_GET_DATA).

 We could see that PDT’s are also not getting connected to the Service
 Requests. (adding screenshot).





 We are getting time out errors in the plugin and ARERR logs for the
 certain requests.



 *Error-1:*



 2018-02-14 09:12:55,772 ERROR [pool-2-thread-4]
 com.bmc.itsm.cai.filterapi.cai.worker.BaseEventWorker
 (BaseEventWorker.java:166) - CAI plugin failed to update error code of
 event:TMHAA5V0GJ3KKAPEER8HBGHK64BVRP. Failed with following exception.
 ERROR (94): Timeout during database query -- consider using more specific
 search criteria to narrow the results, and retry the operation;
 ATVIEUUSMS006:2000 ONC/RPC call timed out



 *Error-2:*



 Wed Feb 14 09:24:49 2018 390627 : The SQL database operation failed.
 (ARERR 552)

 Wed Feb 14 09:24:49 2018 ORA-03135: connection lost contact

 Process ID: 315236

 Session ID: 411 Serial number: 60447

 Wed Feb 14 09:24:49 2018 lastsql UPDATE T2457 SET
 C112=';100018;',C5='Remedy Application Service',C6=1518531886 WHERE C1
 = 'PDT5704'

 Wed Feb 14 09:24:53 2018 390627 : SQL database is now available (ARNOTE
 592)

 SRM:ProcessDefinitionTemplate_Base(T2457) is getting locked and we are
 not able to create any request fulfillment after that.  Then we need
 to manually release.



 Has one faced this kind of issues.


 Thanks,

 Dinesh kumar.

 --
 ARSList mailing list
 ARSList@arslist.org
 https://mailman.rrr.se/cgi/listinfo/arslist


>>>
>>> --
>>> ARSList mailing list
>>> ARSList@arslist.org
>>> https://mailman.rrr.se/cgi/listinfo/arslist
>>>
>>>
>>
>> --
>> ARSList mailing list
>> ARSList@arslist.org
>> https://mailman.rrr.se/cgi/listinfo/arslist
>>
>>
>
> --
> ARSList mailing list
> ARSList@arslist.org
> https://mailman.rrr.se/cgi/listinfo/arslist
>
>
-- 
ARSList mailing list
ARSList@arslist.org
https://mailman.rrr.se/cgi/listinfo/arslist

Re: SRM Fulfillment creation issue

2018-03-26 Thread LJ LongWing 
Dinesh,
Remedy does a 'BEGIN' at the beginning of an API call and does a 'COMMIT'
at the end of an API callyou'll need to run some API/SQL logs on your
server to help identify what processes are causing the blocking to occur at
the DB level...

Now, that could easily describe why #1 is occurringbut for #2, the db
having blocking would not and should not cause the Remedy server to
disconnect.  That is being caused at the db level, possibly by killing a
job causing the blocking, but it's not being caused on the Remedy side...

On Mon, Mar 26, 2018 at 12:48 PM, Dinesh Kumar  wrote:

> Hi LJ,
>
> DB team is saying this is boz of "TX - row lock contention".
> "enq: TX - row lock contention" waits are generally related to the
> application code being executed and do not indicate a problem with the DB
> itself.
>
> The reason to this could be as the commit is still not issued , thus the
> lock is still not released. Causing other queries to wait for the lock.
>
>
> When they check the segments where most amount of waits are spent we could
> see that table T2457 takes about 95% of the time
>
> Thus all update queries running on table T2457 need to be checked by the
> application team
>
> TX lock is an application coding, design and usage problem and can ONLY be
> fixed by changing application code with more frequent and explicit COMMIT
> statements and any other minor code changes. DBA team cannot fix TX lock
> wait issues other than helping to identify the objects and commands causing
> the waits.
>
>
> Thanks,
>
> Dinesh kumar.
>
>
>
> On Mon, Mar 26, 2018 at 8:03 PM, LJ LongWing 
> wrote:
>
>> Well, for #1, you are issuing a query to the db that's not returning in
>> the 120 second timeoutfor #2 you are getting disconnected from the
>> DBboth scenarios are pointing to a DB that is either overwhelmed, or
>> under performing...I would highly recommend working with your DBA to figure
>> out what's happening at the DB level first.
>>
>> On Mon, Mar 26, 2018 at 11:52 AM, Dinesh Kumar  wrote:
>>
>>> Hi All,
>>>
>>> We are facing below issue in our environment which causes a higher
>>> impact and Business critical.
>>>
>>>
>>>
>>> SRM Fulfillment requests are not getting generated.
>>>
>>>
>>>
>>> The requests are getting struck in the *CAI Events* form in the Running
>>> status for all the events (SRM_OUT_RESTART_SR_SUBMIT, TMS_OUT_GET_DATA).
>>>
>>> We could see that PDT’s are also not getting connected to the Service
>>> Requests. (adding screenshot).
>>>
>>>
>>>
>>>
>>>
>>> We are getting time out errors in the plugin and ARERR logs for the
>>> certain requests.
>>>
>>>
>>>
>>> *Error-1:*
>>>
>>>
>>>
>>> 2018-02-14 09:12:55,772 ERROR [pool-2-thread-4]
>>> com.bmc.itsm.cai.filterapi.cai.worker.BaseEventWorker
>>> (BaseEventWorker.java:166) - CAI plugin failed to update error code of
>>> event:TMHAA5V0GJ3KKAPEER8HBGHK64BVRP. Failed with following exception.
>>> ERROR (94): Timeout during database query -- consider using more specific
>>> search criteria to narrow the results, and retry the operation;
>>> ATVIEUUSMS006:2000 ONC/RPC call timed out
>>>
>>>
>>>
>>> *Error-2:*
>>>
>>>
>>>
>>> Wed Feb 14 09:24:49 2018 390627 : The SQL database operation failed.
>>> (ARERR 552)
>>>
>>> Wed Feb 14 09:24:49 2018 ORA-03135: connection lost contact
>>>
>>> Process ID: 315236
>>>
>>> Session ID: 411 Serial number: 60447
>>>
>>> Wed Feb 14 09:24:49 2018 lastsql UPDATE T2457 SET
>>> C112=';100018;',C5='Remedy Application Service',C6=1518531886 WHERE C1
>>> = 'PDT5704'
>>>
>>> Wed Feb 14 09:24:53 2018 390627 : SQL database is now available (ARNOTE
>>> 592)
>>>
>>> SRM:ProcessDefinitionTemplate_Base(T2457) is getting locked and we are
>>> not able to create any request fulfillment after that.  Then we need to
>>> manually release.
>>>
>>>
>>>
>>> Has one faced this kind of issues.
>>>
>>>
>>> Thanks,
>>>
>>> Dinesh kumar.
>>>
>>> --
>>> ARSList mailing list
>>> ARSList@arslist.org
>>> https://mailman.rrr.se/cgi/listinfo/arslist
>>>
>>>
>>
>> --
>> ARSList mailing list
>> ARSList@arslist.org
>> https://mailman.rrr.se/cgi/listinfo/arslist
>>
>>
>
> --
> ARSList mailing list
> ARSList@arslist.org
> https://mailman.rrr.se/cgi/listinfo/arslist
>
>
-- 
ARSList mailing list
ARSList@arslist.org
https://mailman.rrr.se/cgi/listinfo/arslist

Re: SRM Fulfillment creation issue

2018-03-26 Thread Dinesh Kumar 
Hi LJ,

DB team is saying this is boz of "TX - row lock contention".
"enq: TX - row lock contention" waits are generally related to the
application code being executed and do not indicate a problem with the DB
itself.

The reason to this could be as the commit is still not issued , thus the
lock is still not released. Causing other queries to wait for the lock.


When they check the segments where most amount of waits are spent we could
see that table T2457 takes about 95% of the time

Thus all update queries running on table T2457 need to be checked by the
application team

TX lock is an application coding, design and usage problem and can ONLY be
fixed by changing application code with more frequent and explicit COMMIT
statements and any other minor code changes. DBA team cannot fix TX lock
wait issues other than helping to identify the objects and commands causing
the waits.


Thanks,

Dinesh kumar.



On Mon, Mar 26, 2018 at 8:03 PM, LJ LongWing  wrote:

> Well, for #1, you are issuing a query to the db that's not returning in
> the 120 second timeoutfor #2 you are getting disconnected from the
> DBboth scenarios are pointing to a DB that is either overwhelmed, or
> under performing...I would highly recommend working with your DBA to figure
> out what's happening at the DB level first.
>
> On Mon, Mar 26, 2018 at 11:52 AM, Dinesh Kumar  wrote:
>
>> Hi All,
>>
>> We are facing below issue in our environment which causes a higher impact
>> and Business critical.
>>
>>
>>
>> SRM Fulfillment requests are not getting generated.
>>
>>
>>
>> The requests are getting struck in the *CAI Events* form in the Running
>> status for all the events (SRM_OUT_RESTART_SR_SUBMIT, TMS_OUT_GET_DATA).
>>
>> We could see that PDT’s are also not getting connected to the Service
>> Requests. (adding screenshot).
>>
>>
>>
>>
>>
>> We are getting time out errors in the plugin and ARERR logs for the
>> certain requests.
>>
>>
>>
>> *Error-1:*
>>
>>
>>
>> 2018-02-14 09:12:55,772 ERROR [pool-2-thread-4]
>> com.bmc.itsm.cai.filterapi.cai.worker.BaseEventWorker
>> (BaseEventWorker.java:166) - CAI plugin failed to update error code of
>> event:TMHAA5V0GJ3KKAPEER8HBGHK64BVRP. Failed with following exception.
>> ERROR (94): Timeout during database query -- consider using more specific
>> search criteria to narrow the results, and retry the operation;
>> ATVIEUUSMS006:2000 ONC/RPC call timed out
>>
>>
>>
>> *Error-2:*
>>
>>
>>
>> Wed Feb 14 09:24:49 2018 390627 : The SQL database operation failed.
>> (ARERR 552)
>>
>> Wed Feb 14 09:24:49 2018 ORA-03135: connection lost contact
>>
>> Process ID: 315236
>>
>> Session ID: 411 Serial number: 60447
>>
>> Wed Feb 14 09:24:49 2018 lastsql UPDATE T2457 SET
>> C112=';100018;',C5='Remedy Application Service',C6=1518531886 WHERE C1
>> = 'PDT5704'
>>
>> Wed Feb 14 09:24:53 2018 390627 : SQL database is now available (ARNOTE
>> 592)
>>
>> SRM:ProcessDefinitionTemplate_Base(T2457) is getting locked and we are
>> not able to create any request fulfillment after that.  Then we need to
>> manually release.
>>
>>
>>
>> Has one faced this kind of issues.
>>
>>
>> Thanks,
>>
>> Dinesh kumar.
>>
>> --
>> ARSList mailing list
>> ARSList@arslist.org
>> https://mailman.rrr.se/cgi/listinfo/arslist
>>
>>
>
> --
> ARSList mailing list
> ARSList@arslist.org
> https://mailman.rrr.se/cgi/listinfo/arslist
>
>
-- 
ARSList mailing list
ARSList@arslist.org
https://mailman.rrr.se/cgi/listinfo/arslist

Re: SRM Fulfillment creation issue

2018-03-26 Thread Tauf Chowdhury 
Totally doing this from memory but I’m pretty sure the reason is that the query 
is looking at the product catalog form and timing out. You can try to be more 
specific in your query or do some tuning on those tables. 


Sent from my iPhone

> On Mar 26, 2018, at 2:03 PM, LJ LongWing  wrote:
> 
> Well, for #1, you are issuing a query to the db that's not returning in the 
> 120 second timeoutfor #2 you are getting disconnected from the DBboth 
> scenarios are pointing to a DB that is either overwhelmed, or under 
> performing...I would highly recommend working with your DBA to figure out 
> what's happening at the DB level first.
> 
>> On Mon, Mar 26, 2018 at 11:52 AM, Dinesh Kumar  wrote:
>> Hi All,
>> 
>> We are facing below issue in our environment which causes a higher impact 
>> and Business critical.
>>  
>> SRM Fulfillment requests are not getting generated.
>>  
>> The requests are getting struck in the CAI Events form in the Running status 
>> for all the events (SRM_OUT_RESTART_SR_SUBMIT, TMS_OUT_GET_DATA).
>> We could see that PDT’s are also not getting connected to the Service 
>> Requests. (adding screenshot).
>>  
>> 
>> 
>> We are getting time out errors in the plugin and ARERR logs for the certain 
>> requests.
>>  
>> Error-1:
>>  
>> 2018-02-14 09:12:55,772 ERROR [pool-2-thread-4] 
>> com.bmc.itsm.cai.filterapi.cai.worker.BaseEventWorker 
>> (BaseEventWorker.java:166) - CAI plugin failed to update error code of 
>> event:TMHAA5V0GJ3KKAPEER8HBGHK64BVRP. Failed with following exception. ERROR 
>> (94): Timeout during database query -- consider using more specific search 
>> criteria to narrow the results, and retry the operation; ATVIEUUSMS006:2000 
>> ONC/RPC call timed out
>>  
>> Error-2:
>>  
>> Wed Feb 14 09:24:49 2018 390627 : The SQL database operation failed. (ARERR 
>> 552)
>> Wed Feb 14 09:24:49 2018 ORA-03135: connection lost contact
>> Process ID: 315236
>> Session ID: 411 Serial number: 60447
>> Wed Feb 14 09:24:49 2018 lastsql UPDATE T2457 SET 
>> C112=';100018;',C5='Remedy Application Service',C6=1518531886 WHERE C1 = 
>> 'PDT5704'
>> Wed Feb 14 09:24:53 2018 390627 : SQL database is now available (ARNOTE 592)
>> 
>> SRM:ProcessDefinitionTemplate_Base(T2457) is getting locked and we are not 
>> able to create any request fulfillment after that.  Then we need to manually 
>> release.
>>  
>> Has one faced this kind of issues.
>> 
>> Thanks,
>> Dinesh kumar.
>> 
>> --
>> ARSList mailing list
>> ARSList@arslist.org
>> https://mailman.rrr.se/cgi/listinfo/arslist
>> 
> 
> -- 
> ARSList mailing list
> ARSList@arslist.org
> https://mailman.rrr.se/cgi/listinfo/arslist
-- 
ARSList mailing list
ARSList@arslist.org
https://mailman.rrr.se/cgi/listinfo/arslist

Re: SRM Fulfillment creation issue

2018-03-26 Thread LJ LongWing 
Well, for #1, you are issuing a query to the db that's not returning in the
120 second timeoutfor #2 you are getting disconnected from the
DBboth scenarios are pointing to a DB that is either overwhelmed, or
under performing...I would highly recommend working with your DBA to figure
out what's happening at the DB level first.

On Mon, Mar 26, 2018 at 11:52 AM, Dinesh Kumar  wrote:

> Hi All,
>
> We are facing below issue in our environment which causes a higher impact
> and Business critical.
>
>
>
> SRM Fulfillment requests are not getting generated.
>
>
>
> The requests are getting struck in the *CAI Events* form in the Running
> status for all the events (SRM_OUT_RESTART_SR_SUBMIT, TMS_OUT_GET_DATA).
>
> We could see that PDT’s are also not getting connected to the Service
> Requests. (adding screenshot).
>
>
>
>
>
> We are getting time out errors in the plugin and ARERR logs for the
> certain requests.
>
>
>
> *Error-1:*
>
>
>
> 2018-02-14 09:12:55,772 ERROR [pool-2-thread-4] 
> com.bmc.itsm.cai.filterapi.cai.worker.BaseEventWorker
> (BaseEventWorker.java:166) - CAI plugin failed to update error code of
> event:TMHAA5V0GJ3KKAPEER8HBGHK64BVRP. Failed with following exception.
> ERROR (94): Timeout during database query -- consider using more specific
> search criteria to narrow the results, and retry the operation;
> ATVIEUUSMS006:2000 ONC/RPC call timed out
>
>
>
> *Error-2:*
>
>
>
> Wed Feb 14 09:24:49 2018 390627 : The SQL database operation failed.
> (ARERR 552)
>
> Wed Feb 14 09:24:49 2018 ORA-03135: connection lost contact
>
> Process ID: 315236
>
> Session ID: 411 Serial number: 60447
>
> Wed Feb 14 09:24:49 2018 lastsql UPDATE T2457 SET
> C112=';100018;',C5='Remedy Application Service',C6=1518531886 WHERE C1
> = 'PDT5704'
>
> Wed Feb 14 09:24:53 2018 390627 : SQL database is now available (ARNOTE
> 592)
>
> SRM:ProcessDefinitionTemplate_Base(T2457) is getting locked and we are
> not able to create any request fulfillment after that.  Then we need to
> manually release.
>
>
>
> Has one faced this kind of issues.
>
>
> Thanks,
>
> Dinesh kumar.
>
> --
> ARSList mailing list
> ARSList@arslist.org
> https://mailman.rrr.se/cgi/listinfo/arslist
>
>
-- 
ARSList mailing list
ARSList@arslist.org
https://mailman.rrr.se/cgi/listinfo/arslist