Hi Alapan, For SQL error 1, can you try out the the solution suggested in stackoverflow[1] For SQL error 2, we have already fixed this in APIM Analytics 2.2.0. Issue reported for this can be found in here[2] and the fix can be found in here[3]
[1] https://stackoverflow.com/a/42787873/1110305 [2] https://wso2.org/jira/browse/ANLYAPIM-154 [3] https://github.com/wso2/analytics-apim/pull/369/files Chamila. On Wed, Apr 25, 2018 at 11:31 AM, <[email protected]> wrote: > Also Abi & Nuwan, > > > > I am getting below SQL errors every minute : > > > > *SQL Error 1:* > > > > <PostgreSQL JDBC Driver, apimuser, wso2_analytics_processed_data_store_db, > 10.138.120.215, 2018-04-23 12:03:57 CDT>STATEMENT: INSERT INTO > ANX___7LtOYcOI_ (partition_key, timestamp, data, record_id) VALUES ($1, $2, > $3, $4) > > <PostgreSQL JDBC Driver, apimuser, wso2_analytics_processed_data_store_db, > 10.138.120.215, 2018-04-23 12:03:57 CDT>ERROR: duplicate key value > violates unique constraint "anx___7ltoycoi__pkey" > > <PostgreSQL JDBC Driver, apimuser, wso2_analytics_processed_data_store_db, > 10.138.120.215, 2018-04-23 12:03:57 CDT>DETAIL: Key > (record_id)=(1fc48692-f672-3b1a-be77-2583bfd05bfa) already exists. > > <PostgreSQL JDBC Driver, apimuser, wso2_analytics_processed_data_store_db, > 10.138.120.215, 2018-04-23 12:03:57 CDT>STATEMENT: INSERT INTO > ANX___7LtOYcOI_ (partition_key, timestamp, data, record_id) VALUES ($1, $2, > $3, $4) > > <PostgreSQL JDBC Driver, apimuser, wso2_analytics_processed_data_store_db, > 10.138.120.215, 2018-04-23 12:03:57 CDT>ERROR: duplicate key value > violates unique constraint "anx___7ltoycoi__pkey" > > <PostgreSQL JDBC Driver, apimuser, wso2_analytics_processed_data_store_db, > 10.138.120.215, 2018-04-23 12:03:57 CDT>DETAIL: Key > (record_id)=(1da80896-bc33-34b2-92f1-2665cbc8c11f) already exists. > > <PostgreSQL JDBC Driver, apimuser, wso2_analytics_processed_data_store_db, > 10.138.120.215, 2018-04-23 12:03:57 CDT>STATEMENT: INSERT INTO > ANX___7LtOYcOI_ (partition_key, timestamp, data, record_id) VALUES ($1, $2, > $3, $4) > > <PostgreSQL JDBC Driver, apimuser, wso2_analytics_processed_data_store_db, > 10.138.120.215, 2018-04-23 12:03:57 CDT>ERROR: duplicate key value > violates unique constraint "anx___7ltoycoi__pkey" > > <PostgreSQL JDBC Driver, apimuser, wso2_analytics_processed_data_store_db, > 10.138.120.215, 2018-04-23 12:03:57 CDT>DETAIL: Key > (record_id)=(1da80896-bc33-34b2-92f1-2665cbc8c11f) already exists. > > > > *SQL Error 2:* > > > > Driver stacktrace: > > at org.apache.spark.sql.jdbc.carbon.package$CarbonJDBCWrite$.saveTable( > carbon.scala:78) > > at org.apache.spark.sql.jdbc.carbon.CarbonJDBCRelation. > insert(JDBCRelation.scala:263) > > ... 26 more > > Caused by: org.apache.spark.SparkException: Job aborted due to stage > failure: Task 0 in stage 3793.0 failed 1 times, most recent failure: Lost > task 0.0 in stage 3793.0 (TID 6627, localhost): > java.sql.BatchUpdateException: Batch entry 3 INSERT INTO > API_REQ_USER_BROW_SUMMARY (api, version, apiPublisher, tenantDomain, > total_request_count, year, month, day, requestTime, os, browser) VALUES > ('Hardware-Products', 'v1', 'synnex', 'carbon.super', 4, 2018, 4, 20, > 1524189658000, NULL, NULL) ON CONFLICT (api,version,apiPublisher, > year,month,day,os,browser,tenantDomain) DO UPDATE SET > total_request_count=EXCLUDED.total_request_count, > requestTime=EXCLUDED.requestTime > was aborted: ERROR: null value in column "os" violates not-null constraint > > Detail: Failing row contains (Hardware-Products, v1, synnex, > carbon.super, 4, 2018, 4, 20, 1524189658000, null, null). Call > getNextException to see other errors in the batch. > > at org.postgresql.jdbc.BatchResultHandler.handleError( > BatchResultHandler.java:148) > > at org.postgresql.core.v3.QueryExecutorImpl.processResults( > QueryExecutorImpl.java:2191) > > at org.postgresql.core.v3.QueryExecutorImpl.execute( > QueryExecutorImpl.java:472) > > at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:791) > > at org.postgresql.jdbc.PgPreparedStatement.executeBatch( > PgPreparedStatement.java:1563) > > at org.apache.spark.sql.jdbc.carbon.package$CarbonJDBCWrite$. > savePartition(carbon.scala:149) > > at org.apache.spark.sql.jdbc.carbon.package$CarbonJDBCWrite$$anonfun$ > saveTable$1.apply(carbon.scala:72) > > at org.apache.spark.sql.jdbc.carbon.package$CarbonJDBCWrite$$anonfun$ > saveTable$1.apply(carbon.scala:71) > > at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$ > anonfun$apply$33.apply(RDD.scala:920) > > at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun > > Best regards > > *Alapan Sur* > > Project Manager - SD > > ValueLabs - *'inspired by potential'* > > Desk: +91-40-6623-9000 | Ext: 32112 > > Mobile: +91-7799681717 > > This e-mail may contain information that is proprietary, confidential or > otherwise protected from disclosure and is sent for the intended > recipient(s) only. For email disclaimer details, please visit: > http://www.valuelabs.com/terms-of-use/ > > > > *From:* [email protected] [mailto:[email protected]] > *Sent:* Wednesday, April 25, 2018 10:01 AM > *To:* Abimaran Kugathasan <[email protected]> > *Cc:* Nuwan Dias <[email protected]>; dev <[email protected]> > *Subject:* RE: [Dev] One Specific API not responding > > > > Yes Abi. It hits and gives the response back in couple of seconds. > > > > Best regards > > *Alapan Sur* > > Project Manager - SD > > ValueLabs - *'inspired by potential'* > > Desk: +91-40-6623-9000 | Ext: 32112 > > Mobile: +91-7799681717 > > This e-mail may contain information that is proprietary, confidential or > otherwise protected from disclosure and is sent for the intended > recipient(s) only. For email disclaimer details, please visit: > http://www.valuelabs.com/terms-of-use/ > > > > *From:* Abimaran Kugathasan [mailto:[email protected] <[email protected]>] > > *Sent:* Wednesday, April 25, 2018 9:36 AM > *To:* [email protected] > *Cc:* Nuwan Dias <[email protected]>; dev <[email protected]> > > *Subject:* Re: [Dev] One Specific API not responding > > > > Hi Alapan, > > > > When you get Gateway timeout errors, do you see the request hit to your > backend services? > > > > On Tue, Apr 24, 2018 at 6:10 PM, <[email protected]> wrote: > > Thanks Nuwan, we will try that. > > > > Just to give you few more data pointers: > > 1. We are getting around 100 Kb of data as response. > > 2. When we do restart of the application everything work like charm > but after few 5 – 6 hours or 10 K – 15 K API hits, we are started getting > Gateway timeout errors. > > Best regards > > *Alapan Sur* > > Project Manager - SD > > ValueLabs - *'inspired by potential'* > > Desk: +91-40-6623-9000 | Ext: 32112 > > Mobile: +91-7799681717 > > This e-mail may contain information that is proprietary, confidential or > otherwise protected from disclosure and is sent for the intended > recipient(s) only. For email disclaimer details, please visit: > http://www.valuelabs.com/terms-of-use/ > > > > *From:* Nuwan Dias [mailto:[email protected]] > *Sent:* Tuesday, April 24, 2018 5:55 PM > > > *To:* [email protected] > *Cc:* Abimaran Kugathasan <[email protected]>; dev <[email protected]> > *Subject:* Re: [Dev] One Specific API not responding > > > > I was specifically interested in seeing the value of the Transfer-Encoding > request header. Since it appears to be missing, can you try disabling > chunking of this particular API by setting <property > name="DISABLE_CHUNKING" value="true" scope="axis2"/> in the api xml file > located at > <APIM_HOME>/repository/deployment/server/synapse-configs/default/api > ? You should put this config within the <inSequence> of the <resource> > being executed. > > > > If the above works, you could apply this solution as documented at [1]. > > > > [1] - https://docs.wso2.com/display/AM210/Disable+Message+Chunking > > > > Thanks, > > Nuwan. > > > > On Tue, Apr 24, 2018 at 5:47 PM, <[email protected]> wrote: > > Hello Nuwan, > > > > Please find the requested information for backend API directly : > > > > *Request Headers --* > > Source-Identifier: A076FCBB-41F3-4FB5-A9P8-DA2DCF6YR942 > > Content-Type: application/json > > Authorization-Token : > > > > *Response Headers -- * > > Cache-Control →no-cache > > Content-Length →25078 > > Content-Type →application/json; charset=utf-8 > > Date →Tue, 24 Apr 2018 12:11:06 GMT > > Expires →-1 > > Pragma →no-cache > > Server →Microsoft-IIS/8.0 > > X-AspNet-Version →4.0.30319 > > X-Powered-By →ASP.NET > > > > Best regards > > *Alapan Sur* > > Project Manager - SD > > ValueLabs - *'inspired by potential'* > > Desk: +91-40-6623-9000 | Ext: 32112 > > Mobile: +91-7799681717 > > This e-mail may contain information that is proprietary, confidential or > otherwise protected from disclosure and is sent for the intended > recipient(s) only. For email disclaimer details, please visit: > http://www.valuelabs.com/terms-of-use/ > > > > *From:* Nuwan Dias [mailto:[email protected]] > *Sent:* Tuesday, April 24, 2018 5:24 PM > *To:* [email protected] > *Cc:* Abimaran Kugathasan <[email protected]>; dev <[email protected]> > > > *Subject:* Re: [Dev] One Specific API not responding > > > > Hi Alan, > > > > Can you send the list of Request headers and Response headers from a > successful request that goes to your back-end directly (without going > through the Gateway)? > > > > Thanks, > > Nuwan. > > > > On Tue, Apr 24, 2018 at 5:03 PM, <[email protected]> wrote: > > Hello Abimaran, > > > > I have increased socket timeout and connection timeout to 3 mints and also > disabled the suspension of API, I am still getting 504 gateway timeout > error and sometime I get the response. Also I have checked the backend API > but we are getting response from backend server within 2 -3 seconds . > > > > I need immediate help to sort this out as we already moved this to > production and customers are facing issue with timeout. > > > > Best regards > > *Alapan Sur* > > Project Manager - SD > > ValueLabs - *'inspired by potential'* > > Desk: +91-40-6623-9000 | Ext: 32112 > > Mobile: +91-7799681717 > > This e-mail may contain information that is proprietary, confidential or > otherwise protected from disclosure and is sent for the intended > recipient(s) only. For email disclaimer details, please visit: > http://www.valuelabs.com/terms-of-use/ > > > > *From:* Abimaran Kugathasan [mailto:[email protected]] > *Sent:* Tuesday, April 24, 2018 4:41 PM > *To:* [email protected] > *Cc:* dev <[email protected]> > *Subject:* Re: [Dev] One Specific API not responding > > > > Hi Alapan, > > > > Do you want to increase socket timeout or remove the suspension of API? > > > > On Fri, Apr 20, 2018 at 11:24 AM, <[email protected]> wrote: > > Hello Team, > > > > Any idea on the below ? > > > > Best regards > > *Alapan Sur* > > Project Manager - SD > > ValueLabs - *'inspired by potential'* > > Desk: +91-40-6623-9000 | Ext: 32112 > > Mobile: +91-7799681717 > > This e-mail may contain information that is proprietary, confidential or > otherwise protected from disclosure and is sent for the intended > recipient(s) only. For email disclaimer details, please visit: > http://www.valuelabs.com/terms-of-use/ > > > > *From:* [email protected] [mailto:[email protected]] > *Sent:* Thursday, April 19, 2018 1:24 PM > *To:* '[email protected]' <[email protected]> > *Subject:* One Specific API not responding > > > > Hello Team, > > > > We are facing an issue file working with WSO2 API Manager 2.1.0. > > > > There is one request for a specific endpoint is continuously failing for > some reason and as a result, this specific API is getting down and we are > not getting an response from WSO2 APIM gateway although we are receiving > the response from backend API before the socket time out error. Please > find the below error we are getting : > > > > TID: [-1] [] [2018-04-19 04:54:24,294] WARN > {org.apache.synapse.transport.passthru.SourceHandler} > - Connection time out after request is read: http-incoming-106261 Socket > Timeout : 60000 Remote Address : /10.138.120.228:36754 > <https://l.facebook.com/l.php?u=http%3A%2F%2F10.138.120.228%3A36754%2F&h=ATOl7IzK7wN9mVsfkzG0fh6ZtnMR7L9K40MZWk0LYuJEbmNeWspvtYMO4BNt3VE7oDGgr23jwDtmmzAmNzl9jDrCHFvSnNLITEmpFqP6iWx4v_OkfvIqNnIxGpvMv6PB> > {org.apache.synapse.transport.passthru.SourceHandler} > > > Is there any threshold configuration which is configured to handle the > multiple API failure and WSO2 skips suspend the end point ? > <http://l.facebook.com/l.php?u=http%3A%2F%2F10.138.120.228%3A36754%2F&h=ATPsubv-DfKlSaiNx21YeahwUalUYsKADBcz4F-OaA0nNb7IrCxSzP9E4OvrEmhKGyMow8idhB7i0Ij9HBZZRiJUfMgIi7R31JXKm0GM27z_dVvC_69FV9pEg9RG9I6_2afAkKPvJcE&s=1> > > > > Can anyone help in this regard ? > > Best regards > > *Alapan Sur* > > Project Manager - SD > > ValueLabs - *'inspired by potential'* > > Desk: +91-40-6623-9000 | Ext: 32112 > > Mobile: +91-7799681717 > > This e-mail may contain information that is proprietary, confidential or > otherwise protected from disclosure and is sent for the intended > recipient(s) only. For email disclaimer details, please visit: > http://www.valuelabs.com/terms-of-use/ > > > > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > > > > > -- > > Thanks > > Abimaran Kugathasan > > Senior Software Engineer - API Technologies > > > > Email : [email protected] > > Mobile : +94 773922820 > > > > <http://stackoverflow.com/users/515034> > <http://lk.linkedin.com/in/abimaran> > <http://www.lkabimaran.blogspot.com/> <https://github.com/abimarank> > <https://twitter.com/abimaran> > > > > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > > > > > -- > > Nuwan Dias > > > > Software Architect - WSO2, Inc. http://wso2.com > > email : [email protected] > > Phone : +94 777 775 729 > > > > > > -- > > Nuwan Dias > > > > Software Architect - WSO2, Inc. http://wso2.com > > email : [email protected] > > Phone : +94 777 775 729 > > > > > > -- > > Thanks > > Abimaran Kugathasan > > Senior Software Engineer - API Technologies > > > > Email : [email protected] > > Mobile : +94 773922820 > > > > <http://stackoverflow.com/users/515034> > <http://lk.linkedin.com/in/abimaran> > <http://www.lkabimaran.blogspot.com/> <https://github.com/abimarank> > <https://twitter.com/abimaran> > > > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- Regards, Chamila Adhikarinayake Senior Software Engineer WSO2, Inc. Mobile - +94712346437 Email - [email protected] Blog - http://helpfromadhi.blogspot.com/
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
