Thank you Supun for that information. Regards, Malith
On Thu, Sep 22, 2016 at 11:41 AM, Supun Sethunga <sup...@wso2.com> wrote: > BTW, This issue has already been fixed in the analytics-apim master branch. > > Regards, > Supun > > On Wed, Sep 21, 2016 at 3:04 PM, Malith Munasinghe <mali...@wso2.com> > wrote: > >> Hi All, >> >> Thanks for the prompt responses we will do the needful. >> >> Regards, >> Malith >> >> On Wed, Sep 21, 2016 at 2:54 PM, Rukshan Premathunga <ruks...@wso2.com> >> wrote: >> >>> Hi Malith, >>> >>> cApp we provided to Analytics APIM will not work for the DAS because of >>> the changes happen to the DAS 3.1.0. Because of that we need to use >>> Analytics APIM or have to update capp with the above changes. >>> >>> Thanks and Regards. >>> >>> On Wed, Sep 21, 2016 at 2:48 PM, Niranda Perera <nira...@wso2.com> >>> wrote: >>> >>>> Hi Malith, >>>> >>>> Yes, correct! you need to change the script. >>>> >>>> Additionally, there are some changes in the carbonJdbc connector as >>>> well... so, you might need to watch out for it! >>>> >>>> Please check with the APIM team and ESB team whether we are doing a >>>> feature release with the DAS 310 changes? >>>> >>>> cheers >>>> >>>> On Wed, Sep 21, 2016 at 5:11 AM, Malith Munasinghe <mali...@wso2.com> >>>> wrote: >>>> >>>>> Hi All, >>>>> >>>>> While preparing a DAS 3.1.0 to run APIM Analytics I have added >>>>> features as in [1] >>>>> <https://docs.wso2.com/display/AM200/Installing+WSO2+APIM+Analytics+Features>. >>>>> After deploying the CApp for APIM Analytics I run in to below error. >>>>> According to the error that *incrementalProcessing *is not a valid >>>>> option. Also according to [2] >>>>> <https://docs.wso2.com/display/DAS310/Incremental+Processing> the >>>>> syntax to parse this option is *incrementalParams. *In order to get >>>>> DAS 3.1.0 to process APIM Analytics >>>>> do we have to change the scripts with this option as well ? >>>>> >>>>> >>>>> TID: [-1234] [] [2016-09-21 08:54:00,019] ERROR >>>>> {org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService} >>>>> - Error while executing query : CREATE TEMPORARY TABLE >>>>> APIMGT_PERMINUTE_REQUEST_DATA USING CarbonAnalytics OPTIONS(tableName >>>>> "ORG_WSO2_APIMGT_STATISTICS_PERMINUTEREQUEST", schema " year INT >>>>> -i, month INT -i, day INT -i, hour INT -i, minute INT -i, consumerKey >>>>> STRING, context STRING, api_version STRING, api STRING, version STRING, >>>>> requestTime LONG, userId STRING, hostName STRING, apiPublisher STRING, >>>>> total_request_count LONG, resourceTemplate STRING, method STRING, >>>>> applicationName STRING, tenantDomain STRING, userAgent STRING, >>>>> resourcePath STRING, request INT, applicationId STRING, tier STRING, >>>>> throttledOut BOOLEAN, clientIp STRING, applicationOwner STRING, >>>>> _timestamp LONG -i", primaryKeys "year, month, day, hour, minute, >>>>> consumerKey, context, api_version, userId, hostName, apiPublisher, >>>>> resourceTemplate, method, userAgent, clientIp", incrementalProcessing >>>>> "APIMGT_PERMINUTE_REQUEST_DATA, HOUR", mergeSchema "false") >>>>> {org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService} >>>>> org.wso2.carbon.analytics.spark.core.exception.AnalyticsExecutionException: >>>>> Exception in executing query CREATE TEMPORARY TABLE >>>>> APIMGT_PERMINUTE_REQUEST_DATA USING CarbonAnalytics OPTIONS(tableName >>>>> "ORG_WSO2_APIMGT_STATISTICS_PERMINUTEREQUEST", schema " year INT >>>>> -i, month INT -i, day INT -i, hour INT -i, minute INT -i, consumerKey >>>>> STRING, context STRING, api_version STRING, api STRING, version STRING, >>>>> requestTime LONG, userId STRING, hostName STRING, apiPublisher STRING, >>>>> total_request_count LONG, resourceTemplate STRING, method STRING, >>>>> applicationName STRING, tenantDomain STRING, userAgent STRING, >>>>> resourcePath STRING, request INT, applicationId STRING, tier STRING, >>>>> throttledOut BOOLEAN, clientIp STRING, applicationOwner STRING, >>>>> _timestamp LONG -i", primaryKeys "year, month, day, hour, minute, >>>>> consumerKey, context, api_version, userId, hostName, apiPublisher, >>>>> resourceTemplate, method, userAgent, clientIp", incrementalProcessing >>>>> "APIMGT_PERMINUTE_REQUEST_DATA, HOUR", mergeSchema "false") >>>>> at org.wso2.carbon.analytics.spark.core.internal.SparkAnalytics >>>>> Executor.executeQueryLocal(SparkAnalyticsExecutor.java:764) >>>>> at org.wso2.carbon.analytics.spark.core.internal.SparkAnalytics >>>>> Executor.executeQuery(SparkAnalyticsExecutor.java:721) >>>>> at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcesso >>>>> rService.executeQuery(CarbonAnalyticsProcessorService.java:201) >>>>> at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcesso >>>>> rService.executeScript(CarbonAnalyticsProcessorService.java:151) >>>>> at org.wso2.carbon.analytics.spark.core.AnalyticsTask.execute(A >>>>> nalyticsTask.java:60) >>>>> at org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute >>>>> (TaskQuartzJobAdapter.java:67) >>>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executor >>>>> s.java:471) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>>>> Executor.java:1145) >>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>>>> lExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> Caused by: java.lang.RuntimeException: Unknown options : >>>>> incrementalprocessing >>>>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati >>>>> onProvider.checkParameters(AnalyticsRelationProvider.java:123) >>>>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati >>>>> onProvider.setParameters(AnalyticsRelationProvider.java:113) >>>>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati >>>>> onProvider.createRelation(AnalyticsRelationProvider.java:75) >>>>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati >>>>> onProvider.createRelation(AnalyticsRelationProvider.java:45) >>>>> at org.apache.spark.sql.execution.datasources.ResolvedDataSourc >>>>> e$.apply(ResolvedDataSource.scala:158) >>>>> at org.apache.spark.sql.execution.datasources.CreateTempTableUs >>>>> ing.run(ddl.scala:92) >>>>> at org.apache.spark.sql.execution.ExecutedCommand.sideEffectRes >>>>> ult$lzycompute(commands.scala:58) >>>>> at org.apache.spark.sql.execution.ExecutedCommand.sideEffectRes >>>>> ult(commands.scala:56) >>>>> at org.apache.spark.sql.execution.ExecutedCommand.doExecute(com >>>>> mands.scala:70) >>>>> at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5. >>>>> apply(SparkPlan.scala:132) >>>>> at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5. >>>>> apply(SparkPlan.scala:130) >>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperati >>>>> onScope.scala:150) >>>>> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.s >>>>> cala:130) >>>>> at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompu >>>>> te(QueryExecution.scala:55) >>>>> at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExe >>>>> cution.scala:55) >>>>> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:145) >>>>> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:130) >>>>> at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52) >>>>> at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817) >>>>> at org.wso2.carbon.analytics.spark.core.internal.SparkAnalytics >>>>> Executor.executeQueryLocal(SparkAnalyticsExecutor.java:760) >>>>> ... 11 more >>>>> TID: [-1234] [] [2016-09-21 08:54:00,020] ERROR >>>>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Error while >>>>> executing the scheduled task for the script: >>>>> APIM_INCREMENTAL_PROCESSING_SCRIPT >>>>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask} >>>>> org.wso2.carbon.analytics.spark.core.exception.AnalyticsExecutionException: >>>>> Exception in executing query CREATE TEMPORARY TABLE >>>>> APIMGT_PERMINUTE_REQUEST_DATA USING CarbonAnalytics OPTIONS(tableName >>>>> "ORG_WSO2_APIMGT_STATISTICS_PERMINUTEREQUEST", schema " year INT >>>>> -i, month INT -i, day INT -i, hour INT -i, minute INT -i, consumerKey >>>>> STRING, context STRING, api_version STRING, api STRING, version STRING, >>>>> requestTime LONG, userId STRING, hostName STRING, apiPublisher STRING, >>>>> total_request_count LONG, resourceTemplate STRING, method STRING, >>>>> applicationName STRING, tenantDomain STRING, userAgent STRING, >>>>> resourcePath STRING, request INT, applicationId STRING, tier STRING, >>>>> throttledOut BOOLEAN, clientIp STRING, applicationOwner STRING, >>>>> _timestamp LONG -i", primaryKeys "year, month, day, hour, minute, >>>>> consumerKey, context, api_version, userId, hostName, apiPublisher, >>>>> resourceTemplate, method, userAgent, clientIp", incrementalProcessing >>>>> "APIMGT_PERMINUTE_REQUEST_DATA, HOUR", mergeSchema "false") >>>>> at org.wso2.carbon.analytics.spark.core.internal.SparkAnalytics >>>>> Executor.executeQueryLocal(SparkAnalyticsExecutor.java:764) >>>>> at org.wso2.carbon.analytics.spark.core.internal.SparkAnalytics >>>>> Executor.executeQuery(SparkAnalyticsExecutor.java:721) >>>>> at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcesso >>>>> rService.executeQuery(CarbonAnalyticsProcessorService.java:201) >>>>> at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcesso >>>>> rService.executeScript(CarbonAnalyticsProcessorService.java:151) >>>>> at org.wso2.carbon.analytics.spark.core.AnalyticsTask.execute(A >>>>> nalyticsTask.java:60) >>>>> at org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute >>>>> (TaskQuartzJobAdapter.java:67) >>>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executor >>>>> s.java:471) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>>>> Executor.java:1145) >>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>>>> lExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> Caused by: java.lang.RuntimeException: Unknown options : >>>>> incrementalprocessing >>>>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati >>>>> onProvider.checkParameters(AnalyticsRelationProvider.java:123) >>>>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati >>>>> onProvider.setParameters(AnalyticsRelationProvider.java:113) >>>>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati >>>>> onProvider.createRelation(AnalyticsRelationProvider.java:75) >>>>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati >>>>> onProvider.createRelation(AnalyticsRelationProvider.java:45) >>>>> at org.apache.spark.sql.execution.datasources.ResolvedDataSourc >>>>> e$.apply(ResolvedDataSource.scala:158) >>>>> at org.apache.spark.sql.execution.datasources.CreateTempTableUs >>>>> ing.run(ddl.scala:92) >>>>> at org.apache.spark.sql.execution.ExecutedCommand.sideEffectRes >>>>> ult$lzycompute(commands.scala:58) >>>>> at org.apache.spark.sql.execution.ExecutedCommand.sideEffectRes >>>>> ult(commands.scala:56) >>>>> at org.apache.spark.sql.execution.ExecutedCommand.doExecute(com >>>>> mands.scala:70) >>>>> at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5. >>>>> apply(SparkPlan.scala:132) >>>>> at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5. >>>>> apply(SparkPlan.scala:130) >>>>> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperati >>>>> onScope.scala:150) >>>>> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.s >>>>> cala:130) >>>>> at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompu >>>>> te(QueryExecution.scala:55) >>>>> at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExe >>>>> cution.scala:55) >>>>> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:145) >>>>> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:130) >>>>> at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52) >>>>> at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817) >>>>> at org.wso2.carbon.analytics.spark.core.internal.SparkAnalytics >>>>> Executor.executeQueryLocal(SparkAnalyticsExecutor.java:760) >>>>> ... 11 more >>>>> >>>>> >>>>> [1] https://docs.wso2.com/display/AM200/Installing+WSO2+APIM >>>>> +Analytics+Features >>>>> [2] https://docs.wso2.com/display/DAS310/Incremental+Processing >>>>> >>>>> Regards, >>>>> Malith >>>>> -- >>>>> Malith Munasinghe | Software Engineer >>>>> M: +94 (71) 9401122 >>>>> E: mali...@wso2.com >>>>> W: http://wso2.com >>>>> <http://wso2.com/signature> >>>>> >>>> >>>> >>>> >>>> -- >>>> *Niranda Perera* >>>> Software Engineer, WSO2 Inc. >>>> Mobile: +94-71-554-8430 >>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>> https://pythagoreanscript.wordpress.com/ >>>> >>> >>> >>> >>> -- >>> Rukshan Chathuranga. >>> Software Engineer. >>> WSO2, Inc. >>> >> >> >> >> -- >> Malith Munasinghe | Software Engineer >> M: +94 (71) 9401122 >> E: mali...@wso2.com >> W: http://wso2.com >> <http://wso2.com/signature> >> >> _______________________________________________ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > *Supun Sethunga* > Senior Software Engineer > WSO2, Inc. > http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > Blog: http://supunsetunga.blogspot.com > -- Malith Munasinghe | Software Engineer M: +94 (71) 9401122 E: mali...@wso2.com W: http://wso2.com <http://wso2.com/signature>
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
