Hi Malith,

cApp we provided to Analytics APIM will not work for the DAS because of the
changes happen to the DAS 3.1.0. Because of that we need to use Analytics
APIM or have to update capp with the above changes.

Thanks and Regards.

On Wed, Sep 21, 2016 at 2:48 PM, Niranda Perera <nira...@wso2.com> wrote:

> Hi Malith,
>
> Yes, correct! you need to change the script.
>
> Additionally, there are some changes in the carbonJdbc connector as
> well... so, you might need to watch out for it!
>
> Please check with the APIM team and ESB team whether we are doing a
> feature release with the DAS 310 changes?
>
> cheers
>
> On Wed, Sep 21, 2016 at 5:11 AM, Malith Munasinghe <mali...@wso2.com>
> wrote:
>
>> Hi All,
>>
>> While preparing a DAS 3.1.0 to run APIM Analytics I have added features
>> as in [1]
>> <https://docs.wso2.com/display/AM200/Installing+WSO2+APIM+Analytics+Features>.
>> After deploying the CApp for APIM Analytics I run in to below error.
>> According to the error that *incrementalProcessing *is not a valid
>> option. Also according to [2]
>> <https://docs.wso2.com/display/DAS310/Incremental+Processing> the syntax
>> to parse this option is *incrementalParams. *In order to get DAS 3.1.0
>> to process APIM Analytics
>> do we have to change the scripts with this option as well ?
>>
>>
>> TID: [-1234] [] [2016-09-21 08:54:00,019] ERROR
>> {org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService} -
>>  Error while executing query :         CREATE TEMPORARY TABLE
>> APIMGT_PERMINUTE_REQUEST_DATA USING CarbonAnalytics OPTIONS(tableName
>> "ORG_WSO2_APIMGT_STATISTICS_PERMINUTEREQUEST", schema "    year INT -i,
>> month INT -i, day INT -i, hour INT -i, minute INT -i,    consumerKey
>> STRING, context STRING, api_version STRING, api STRING, version STRING,
>> requestTime LONG, userId STRING, hostName STRING,    apiPublisher STRING,
>> total_request_count LONG, resourceTemplate STRING, method STRING,
>> applicationName STRING, tenantDomain STRING,    userAgent STRING,
>> resourcePath STRING, request INT, applicationId STRING, tier STRING,
>> throttledOut BOOLEAN, clientIp STRING,    applicationOwner STRING,
>> _timestamp LONG -i",    primaryKeys "year, month, day, hour, minute,
>> consumerKey, context, api_version, userId, hostName, apiPublisher,
>> resourceTemplate, method, userAgent, clientIp",    incrementalProcessing
>> "APIMGT_PERMINUTE_REQUEST_DATA, HOUR",    mergeSchema "false")
>> {org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService}
>> org.wso2.carbon.analytics.spark.core.exception.AnalyticsExecutionException:
>> Exception in executing query CREATE TEMPORARY TABLE
>> APIMGT_PERMINUTE_REQUEST_DATA USING CarbonAnalytics OPTIONS(tableName
>> "ORG_WSO2_APIMGT_STATISTICS_PERMINUTEREQUEST", schema "    year INT -i,
>> month INT -i, day INT -i, hour INT -i, minute INT -i,    consumerKey
>> STRING, context STRING, api_version STRING, api STRING, version STRING,
>> requestTime LONG, userId STRING, hostName STRING,    apiPublisher STRING,
>> total_request_count LONG, resourceTemplate STRING, method STRING,
>> applicationName STRING, tenantDomain STRING,    userAgent STRING,
>> resourcePath STRING, request INT, applicationId STRING, tier STRING,
>> throttledOut BOOLEAN, clientIp STRING,    applicationOwner STRING,
>> _timestamp LONG -i",    primaryKeys "year, month, day, hour, minute,
>> consumerKey, context, api_version, userId, hostName, apiPublisher,
>> resourceTemplate, method, userAgent, clientIp",    incrementalProcessing
>> "APIMGT_PERMINUTE_REQUEST_DATA, HOUR",    mergeSchema "false")
>> at org.wso2.carbon.analytics.spark.core.internal.SparkAnalytics
>> Executor.executeQueryLocal(SparkAnalyticsExecutor.java:764)
>> at org.wso2.carbon.analytics.spark.core.internal.SparkAnalytics
>> Executor.executeQuery(SparkAnalyticsExecutor.java:721)
>> at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcesso
>> rService.executeQuery(CarbonAnalyticsProcessorService.java:201)
>> at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcesso
>> rService.executeScript(CarbonAnalyticsProcessorService.java:151)
>> at org.wso2.carbon.analytics.spark.core.AnalyticsTask.execute(
>> AnalyticsTask.java:60)
>> at org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute
>> (TaskQuartzJobAdapter.java:67)
>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>> at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.RuntimeException: Unknown options :
>> incrementalprocessing
>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati
>> onProvider.checkParameters(AnalyticsRelationProvider.java:123)
>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati
>> onProvider.setParameters(AnalyticsRelationProvider.java:113)
>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati
>> onProvider.createRelation(AnalyticsRelationProvider.java:75)
>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati
>> onProvider.createRelation(AnalyticsRelationProvider.java:45)
>> at org.apache.spark.sql.execution.datasources.ResolvedDataSourc
>> e$.apply(ResolvedDataSource.scala:158)
>> at org.apache.spark.sql.execution.datasources.CreateTempTableUs
>> ing.run(ddl.scala:92)
>> at org.apache.spark.sql.execution.ExecutedCommand.sideEffectRes
>> ult$lzycompute(commands.scala:58)
>> at org.apache.spark.sql.execution.ExecutedCommand.sideEffectRes
>> ult(commands.scala:56)
>> at org.apache.spark.sql.execution.ExecutedCommand.doExecute(
>> commands.scala:70)
>> at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.
>> apply(SparkPlan.scala:132)
>> at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.
>> apply(SparkPlan.scala:130)
>> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperati
>> onScope.scala:150)
>> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
>> at org.apache.spark.sql.execution.QueryExecution.toRdd$
>> lzycompute(QueryExecution.scala:55)
>> at org.apache.spark.sql.execution.QueryExecution.toRdd(
>> QueryExecution.scala:55)
>> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:145)
>> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:130)
>> at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52)
>> at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817)
>> at org.wso2.carbon.analytics.spark.core.internal.SparkAnalytics
>> Executor.executeQueryLocal(SparkAnalyticsExecutor.java:760)
>> ... 11 more
>> TID: [-1234] [] [2016-09-21 08:54:00,020] ERROR
>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask} -  Error while
>> executing the scheduled task for the script: 
>> APIM_INCREMENTAL_PROCESSING_SCRIPT
>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask}
>> org.wso2.carbon.analytics.spark.core.exception.AnalyticsExecutionException:
>> Exception in executing query CREATE TEMPORARY TABLE
>> APIMGT_PERMINUTE_REQUEST_DATA USING CarbonAnalytics OPTIONS(tableName
>> "ORG_WSO2_APIMGT_STATISTICS_PERMINUTEREQUEST", schema "    year INT -i,
>> month INT -i, day INT -i, hour INT -i, minute INT -i,    consumerKey
>> STRING, context STRING, api_version STRING, api STRING, version STRING,
>> requestTime LONG, userId STRING, hostName STRING,    apiPublisher STRING,
>> total_request_count LONG, resourceTemplate STRING, method STRING,
>> applicationName STRING, tenantDomain STRING,    userAgent STRING,
>> resourcePath STRING, request INT, applicationId STRING, tier STRING,
>> throttledOut BOOLEAN, clientIp STRING,    applicationOwner STRING,
>> _timestamp LONG -i",    primaryKeys "year, month, day, hour, minute,
>> consumerKey, context, api_version, userId, hostName, apiPublisher,
>> resourceTemplate, method, userAgent, clientIp",    incrementalProcessing
>> "APIMGT_PERMINUTE_REQUEST_DATA, HOUR",    mergeSchema "false")
>> at org.wso2.carbon.analytics.spark.core.internal.SparkAnalytics
>> Executor.executeQueryLocal(SparkAnalyticsExecutor.java:764)
>> at org.wso2.carbon.analytics.spark.core.internal.SparkAnalytics
>> Executor.executeQuery(SparkAnalyticsExecutor.java:721)
>> at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcesso
>> rService.executeQuery(CarbonAnalyticsProcessorService.java:201)
>> at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcesso
>> rService.executeScript(CarbonAnalyticsProcessorService.java:151)
>> at org.wso2.carbon.analytics.spark.core.AnalyticsTask.execute(
>> AnalyticsTask.java:60)
>> at org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute
>> (TaskQuartzJobAdapter.java:67)
>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>> at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.RuntimeException: Unknown options :
>> incrementalprocessing
>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati
>> onProvider.checkParameters(AnalyticsRelationProvider.java:123)
>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati
>> onProvider.setParameters(AnalyticsRelationProvider.java:113)
>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati
>> onProvider.createRelation(AnalyticsRelationProvider.java:75)
>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati
>> onProvider.createRelation(AnalyticsRelationProvider.java:45)
>> at org.apache.spark.sql.execution.datasources.ResolvedDataSourc
>> e$.apply(ResolvedDataSource.scala:158)
>> at org.apache.spark.sql.execution.datasources.CreateTempTableUs
>> ing.run(ddl.scala:92)
>> at org.apache.spark.sql.execution.ExecutedCommand.sideEffectRes
>> ult$lzycompute(commands.scala:58)
>> at org.apache.spark.sql.execution.ExecutedCommand.sideEffectRes
>> ult(commands.scala:56)
>> at org.apache.spark.sql.execution.ExecutedCommand.doExecute(
>> commands.scala:70)
>> at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.
>> apply(SparkPlan.scala:132)
>> at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.
>> apply(SparkPlan.scala:130)
>> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperati
>> onScope.scala:150)
>> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
>> at org.apache.spark.sql.execution.QueryExecution.toRdd$
>> lzycompute(QueryExecution.scala:55)
>> at org.apache.spark.sql.execution.QueryExecution.toRdd(
>> QueryExecution.scala:55)
>> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:145)
>> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:130)
>> at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52)
>> at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817)
>> at org.wso2.carbon.analytics.spark.core.internal.SparkAnalytics
>> Executor.executeQueryLocal(SparkAnalyticsExecutor.java:760)
>> ... 11 more
>>
>>
>> [1] https://docs.wso2.com/display/AM200/Installing+WSO2+APIM
>> +Analytics+Features
>> [2] https://docs.wso2.com/display/DAS310/Incremental+Processing
>>
>> Regards,
>> Malith
>> --
>> Malith Munasinghe | Software Engineer
>> M: +94 (71) 9401122
>> E: mali...@wso2.com
>> W: http://wso2.com
>> <http://wso2.com/signature>
>>
>
>
>
> --
> *Niranda Perera*
> Software Engineer, WSO2 Inc.
> Mobile: +94-71-554-8430
> Twitter: @n1r44 <https://twitter.com/N1R44>
> https://pythagoreanscript.wordpress.com/
>



-- 
Rukshan Chathuranga.
Software Engineer.
WSO2, Inc.
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Reply via email to