Hi All,

While preparing a DAS 3.1.0 to run APIM Analytics I have added features as
in [1]
<https://docs.wso2.com/display/AM200/Installing+WSO2+APIM+Analytics+Features>.
After deploying the CApp for APIM Analytics I run in to below error.
According to the error that *incrementalProcessing *is not a valid option.
Also according to [2]
<https://docs.wso2.com/display/DAS310/Incremental+Processing> the syntax to
parse this option is *incrementalParams. *In order to get DAS 3.1.0 to
process APIM Analytics
do we have to change the scripts with this option as well ?


TID: [-1234] [] [2016-09-21 08:54:00,019] ERROR
{org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService} -
 Error while executing query :         CREATE TEMPORARY TABLE
APIMGT_PERMINUTE_REQUEST_DATA USING CarbonAnalytics OPTIONS(tableName
"ORG_WSO2_APIMGT_STATISTICS_PERMINUTEREQUEST", schema "    year INT -i,
month INT -i, day INT -i, hour INT -i, minute INT -i,    consumerKey
STRING, context STRING, api_version STRING, api STRING, version STRING,
requestTime LONG, userId STRING, hostName STRING,    apiPublisher STRING,
total_request_count LONG, resourceTemplate STRING, method STRING,
applicationName STRING, tenantDomain STRING,    userAgent STRING,
resourcePath STRING, request INT, applicationId STRING, tier STRING,
throttledOut BOOLEAN, clientIp STRING,    applicationOwner STRING,
_timestamp LONG -i",    primaryKeys "year, month, day, hour, minute,
consumerKey, context, api_version, userId, hostName, apiPublisher,
resourceTemplate, method, userAgent, clientIp",    incrementalProcessing
"APIMGT_PERMINUTE_REQUEST_DATA, HOUR",    mergeSchema "false")
{org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService}
org.wso2.carbon.analytics.spark.core.exception.AnalyticsExecutionException:
Exception in executing query CREATE TEMPORARY TABLE
APIMGT_PERMINUTE_REQUEST_DATA USING CarbonAnalytics OPTIONS(tableName
"ORG_WSO2_APIMGT_STATISTICS_PERMINUTEREQUEST", schema "    year INT -i,
month INT -i, day INT -i, hour INT -i, minute INT -i,    consumerKey
STRING, context STRING, api_version STRING, api STRING, version STRING,
requestTime LONG, userId STRING, hostName STRING,    apiPublisher STRING,
total_request_count LONG, resourceTemplate STRING, method STRING,
applicationName STRING, tenantDomain STRING,    userAgent STRING,
resourcePath STRING, request INT, applicationId STRING, tier STRING,
throttledOut BOOLEAN, clientIp STRING,    applicationOwner STRING,
_timestamp LONG -i",    primaryKeys "year, month, day, hour, minute,
consumerKey, context, api_version, userId, hostName, apiPublisher,
resourceTemplate, method, userAgent, clientIp",    incrementalProcessing
"APIMGT_PERMINUTE_REQUEST_DATA, HOUR",    mergeSchema "false")
at
org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQueryLocal(SparkAnalyticsExecutor.java:764)
at
org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQuery(SparkAnalyticsExecutor.java:721)
at
org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeQuery(CarbonAnalyticsProcessorService.java:201)
at
org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeScript(CarbonAnalyticsProcessorService.java:151)
at
org.wso2.carbon.analytics.spark.core.AnalyticsTask.execute(AnalyticsTask.java:60)
at
org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:67)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Unknown options :
incrementalprocessing
at
org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelationProvider.checkParameters(AnalyticsRelationProvider.java:123)
at
org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelationProvider.setParameters(AnalyticsRelationProvider.java:113)
at
org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelationProvider.createRelation(AnalyticsRelationProvider.java:75)
at
org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelationProvider.createRelation(AnalyticsRelationProvider.java:45)
at
org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:158)
at
org.apache.spark.sql.execution.datasources.CreateTempTableUsing.run(ddl.scala:92)
at
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
at
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)
at
org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55)
at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:145)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:130)
at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817)
at
org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQueryLocal(SparkAnalyticsExecutor.java:760)
... 11 more
TID: [-1234] [] [2016-09-21 08:54:00,020] ERROR
{org.wso2.carbon.analytics.spark.core.AnalyticsTask} -  Error while
executing the scheduled task for the script:
APIM_INCREMENTAL_PROCESSING_SCRIPT
{org.wso2.carbon.analytics.spark.core.AnalyticsTask}
org.wso2.carbon.analytics.spark.core.exception.AnalyticsExecutionException:
Exception in executing query CREATE TEMPORARY TABLE
APIMGT_PERMINUTE_REQUEST_DATA USING CarbonAnalytics OPTIONS(tableName
"ORG_WSO2_APIMGT_STATISTICS_PERMINUTEREQUEST", schema "    year INT -i,
month INT -i, day INT -i, hour INT -i, minute INT -i,    consumerKey
STRING, context STRING, api_version STRING, api STRING, version STRING,
requestTime LONG, userId STRING, hostName STRING,    apiPublisher STRING,
total_request_count LONG, resourceTemplate STRING, method STRING,
applicationName STRING, tenantDomain STRING,    userAgent STRING,
resourcePath STRING, request INT, applicationId STRING, tier STRING,
throttledOut BOOLEAN, clientIp STRING,    applicationOwner STRING,
_timestamp LONG -i",    primaryKeys "year, month, day, hour, minute,
consumerKey, context, api_version, userId, hostName, apiPublisher,
resourceTemplate, method, userAgent, clientIp",    incrementalProcessing
"APIMGT_PERMINUTE_REQUEST_DATA, HOUR",    mergeSchema "false")
at
org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQueryLocal(SparkAnalyticsExecutor.java:764)
at
org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQuery(SparkAnalyticsExecutor.java:721)
at
org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeQuery(CarbonAnalyticsProcessorService.java:201)
at
org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeScript(CarbonAnalyticsProcessorService.java:151)
at
org.wso2.carbon.analytics.spark.core.AnalyticsTask.execute(AnalyticsTask.java:60)
at
org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:67)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Unknown options :
incrementalprocessing
at
org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelationProvider.checkParameters(AnalyticsRelationProvider.java:123)
at
org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelationProvider.setParameters(AnalyticsRelationProvider.java:113)
at
org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelationProvider.createRelation(AnalyticsRelationProvider.java:75)
at
org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelationProvider.createRelation(AnalyticsRelationProvider.java:45)
at
org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:158)
at
org.apache.spark.sql.execution.datasources.CreateTempTableUsing.run(ddl.scala:92)
at
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
at
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)
at
org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55)
at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:145)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:130)
at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817)
at
org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQueryLocal(SparkAnalyticsExecutor.java:760)
... 11 more


[1]
https://docs.wso2.com/display/AM200/Installing+WSO2+APIM+Analytics+Features
[2] https://docs.wso2.com/display/DAS310/Incremental+Processing

Regards,
Malith
-- 
Malith Munasinghe | Software Engineer
M: +94 (71) 9401122
E: mali...@wso2.com
W: http://wso2.com
<http://wso2.com/signature>
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Reply via email to