[
https://issues.apache.org/jira/browse/EAGLE-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15624397#comment-15624397
]
ASF GitHub Bot commented on EAGLE-704:
--------------------------------------
Github user wujinhu commented on a diff in the pull request:
https://github.com/apache/incubator-eagle/pull/591#discussion_r85876408
--- Diff:
eagle-jpm/eagle-jpm-spark-history/src/main/resources/META-INF/providers/org.apache.eagle.jpm.spark.history.SparkHistoryJobAppProvider.xml
---
@@ -18,179 +18,127 @@
<application>
<type>SPARK_HISTORY_JOB_APP</type>
- <name>Spark History Job Monitoring</name>
+ <name>Spark History Job Monitor</name>
<version>0.5.0-incubating</version>
<appClass>org.apache.eagle.jpm.spark.history.SparkHistoryJobApp</appClass>
<configuration>
- <!-- org.apache.eagle.jpm.spark.history.SparkHistoryJobAppConfig
-->
+ <!-- topology config -->
<property>
- <name>basic.cluster</name>
- <displayName>cluster</displayName>
- <description>Cluster Name</description>
- <value>sandbox</value>
+ <name>workers</name>
+ <displayName>topology workers</displayName>
+ <description>topology workers</description>
+ <value>1</value>
</property>
<property>
- <name>basic.dataCenter</name>
- <displayName>dataCenter</displayName>
- <description>Data Center</description>
- <value>sandbox</value>
+ <name>topology.numOfSpoutExecutors</name>
+ <displayName>spout executors</displayName>
+ <description>Parallelism of sparkHistoryJobFetchSpout
</description>
+ <value>1</value>
</property>
<property>
- <name>basic.jobConf.additional.info</name>
- <displayName>jobConf.additional.info</displayName>
- <description>Additional info in Job Configs</description>
- <value></value>
+ <name>topology.numOfSpoutTasks</name>
+ <displayName>spout tasks</displayName>
+ <description>Tasks Num of sparkHistoryJobFetchSpout
</description>
+ <value>4</value>
+ </property>
+ <property>
+ <name>topology.numOfParseBoltExecutors</name>
+ <displayName>parser bolt parallelism hint</displayName>
+ <description>Parallelism of sparkHistoryJobParseBolt
</description>
+ <value>1</value>
+ </property>
+ <property>
+ <name>topology.numOfParserBoltTasks</name>
+ <displayName>parser bolt tasks</displayName>
+ <description>Tasks Num of
sparkHistoryJobParseBolt</description>
+ <value>4</value>
+ </property>
+ <property>
+ <name>topology.spoutCrawlInterval</name>
+ <displayName>spout crawl interval</displayName>
+ <description>Spout crawl interval (in
milliseconds)</description>
+ <value>10000</value>
</property>
<property>
- <name>dataSourceConfig.zkQuorum</name>
- <displayName>zkQuorum</displayName>
- <description>Zookeeper Quorum</description>
+ <name>topology.message.timeout.secs</name>
+ <displayName>topology message timeout (secs)</displayName>
+ <description>default timeout is 30s</description>
+ <value>300</value>
+ </property>
+ <!-- zookeeper config -->
+ <property>
+ <name>zkStateConfig.zkQuorum</name>
+ <displayName>zookeeper quorum list</displayName>
+ <description>zookeeper to store topology metadata</description>
<value>sandbox.hortonworks.com:2181</value>
</property>
<property>
- <name>dataSourceConfig.zkRoot</name>
- <displayName>zkRoot</displayName>
+ <name>zkStateConfig.zkRoot</name>
+ <displayName>zookeeper root for topology metadata</displayName>
<description>Zookeeper Root</description>
<value>/sparkHistoryJob</value>
</property>
<property>
- <name>dataSourceConfig.zkPort</name>
- <displayName>zkPort</displayName>
- <description>Zookeeper Port</description>
- <value>2181</value>
- </property>
- <property>
- <name>dataSourceConfig.zkSessionTimeoutMs</name>
- <displayName>zkSessionTimeoutMs</displayName>
+ <name>zkStateConfig.zkSessionTimeoutMs</name>
+ <displayName>zookeeper session timeout (ms)</displayName>
<description>Zookeeper session timeoutMs</description>
<value>15000</value>
</property>
<property>
- <name>zookeeperConfig.zkRetryTimes</name>
- <displayName>zkRetryTimes</displayName>
- <description>zookeeperConfig.zkRetryTimes</description>
+ <name>zkStateConfig.zkRetryTimes</name>
+ <displayName>zookeeper connection retry times</displayName>
+ <description>retry times for zookeeper connection</description>
<value>3</value>
</property>
<property>
- <name>zookeeperConfig.zkRetryInterval</name>
- <displayName>zkRetryInterval</displayName>
- <description>zookeeperConfig.zkRetryInterval</description>
+ <name>zkStateConfig.zkRetryInterval</name>
+ <displayName>zookeeper connection retry interval</displayName>
+ <description>retry interval for zookeeper
connection</description>
<value>20000</value>
</property>
+
+ <!-- datasource config -->
<property>
<name>dataSourceConfig.spark.history.server.url</name>
- <displayName>spark.history.server.url</displayName>
- <description>Spark History Server URL</description>
+ <displayName>spark history server url</displayName>
+ <description>spark history server URL</description>
<value>http://sandbox.hortonworks.com:18080</value>
+ <required>true</required>
</property>
<property>
<name>dataSourceConfig.spark.history.server.username</name>
- <displayName>spark.history.server.username</displayName>
- <description>Spark History Server Auth Username</description>
+ <displayName>user name for spark history server</displayName>
+ <description>spark history server auth username</description>
--- End diff --
It seems we do not use spark history server now, the codes have been
commented
> Update spark history config to integrate with the new application framework
> ---------------------------------------------------------------------------
>
> Key: EAGLE-704
> URL: https://issues.apache.org/jira/browse/EAGLE-704
> Project: Eagle
> Issue Type: Improvement
> Reporter: Zhao, Qingwen
> Assignee: Zhao, Qingwen
>
> This ticket will update SparkHistoryJobAppProvider.xml to integrate with
> application framework. In this way, users can manage this topology on the web
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
