Github user kwmonroe commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/208#discussion_r116296932
--- Diff: bigtop-deploy/puppet/modules/spark/templates/spark-defaults.conf
---
@@ -13,7 +13,15 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+<% if @master_url -%>
spark.master <%= @master_url %>
+<% else -%>
+<% if scope['deploy::roles'].include? 'spark-standalone' -%>
--- End diff --
@evans-ye sorry i didn't catch this earlier (i still get confused with
roles vs components sometimes). However, I think this condition isn't quite
right. The **component** is called `spark-standalone`, while the **role** that
should be handled here is `spark-master` or `spark-worker`.
I checked this a few different ways...
- Using components on a single head node:
- `['deploy::roles']` is `[spark-master, spark-worker]`
- site.yaml:
```
...
bigtop::hadoop_head_node: my.fqdn
hadoop_cluster_node::cluster_components:
- spark-standalone
bigtop::roles_enabled: false
...
```
- Using components where the worker is not on the head node
- `['deploy::roles']` is `[spark-worker]`
- site.yaml:
```
...
bigtop::hadoop_head_node: NOT.my.fqdn
hadoop_cluster_node::cluster_components:
- spark-standalone
bigtop::roles_enabled: false
...
```
- Using roles
- `['deploy::roles']` is `[spark-master, spark-worker]`
- site.yaml:
```
...
bigtop::roles:
- spark-master
- spark-worker
bigtop::roles_enabled: true
...
```
So I think the proper condition here (and in `spark-env.sh` below) is to
check if `scope['deploy::roles']` includes either `spark-master` or
`spark-worker` instead of checking for `spark-standalone`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
