bbeaudreault commented on code in PR #4788: URL: https://github.com/apache/hbase/pull/4788#discussion_r1096970501
########## hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskGroup.java: ########## @@ -0,0 +1,114 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.monitoring; + +import java.util.Collection; +import java.util.Collections; +import java.util.concurrent.ConcurrentLinkedDeque; +import org.apache.yetus.audience.InterfaceAudience; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * The {@link TaskGroup} can be seen as a big {@link MonitoredTask}, which contains a list of sub + * monitored tasks. The monitored tasks in the group are still be managed by the + * {@link TaskMonitor}, but whether to clear/expire the monitored tasks in a task group is optional. + * Since the monitored task already has journals, which mark the phases in a task, we still also + * need a task group to monitor a big task/process because the journals in a task is serial but the + * tasks in the task group can be parallel, then we have more flexible ability to monitor the + * process. Grouping the tasks is not strictly necessary but it is cleaner for presentation to + * operators. We might want to display the tasks in a group in a list view where each task can be + * collapsed (probably by default) or expanded. + */ [email protected] +public class TaskGroup extends MonitoredTaskImpl { + private static final Logger LOG = LoggerFactory.getLogger(TaskGroup.class); + + private final ConcurrentLinkedDeque<MonitoredTask> tasks = new ConcurrentLinkedDeque<>(); + + /** Whether to clear/expire the monitored tasks by {@link TaskMonitor} */ + private final boolean ignoreClearStatus; + + public TaskGroup(boolean ignoreClearStatus, String description) { + super(false, description); + this.ignoreClearStatus = ignoreClearStatus; + } + + public TaskGroup(String description) { + this(false, description); + } + + public static TaskGroup createTaskGroup(boolean ignoreClearStatus, String description) { + return new TaskGroup(ignoreClearStatus, description); Review Comment: I think when creating a task group (which extends MonitoredTaskImpl), we should probably make the TaskGroup itself be monitored in the TaskMonitor. Otherwise there will be no indication in the main master UI that startup is in progress. I think a good user experience here is -- user loads master UI and see's "Master startup" task in main task monitor. This prompts them to load the "Startup Progress" page. In order to do that, TaskGroup itself must be a monitored task. I'd recommend following a delegate pattern here: ```java public static TaskGroup createTaskGroup(boolean ignoreClearStatus, String description) { MonitoredTask delegate = TaskMonitor.get().createStatus(description, false, true) return new TaskGroup(ignoreClearStatus, delegate); } @Override public synchronized void markComplete(String msg) { delegate.setState(State.COMPLETE); delegate.setStatus(msg); if (tasks.getLast() != null) { tasks.getLast().markComplete(msg); } } ``` ########## hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java: ########## @@ -2420,8 +2431,6 @@ private void startActiveMasterManager(int infoPort) throws KeeperException { } else { abort("Unhandled exception. Starting shutdown.", t); } - } finally { - status.cleanup(); Review Comment: Thanks that clarifies. That's what we want -- retain history for process lifetime so it's always visible on startupProgress UI. So removing config is good IMO, thanks. I imagine the parent task will be removed from main master UI since we markComplete at end. ########## hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskGroup.java: ########## @@ -0,0 +1,114 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.monitoring; + +import java.util.Collection; +import java.util.Collections; +import java.util.concurrent.ConcurrentLinkedDeque; +import org.apache.yetus.audience.InterfaceAudience; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * The {@link TaskGroup} can be seen as a big {@link MonitoredTask}, which contains a list of sub + * monitored tasks. The monitored tasks in the group are still be managed by the + * {@link TaskMonitor}, but whether to clear/expire the monitored tasks in a task group is optional. + * Since the monitored task already has journals, which mark the phases in a task, we still also + * need a task group to monitor a big task/process because the journals in a task is serial but the + * tasks in the task group can be parallel, then we have more flexible ability to monitor the + * process. Grouping the tasks is not strictly necessary but it is cleaner for presentation to + * operators. We might want to display the tasks in a group in a list view where each task can be + * collapsed (probably by default) or expanded. + */ [email protected] +public class TaskGroup extends MonitoredTaskImpl { + private static final Logger LOG = LoggerFactory.getLogger(TaskGroup.class); + + private final ConcurrentLinkedDeque<MonitoredTask> tasks = new ConcurrentLinkedDeque<>(); Review Comment: One concern is TaskMonitor's `tasks` is a CircularFifoQueue with a max size, but a TaskGroup's ConcurrentLinkedDeque can seemingly have unlimited tasks. I wonder if we need a limit on these? ########## hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskGroup.java: ########## @@ -0,0 +1,114 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.monitoring; + +import java.util.Collection; +import java.util.Collections; +import java.util.concurrent.ConcurrentLinkedDeque; +import org.apache.yetus.audience.InterfaceAudience; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * The {@link TaskGroup} can be seen as a big {@link MonitoredTask}, which contains a list of sub + * monitored tasks. The monitored tasks in the group are still be managed by the + * {@link TaskMonitor}, but whether to clear/expire the monitored tasks in a task group is optional. + * Since the monitored task already has journals, which mark the phases in a task, we still also + * need a task group to monitor a big task/process because the journals in a task is serial but the + * tasks in the task group can be parallel, then we have more flexible ability to monitor the + * process. Grouping the tasks is not strictly necessary but it is cleaner for presentation to + * operators. We might want to display the tasks in a group in a list view where each task can be + * collapsed (probably by default) or expanded. + */ [email protected] +public class TaskGroup extends MonitoredTaskImpl { + private static final Logger LOG = LoggerFactory.getLogger(TaskGroup.class); + + private final ConcurrentLinkedDeque<MonitoredTask> tasks = new ConcurrentLinkedDeque<>(); + + /** Whether to clear/expire the monitored tasks by {@link TaskMonitor} */ + private final boolean ignoreClearStatus; Review Comment: Reading TaskMonitor.createStatus I think it would be clearer to call this `showSubTasksInTaskMonitor` (this would involve inverting the boolean, so you'd want to pass false instead of true). Basically, we're making it so when we call `addTask` the tasks do or don't get shown in task monitor, so that's a better name. You're currently passing `true` for `ignoreClearStatus`, which means you dont want all the sub-tasks of the "startupTaskGroup" to show up in the task monitor. If we rename the variable to `showSubTasksInTaskMonitor` you'd instead pass in `false`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
