It sounds like using a multi_hashmap for now allows you to clean up the code and avoid some bugs, without changing the existing behavior.
I agree that we would want a deprecation period if we changed the behavior. It would also be unfortunate if we said we were dis-allowing duplicate task ids but only catch some of the manifestations. — *Joris Van Remoortere* Mesosphere On Mon, Dec 12, 2016 at 7:56 AM, Neil Conway <neil.con...@gmail.com> wrote: > Hi Joris, > > Fair point: I didn't deliberately set out to change the behavior for > duplicate task IDs. Rather, it was a consequence of switching from > boost::circular_buffer to using a hashmap for managing completed > tasks. Using a hashmap has a few minor advantages [1], but we can > certainly continue using circular_buffer (or a multi-hashmap) if we > want to keep the current behavior. > > I think we have the following options: > > (1) Keep the current behavior: reusing task IDs is discouraged but > supported. > > (2) Per Alex's suggestion, we can say that frameworks are no longer > allowed to reuse task IDs. Because the master only keeps a > limited-size cache of completed tasks (which is not preserved across > master restart or failover), we wouldn't be able to reject all > situations in which frameworks attempt to reuse task IDs. > > If we pursue #2, we might need a deprecation period or master > capability to give framework authors some time to migrate. > > For the moment, I'll avoid changing the behavior for duplicate task > IDs; I've opened https://issues.apache.org/jira/browse/MESOS-6779 to > track this issue. If you have an opinion in this change, please > weigh-in, either on this thread or on JIRA. > > Neil > > [1] Specifically, making the management of completed and unreachable > tasks more symmetric and avoiding some bugs/UBI in > boost::circular_buffer. O(1) lookup of completed tasks might be useful > in the future but isn't used right now. > > On Fri, Dec 9, 2016 at 2:13 PM, Joris Van Remoortere > <jo...@mesosphere.io> wrote: > > Hey Neil, > > > > I concur that using duplicate task IDs is bad practice and asking for > > trouble. > > > > Could you please clarify *why* you want to use a hashmap? Is your goal to > > remove duplicate task IDs or is this just a side-effect and you have a > > different reason (e.g. performance) for using a hashmap? > > > > I'm wondering why a multi-hashmap is not sufficient. This would be clear > if > > you were explicitly *trying* to get rid of duplicates of course :-) > > > > Thanks, > > Joris > > > > — > > *Joris Van Remoortere* > > Mesosphere > > > > On Fri, Dec 9, 2016 at 7:08 AM, Neil Conway <neil.con...@gmail.com> > wrote: > > > >> Folks, > >> > >> The master stores a cache of metadata about recently completed tasks; > >> for example, this information can be accessed via the "/tasks" HTTP > >> endpoint or the "GET_TASKS" call in the new Operator API. > >> > >> The master currently stores this metadata using a list; this means > >> that duplicate task IDs are permitted. We're considering [1] changing > >> this to use a hashmap instead. Using a hashmap would mean that > >> duplicate task IDs would be discarded: if two completed tasks have the > >> same task ID, only the metadata for the most recently completed task > >> would be retained by the master. > >> > >> If this behavior change would cause problems for your framework or > >> other software that relies on Mesos, please let me know. > >> > >> (Note that if you do have two completed tasks with the same ID, you'd > >> need an unambiguous way to tell them apart. As a recommendation, I > >> would strongly encourage framework authors to never reuse task IDs.) > >> > >> Neil > >> > >> [1] https://reviews.apache.org/r/54179/ > >> >
