Re: Duplicate task IDs

2016-12-12 Thread Neil Conway
replacement), this would require adding a new data structure, BoundedMultiHashMap (https://reviews.apache.org/r/54178/). That seems like overkill to me, for now. > It would also be unfortunate if we said we were dis-allowing duplicate task > ids but only catch some of the manifestations. Defi

Re: Duplicate task IDs

2016-12-12 Thread Joris Van Remoortere
It sounds like using a multi_hashmap for now allows you to clean up the code and avoid some bugs, without changing the existing behavior. I agree that we would want a deprecation period if we changed the behavior. It would also be unfortunate if we said we were dis-allowing duplicate task ids

Re: Duplicate task IDs

2016-12-12 Thread Neil Conway
Hi Joris, Fair point: I didn't deliberately set out to change the behavior for duplicate task IDs. Rather, it was a consequence of switching from boost::circular_buffer to using a hashmap for managing completed tasks. Using a hashmap has a few minor advantages [1], but we can certainly continue

Re: Duplicate task IDs

2016-12-11 Thread Alex Rukletsov
gt; Hey Neil, > > I concur that using duplicate task IDs is bad practice and asking for > trouble. > > Could you please clarify *why* you want to use a hashmap? Is your goal to > remove duplicate task IDs or is this just a side-effect and you have a > different reason (e.g. p

Re: Duplicate task IDs

2016-12-09 Thread Joris Van Remoortere
Hey Neil, I concur that using duplicate task IDs is bad practice and asking for trouble. Could you please clarify *why* you want to use a hashmap? Is your goal to remove duplicate task IDs or is this just a side-effect and you have a different reason (e.g. performance) for using a hashmap? I'm

Duplicate task IDs

2016-12-09 Thread Neil Conway
Folks, The master stores a cache of metadata about recently completed tasks; for example, this information can be accessed via the "/tasks" HTTP endpoint or the "GET_TASKS" call in the new Operator API. The master currently stores this metadata using a list; this means tha