----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/60942/#review182262 -----------------------------------------------------------
The failing test case is `TestDiffCommand.test_cron_diff`. It fails to compare two `frozendict`s containing identical resources. We can reproduce this behaviour in isolation: ``` $ ./pants repl src/main/python/apache/aurora/client/ # Let's quickly verify that frozensets works as expected >>> [1,2,3] == [1,2,3] True >>> frozenset([1,2,3]) == frozenset([1,2,3]) True # Good! # Now let's do the same with a thrit Resource objects >>> [Resource(numCpus=1), Resource(ramMb=2)] == [Resource(numCpus=1), >>> Resource(ramMb=2)] True >>> frozenset([Resource(numCpus=1), Resource(ramMb=2)]) == >>> frozenset([Resource(numCpus=1), Resource(ramMb=2)]) False # oh!... ``` Turns out `Resource` implements `__eq__` and `__ne__` but *not* `__hash__`. The [Python docs](https://docs.python.org/2/reference/datamodel.html#object.__hash__) on this: ``` If a class [...] defines __cmp__() or __eq__() but not __hash__(), its instances will not be usable in hashed collections. ``` This points to a problem with Thrift. Indeed there is a related issue https://issues.apache.org/jira/browse/THRIFT-162. It is fixed in 0.10.0 but we are still on 0.9.3. The tested class deals with leaky behaviour of the underlying `DiffFormater` that modifies tasks. If we move the necessary `deepcopy` into the `DiffFormatter`, we can prevent this behaviour from leaking out. This will also prevent us from hitting the faulty Thrift behaviour, thus fixing the test. The necessary diff can be something like: ``` diff --git a/src/main/python/apache/aurora/client/cli/diff_formatter.py b/src/main/python/apache/aurora/client/cli/diff_formatter.py index 8804612..2f66e29 100644 --- a/src/main/python/apache/aurora/client/cli/diff_formatter.py +++ b/src/main/python/apache/aurora/client/cli/diff_formatter.py @@ -126,6 +126,8 @@ class DiffFormatter(object): format_ranges(r for r in chain.from_iterable(s.instances for s in summaries))) def diff_no_update_details(self, local_tasks): + # Deepcopy is important here as tasks will be modified for printing. + local_tasks = [deepcopy(t) for t in local_tasks] api = self.context.get_api(self.cluster) resp = api.query(api.build_query(self.role, self.name, env=self.env, statuses=ACTIVE_STATES)) self.context.log_response_and_raise( diff --git a/src/main/python/apache/aurora/client/cli/jobs.py b/src/main/python/apache/aurora/client/cli/jobs.py index b79ae56..d6d2939 100644 --- a/src/main/python/apache/aurora/client/cli/jobs.py +++ b/src/main/python/apache/aurora/client/cli/jobs.py @@ -201,10 +201,7 @@ class DiffCommand(Verb): err_code=EXIT_INVALID_CONFIGURATION, err_msg="Error loading configuration") local_task = resp.result.populateJobResult.taskConfig - # Deepcopy is important here as tasks will be modified for printing. - local_tasks = [ - deepcopy(local_task) for _ in range(config.instances()) - ] + local_tasks = [local_task for _ in range(config.instances())] instances = (None if context.options.instance_spec.instance == ALL_INSTANCES else context.options.instance_spec.instance) formatter = DiffFormatter(context, config, cluster, role, env, name) ``` What do you think? - Stephan Erb On Aug. 4, 2017, 4:04 p.m., Nicolás Donatucci wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/60942/ > ----------------------------------------------------------- > > (Updated Aug. 4, 2017, 4:04 p.m.) > > > Review request for Aurora, Joshua Cohen, Stephan Erb, and Zameer Manji. > > > Bugs: AURORA-1707 > https://issues.apache.org/jira/browse/AURORA-1707 > > > Repository: aurora > > > Description > ------- > > Removed task level resource fields from the DB and the thrift-API. > To do this, a new DB migration was added. When upgrading, it just drops the > task level resource fields. When downgrading, it creates the fields again and > populates them with information from the task_resource table. > > IMPORTANT: One of the python client tests is failing (test_cron_diff). This > is not serious, I think it is a problem with the ordering of the elements of > Resources (had similar problems with other python client tests that instead > of comparing the Resources as a set, compared them as lists and thus order > mattered). Nevertheless, I could not fully understand the code of that test. > I was hoping someone could give me a hand with that. > But then again, it is a smaller issue and so the patch can start being > reviewed. > > Issue Related: AURORA-1707 > > > Diffs > ----- > > api/src/main/thrift/org/apache/aurora/gen/api.thrift > 3749531b5412d7ca217736aa85eed8e6606225ad > src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java > 186fa1b3a4780c0536fb486d50a33133258110cd > src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java > d2eb6aa6e4a155b2d28debab2ca10dfc76d57213 > src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java > cda55c55680a19ed421299a8949299b21949787b > > src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java > af106a8a9ee8c14122e98bcc0ec44b616f21d61f > > src/main/java/org/apache/aurora/scheduler/storage/db/migration/V011_DropResourceFields.java > PRE-CREATION > > src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java > 138cd5316adc73eed24fc7accc53885dd5d5bee5 > src/main/python/apache/aurora/client/cli/diff_formatter.py > 78717774aa3fbaf83a5fb850bc9f9f4a4038d70f > > src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml > 5422183e4a1fe122fc0e1aa871aa75ae102e322d > src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql > 7a86f47af67adb3e488381d30ddf424549deefbc > src/test/java/org/apache/aurora/scheduler/http/TestUtils.java > 689482c9f6c49bcca781834566edeb975d2f3af2 > > src/test/java/org/apache/aurora/scheduler/storage/db/RowGarbageCollectorTest.java > caaba9b6dff46ff0b037759f1c817a321ae15ee4 > src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java > 43e32eede27bbf26363a3fd1ca34ffe6f8c01a73 > > src/test/java/org/apache/aurora/scheduler/updater/UpdateAgentReserverImplTest.java > 1bc2a778ad3f1543a055023f9ec3fe9e4a9523e3 > src/test/python/apache/aurora/client/cli/test_status.py > b0b7f96d8148f0dd1f6f45d1c5c6f830cabcfd5d > src/test/python/apache/aurora/client/cli/util.py > 43db828ca1cccd91c88f016b4c994fef33182fbf > src/test/python/apache/aurora/config/test_resources.py > 25a20f0b702189744a26b85053db34c37ff5b03c > > > Diff: https://reviews.apache.org/r/60942/diff/3/ > > > Testing > ------- > > src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh > > 1 - Created a new VM with a fresh Aurora 0.18.0, ran the tests so that the db > would have something. Then upgraded to patched version and ran the tests > again. Everything worked and task level resource fields were no longer there. > 2 - Created a new VM with a fresh patched version, ran the tests so that the > db would have something. Then downgraded to 0.18.0 and ran the tests again. > Everything worked and task level resources were there, with the correct > backfilled values. > > > Thanks, > > Nicolás Donatucci > >
