[issue45672] Mutation tests results of typing.py

[Oleg Iarygin]

Oleg Iarygin  added the comment:

> because this modification does not affect behavior

Unfortunately, this one (that I missed by not reading the report thoroughly) 
makes the framework totally unsuitable for CI.

However, looking at such false positives allows to muse about reasons behind 
design choises. For example (in a snippet you cited), why Python runtime stores 
bases as a tuple, not as a linked list. Anyway, application of the method 
resolution order scans them sequentially, they are not of a primitive type (so 
tuples give no optimization), a call of bases.index() may be replaced with a 
reference property inside class internals, etc.

Though, I perfectly understand that such treatment leads to immense amount of 
refactoring with no performance gain, probably counterweighted with removal of 
couple of internal packages that will become unused after it. That's why I 
wrote about "a task of the OP themselves".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45672] Mutation tests results of typing.py

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

The tests are passed because this modification does not affect behavior, it 
just makes the code slightly less efficient. Replacing i+1 with i just adds one 
iteration:

b = bases[i]  # == self
if isinstance(b, _BaseGenericAlias) and b is not self:
return ()

Since b is self, the condition is always false.

It is impossible to catch this change in tests because both codes are 
equivalent.

--
resolution:  -> not a bug

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45672] Mutation tests results of typing.py

[Oleg Iarygin]

Oleg Iarygin  added the comment:

> What is the problem actually?

I guess, Lib/test/test_typing.py has gaps in test coverage. The report provided 
by the OP is a list of random modifications that corrupt logic of Lib/typing.py 
but still pass all test cases.

Mutation testing is validation of tests themselves. A mutation framework takes 
code that the validated test actually tests, makes some random but 
syntactically correct change, and runs the test that should fail some assertion 
here. If so, the code is rolled back and another mutation attempt is done. If 
the validated test catches all thousands of such attempts, we can consider it a 
reliable safety net for any refactoring. Otherwise, we open the report and 
start to supplement tests and probably package documentation to cover missed 
scenarios.

Actually, something like cosmic-ray may be added into a CI pipeline, but I 
suspect that preliminary fixing of test coverage will be a task of the OP 
themselves. There are no people with enough free time to sit down and sort all 
tests out, being overwhelmed with new language features, bug fixing and already 
pending epics of, for example, closing Python internals from extension modules 
to stop constantly breaking backward compatibility, or fixing startup modules 
to properly freeze them for speedup of Python launching.

--
nosy: +arhadthedev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45672] Mutation tests results of typing.py

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

What is the problem actually?

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45672] Mutation tests results of typing.py

[Nikita Sobolev]

Nikita Sobolev  added the comment:

I can send a PR to have 100% mutation coverage for `typing.py`.
I think that a single test can do that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45672] Mutation tests results of typing.py

[Nikita Sobolev]

New submission from Nikita Sobolev :

I've decided to test `typing.py` with `cosmic-ray` mutation testing framework. 
It identified 3 potential problems.

Config:

```
[cosmic-ray]
module-path = "Lib/typing.py"
timeout = 15.0
excluded-modules = []
test-command = "./python.exe -m test -v test_typing --failfast"

[cosmic-ray.distributor]
name = "local"
```

Repro steps:
0. pip install cosmic-ray
1. Copy config above as `typing.toml`
2. cosmic-ray init typing.toml typing.sqlite
3. cosmic-ray exec tutorial.toml tutorial.sqlite 

Survived mutants:

1. ```
--- mutation diff ---
--- aLib/typing.py
+++ bLib/typing.py
@@ -1103,7 +1103,7 @@
 if Protocol in bases:
 return ()
 i = bases.index(self)
-for b in bases[i+1:]:
+for b in bases[i*1:]:
 if isinstance(b, _BaseGenericAlias) and b is not self:
 return ()
 return (self.__origin__,)
```

2. ```
--- mutation diff ---
--- aLib/typing.py
+++ bLib/typing.py
@@ -1103,7 +1103,7 @@
 if Protocol in bases:
 return ()
 i = bases.index(self)
-for b in bases[i+1:]:
+for b in bases[i//1:]:
 if isinstance(b, _BaseGenericAlias) and b is not self:
 return ()
 return (self.__origin__,)
```

3. ```
--- mutation diff ---
--- aLib/typing.py
+++ bLib/typing.py
@@ -1103,7 +1103,7 @@
 if Protocol in bases:
 return ()
 i = bases.index(self)
-for b in bases[i+1:]:
+for b in bases[i**1:]:
 if isinstance(b, _BaseGenericAlias) and b is not self:
 return ()
 return (self.__origin__,)
```

I've attached the full report to this issue if it is interesting to anyone: 
https://gist.github.com/sobolevn/79f1729dcf0b8c4b2a9bcee67027673a

--
components: Tests
messages: 405319
nosy: sobolevn
priority: normal
severity: normal
status: open
title: Mutation tests results of typing.py
type: behavior
versions: Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com