On 21/10/17 16:14, [email protected] wrote:
I think Rob's suggested message is pretty reasonable. I think what we
can do in this situation is to help open a larger conversation about
what is fair and what is desirable for this kind of research.
ajs6f
No problem with that.
I did some editting on it to emphasis the Code of practice, and away
from the incident:
---------------------------
On the Responsible Disclosure of Benchmarking Results
The Apache Jena PMC would like to suggest to the benchmarking community
that they adopt a code of practice that will improve benchmarking
semantic web systems by focusing on the contribution to the literature
and away from transient details.
The PMC was recently made aware of a paper scheduled to be presented at
the Workshop on Benchmarking Linked Data (BLINK) at ISWC 2017. The paper
in question provides a new benchmark for property paths.
We are disappointed that the authors identified a deficiency in our
project's implementation about which they made no attempt to contact us.
Indeed, our public JIRA has independently reported tickets that are
relevant. A fix has been available for some time.
We are by no means the only project affected, other correctness and
performance issues across several projects were identified in this
paper.
We wish to raise a general issue we and others in our community perceive
across this field of research.
Investigation and analysis of algorithms and designs should not be based
on engineering details.
As an open source project maintained by volunteers we rely upon the
wider community, both in industry and academia, to bring issues to our
attention in a timely fashion.
If this was a security flaw the expected standard practice would be to
responsibly disclose the issue to the affected projects and work with
those projects to address the issue.
Many of us have a background in scientific research and appreciate that
research often happens on tight timelines but simply sending a short to
a project about an identified issue is not unreasonable.
We would like to suggest to the benchmarking community that they adopt a
code of good practice that encourages feedback leading to high-quality
results for the long term benefit of other researchers.
If you could raise the topic of responsible disclosure of issues
identified during the course of your workshop that would be much
appreciated.
Regards,
The Apache Jena PMC
