I planned to stay out of this, but
-1 since I am totally confused about the scope of this project at this point.
So are we talking about a generic telemetry framework for Qt applications or
about watching Qt Creator users specifically?
Tino started by making the claim that this is a generic library and then kept
listing Qt Creator specific integrations. You focus entirely onto the generic
part, leaving out creator completely.
What exactly are we talking about here?
My understanding at this point is that this is meant to be a telemetry
framework for Qt applications and first integration point that has been in
discussions is Qt Creator. I intentionally left Qt Creator out as I don’t feel
comfortable talking what they need/don’t need. So instead I concentrated on
what I see Qt 3D Studio needs as I feel much more comfortable to talk about
that based on our experiences, our engagements with customers and the usability
and UI work we’ve done so far in Qt 3D Studio project.
I’m always in favor of iterative development as you very very rarely understand
the scope and needs at the beginning. So I understand the need to have a
“reference integration to try out” even if we talk about aiming towards generic
telemetry plugin. And it seems quite obvious from these discussions that we
don’t yet know ourselves a lot about how to collect data, what type of data is
to be collected and how to process it. We seem to be a bit divided in to
individuals thinking this is inherently bad and morally dubious. And
individuals (like myself) who see this as great idea to be able to get more
data on how people use our tools. Not just from big customers and people who
come to our events. But also from people out there in universities, small
companies, startups, the people we rarely meet in person.
Once we have understanding on what exactly are we after, then it would be good
to move towards a generic Qt framework prototype. Maybe integrate against a
known and trusted third party framework, add features to the API iteratively
based on learnings and iterations. Then move on to publish this as tech preview
and while we investigate integrating against another 3rd party framework so
that we end up with a truly generic, Qt-like wrapper for specific functionality.
A Qt creator spyware plug-in (which does more than what we discuss here I
hope:-) would be a matter of a couple of weeks to do, putting a generic
framework into place with all the bits and pieces for that to be actually
useful in a wide list of possible contexts is a very different beast.
I object to using “Spyware” term in this context. Spyware is SW that does
things behind your back. Siphons contact information, user account info etc.
without telling you. We’re NOT talking about such use cases here! I don’t think
anyone in The Qt Company wants to do such things. From what I hear the
intention is to track certain things in an open, transparent manner, respecting
the communitys clear wish to keep things in the open and for the benefit of our
This is great initiative and very much the way today's app and application
industry works. UX studies performed by UX experts have been minimized and
targeted for specific (usually new/experimental) features or upcoming new
software (like we did with Qt 3D Studio back in last spring). And the mass
information on "how do our users use the SW? do they find the stuff we've put
in there? how often they hit a wall in doing sequence X? how many crashes do
they experience when doing Y?" is collected via automated telemetry. It is
great as it brings data from the actual user in their actual work and you can
then use that to concentrate on functionality that really matters to your
users. Making stuff they repeat hundred times a week easier and faster, making
them more productive.
I see value in this approach when you can do lots of small releases fast. So
you can do evaluate the effect of small changes by doing one change to evaluate
per release and measure how that effects usage.
We can not do more than two releases per year in Qt. Is this approach even
applicable to us?
Qt 3D Studio plans to do 4 releases per year, so for us this is definitely
applicable. Qt Creator does also more than 2 releases a year so perhaps this
could be useful for them as well, but as I said. I’m not comfortable talking on
their behalf on this.
I want to also point out that answering any of the questions you used as an
example require *way* more information than I am even remotely comfortable to
This is where we definitely differ as individuals. I use software from Apple,
Adobe etc. and I’m 100% fine giving them information on my usage statistics as
it means the SW might get better over time for me. If e.g. Blender would some
day ask me “are you ok for us to track your usage of the SW” I’d be 100% ok to
do that as I see that it definitely could improve a lot from observing users. I
see the poitive side of this type of usability and user data collection. So
taking this in to account, perhaps our solution should somehow enforce
possibility to opt-in/opt-out to accommodate for both you and me?
I agree that having a general framework has value to some customers.
Good to hear we’re in agreement on some level at least! :)
Such a framework would need to be integrated into big solutions like Google
analytics or something similar so the users can actually evaluate the collected
data and cross-reference that to other data they collect. Just dumping data
into some server somewhere does not help anybody.
Will the server-side code be part of this project by the way?
What about evaluation of the collected data? Is that in scope for this project,
All these are good questions that I’d rather not try to answer in a
meeting/email thread/chat thread, but rather prototype hands on iteratively.
Just may way of approaching stuff and having experienced so many frustrating
analysis paralysis moments and moments where theoretical discussions have
proven something “absolutely impossible” only for competition then coming out
with that exact feature a few moments later. Just try to do it and see how it
goes, collect experience and modify your target setting based on that. Listen
to customers, community and you shouldn’t go horribly wrong.
Let us do some usability studies, let us talk to users, let us do surveys,
let's evaluate all the data users provide to us on the mailing list, the bug
tracker, the forums, stack overflow and in lots of other places. We do have a
very active and supportive community of users and customers, let us use the
feedback they provide already!
Yup, we’ve done usability studies with Qt 3D Studio, we’ve done talking with
SOME key customers and yes we’ve even fixed a lot of issues they’ve pointed out
in Qt 3D Studio. I’m watching our bug tracker all the time, it’s part of the
daily work. But let’s be honest.. we CAN’T talk to every user out there, only
the biggest ones, only the ones that happen to participate our events. Qt 3D
Studio is not talking to probably 95-97% of our users at the moment as we just
can’t. Even this mailing list doesn’t cover all of our users. And crawling
through all these email threads, no thanks, that is very very inefficient use
of time in my opinion. This forum is great for discussing ideas, formulating,
pointing out what needs to be considered (like in this case, it’s loud and
clear that privacy and opt-in are of great importance to our community for
atelemetry feature). Data collection would (when implemented correctly)
automate the input collection and bring in data from all over the user base,
that is why I see great value in it.
Collecting some semi-random data from a self-selected group of users and
dumping that onto a server somewhere is next to useless compared to all the
other options we have already. Even in an ideal situation (which we do not have
within Qt!), metrics are not comparable to a real user study where you actually
watch users doing their thing.
Perhaps first version will start with a set of data that might turn out to be
”semi random” and “not so useful for understanding users”. But I believe in
learning, iterating and improvement. Next iteration we should improve and
collect less random set of data that also then helps us in understanding our
users better. And so on and so on.
Yes I agree, metrics are ADDITIONAL tool. Not replacement for anything.
Visiting users and talking with them does sometime open a whole new point of
view in to what you are doing. Those can provide revolutionary ideas for you.
That isn’t replaced with metrics. But metrics are great for iterative
development, as you point out above. Honing and fine tuning the set of
functionality you already have in your application.
Qt-creator mailing list