Hello, Slava, Ilya, Denis.
Thanks for joining this discussion!
> - visor (which is deprecated)
Who deprecated visor and when?
Maybe I miss something?
> - web-console (to be honest, I don't quite understand the status of this tool)
+1.
> I am not against the new tool, I just want to understand the motivation to
> not improve the existing sub-projects.
It's true that, for now, Ignite doesn't have "tool strategy"
I think it's a big issue from the user's point of view.
We should solve it in the nearest time.
Feel free to start this activity.
> - new ignite-profiling (which is a monitoring tool as well, judging by the
> provided link [1] )
The general idea is the following:
1. We should have some profiling mechanism that will generate a node-local
event log
2. We should have a tool that can export events to some third-party system.
This system can be an Elastic Search(Kibana) or Ignite performance report or
Kafka log, whatever.
3. Ignite performance report, in the first release, should be a "static" tool.
This means we take static logs(that is not rewritten in the analysis time)
and feed them in the script(or tool or control.sh, whatever)
The script produces static report that can be used for overall performance
analysis.
The primary users of this report is a developer of Ignite based applications
and performance engineers.
Ilya,
> You need to throw in control.sh also, which does some kind of statistics too,
> such as idle_verify.
Please, clarify your idea:
We should use some info from control.sh to the report?
The report should be generated by some control.sh subcommand?
Denis,
> Speaking of the probes/statistics collection approach, is it supposed to
> reuse tracing capabilities that are to be added as part of IEP-35?
For now, we don't have any results of tracing development available in Apache
Ignite.
Hopefully, we got some in a couple of weeks.
After it, we can start a discussion of how to merge two improvements.
> 24 апр. 2020 г., в 20:32, Denis Magda <dma...@apache.org> написал(а):
>
>>
>> Tracing is more deeply takes statistics. If it will be possible, I'm for
>> reuse.
>
>
> Looks like we need to sync up on these activities/initiatives to ensure we
> don't do a duplicate job. If you think a separate discussion is necessary
> let's kick it off.
>
> -
> Denis
>
>
> On Fri, Apr 24, 2020 at 9:18 AM Nikita Amelchev <nsamelc...@gmail.com>
> wrote:
>
>> Denis, Ilya,
>>
>> I will try to integrate profiling functionality into control.sh utility.
>>
>>> Speaking of the probes/statistics collection approach, is it supposed to
>>> reuse tracing capabilities that are to be added as part of IEP-35?
>> Tracing is more deeply takes statistics. If it will be possible, I'm for
>> reuse.
>>
>> пт, 24 апр. 2020 г. в 18:59, Ilya Kasnacheev <ilya.kasnach...@gmail.com>:
>>>
>>> Hello!
>>>
>>> I suggest that it's one of the places where it could be put instead of
>>> adding a new tool.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пт, 24 апр. 2020 г. в 18:56, Nikita Amelchev <nsamelc...@gmail.com>:
>>>
>>>> Ilya,
>>>>
>>>> You suggest using control.sh to build the report?
>>>>
>>>> пт, 24 апр. 2020 г. в 18:20, Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com>:
>>>>>
>>>>> Hello!
>>>>>
>>>>> You need to throw in control.sh also, which does some kind of
>> statistics
>>>>> too, such as idle_verify.
>>>>>
>>>>> Regards,
>>>>> --
>>>>> Ilya Kasnacheev
>>>>>
>>>>>
>>>>> пт, 24 апр. 2020 г. в 18:06, Вячеслав Коптилин <
>> slava.kopti...@gmail.com
>>>>> :
>>>>>
>>>>>> Hello Nikita,
>>>>>>
>>>>>> Perhaps, I am missing something...
>>>>>> Apache Ignite already has a web-console tool. Do we want to
>> improve the
>>>>>> existing tool instead of creating a new one?
>>>>>> It seems to me, this can be confusing for users.
>>>>>> - visor (which is deprecated)
>>>>>> - web-console (to be honest, I don't quite understand the status
>> of
>>>> this
>>>>>> tool)
>>>>>> - new ignite-profiling (which is a monitoring tool as well,
>> judging
>>>> by the
>>>>>> provided link [1] )
>>>>>>
>>>>>> I am not against the new tool, I just want to understand the
>>>> motivation to
>>>>>> not improve the existing sub-projects.
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>>>>>>
>>>>>> Thanks,
>>>>>> S.
>>>>>>
>>>>>> пт, 24 апр. 2020 г. в 14:58, Nikita Amelchev <nsamelc...@gmail.com
>>> :
>>>>>>
>>>>>>> Hi, Igniters.
>>>>>>>
>>>>>>> I'm working on cluster profiling and the tool for creating a
>>>>>>> performance report. [1] I have prepared PoC based on performance
>>>>>>> logging to a separate category of Ignite log. The report
>> contains:
>>>>>>>
>>>>>>> - Cache operations and its distribution by types [2]
>>>>>>> - Transactions and histogram of durations [3]
>>>>>>> - SQL and Scan query statistics, top of slowest queries, logical
>> and
>>>>>>> physical reads by query [4]
>>>>>>> - Compute statistics, top of slowest tasks and their jobs [5]
>>>>>>> Soon I will add:
>>>>>>> - Topology and Ignite versions info
>>>>>>> - Client ID in case of operations from clients
>>>>>>>
>>>>>>> For now, I'm developing a binary logging format to reduce the
>> effect
>>>>>>> on performance. I'll try to reuse Ignite mechanisms.
>>>>>>>
>>>>>>> I would like to hear suggestions for the profiling and the
>>>> performance
>>>>>>> report.
>>>>>>>
>>>>>>> [1]
>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
>>>>>>> [2]
>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
>>>>>>> [3]
>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
>>>>>>> [4]
>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
>>>>>>> [5]
>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/152112279/p5.png
>>>>>>>
>>>>>>> --
>>>>>>> Best wishes,
>>>>>>> Amelchev Nikita
>>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best wishes,
>>>> Amelchev Nikita
>>>>
>>
>>
>>
>> --
>> Best wishes,
>> Amelchev Nikita
>>
