I would start with NODES and NODE_ATTRIBUTES as the most simple thing. On Tue, Feb 13, 2018 at 4:10 AM, Denis Magda <dma...@apache.org> wrote:
> Alex P, sounds like a good plan for me. > > Vladimir, do you have any suggestions or corrections? > > — > Denis > > > On Feb 12, 2018, at 4:57 AM, Alex Plehanov <plehanov.a...@gmail.com> > wrote: > > > > The views engine and the first view are almost ready to merge (review > > comments are resolved). Which views should we take next? My proposal - > > NODES, NODE_ATTRIBUTES, NODE_METRICS, NODE_HOSTS and NODE_ADDRESSES, > since > > these views are clear and all topology data available on each node. > > Any objections? > > > > 2018-01-25 16:27 GMT+03:00 Alex Plehanov <plehanov.a...@gmail.com>: > > > >> Anton, Vladimir, I've made some fixes. There is only one view left and > >> it's renamed to 'IGNITE.LOCAL_TRANSACTIONS'. > >> > >> High level design of solution: > >> When IgniteH2Indexing is starting, it create and start > >> new GridH2SysViewProcessor, which create and register in H2 (via its own > >> table engine) all implementations of system views. Each system view > >> implementation extends base abstract class GridH2SysView. View > >> implementation describes columns, their types and indexes in constructor > >> and must override method getRows for data retrieval (this method called > by > >> H2-compatible table and index implementations for ignite system views). > >> Almost no fixes to existing parsing engine was made, except some places, > >> where GridH2Table instance was expected, but for system views there is > >> another class. > >> > >> New PR: [1]. Please have a look. > >> > >> [1] https://github.com/apache/ignite/pull/3433 > >> > >> 2018-01-24 19:12 GMT+03:00 Anton Vinogradov <avinogra...@gridgain.com>: > >> > >>> I've created IEP-13 [1] to cover all cases. > >>> Feel free to create issues. > >>> > >>> [1] > >>> https://cwiki.apache.org/confluence/pages/viewpage. > action?pageId=75962769 > >>> > >>> On Wed, Jan 24, 2018 at 6:10 PM, Vladimir Ozerov <voze...@gridgain.com > > > >>> wrote: > >>> > >>>> Let's start with a single and the most simple view, e.g. > >>>> LOCAL_TRANSACTIONS. We will review and merge it along with necessary > >>>> infrastructure. Then will handle the rest view in separate tickets and > >>>> separate focused discussions. > >>>> > >>>> On Wed, Jan 24, 2018 at 5:29 PM, Alex Plehanov < > plehanov.a...@gmail.com > >>>> > >>>> wrote: > >>>> > >>>>> 1) It’s not a principal point, I can change schema. The > >>>> INFORMATION_SCHEMA > >>>>> was used because it’s already exists and usually used for metadata > >>> tables > >>>>> and views. Your proposal is to use schema “IGNITE”, am I understand > >>> you > >>>>> right? BTW, for now, we can’t query another (H2) meta tables from the > >>>>> INFORMATION_SCHEMA, so, “Ignite system views” is only available views > >>> to > >>>>> query from this schema. > >>>>> 2) Exactly for this reason the IGNITE_INSTANCE view is useful: to > >>>> determine > >>>>> which node we are connected to. > >>>>> 3) As the first phase, in my opinion, local views will be enough. > >>>>> Performance and caching of distributed views should be discussed at > >>> next > >>>>> phases, when distributed views implementation will be planned. In > >>> current > >>>>> implementation I tried to use indexing for local views wherever it’s > >>>>> possible. > >>>>> 4) I don’t think, that JVM info is more critical information than, > for > >>>>> example, caches or nodes information. When authorization capabilities > >>>>> planned to implement? > >>>>> > >>>>> About local data: yes, we can rename all currently implemented views > >>> for > >>>>> the local node data as LOCAL_..., and create (someday) new whole > >>> cluster > >>>>> views (which use distributed requests) without prefix or, for > example, > >>>> with > >>>>> CLUSTER_ prefix. But some views can show all cluster information > using > >>>> only > >>>>> local node data, without distributed requests (for example > >>>>> IGNITE_NODE_METRICS, IGNITE_PART_ASSIGNMENT, IGNITE_PART_ALLOCATION, > >>>>> IGNITE_NODES, etc). Are they local or cluster views in this concept? > >>>> Which > >>>>> prefix should be used? And what about caches? Are they local or > >>> cluster? > >>>> On > >>>>> local node we can see cluster wide caches (replicated and > distributed) > >>>> and > >>>>> caches for current node only. Local caches list may differ from node > >>> to > >>>>> node. Which prefix should be used for this view? And one more, there > >>> is > >>>> no > >>>>> sense for some views to make them cluster wide (for example > >>>>> INGNITE_INSTANCE). Should we name it LOCAL_INSTANCE without creating > >>>>> INSTANCE view? > >>>>> > >>>>> So, next steps: split PR, change schema name (IGNITE?), change view > >>> name > >>>>> for caches (CACHES, LOCAL_CACHES?) > >>>>> > >>>>> > >>>>> 2018-01-24 13:03 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>: > >>>>> > >>>>>> Hi Alex, > >>>>>> > >>>>>> System views could be extremely valuable addition for Ignite. > >>> Ideally, > >>>>> user > >>>>>> should be able to monitor and manage state of the whole cluster > >>> with a > >>>>>> single SQL command line. We have plans to implement it for a very > >>> long > >>>>>> time. However, this is very sensitive task which should take a lot > >>> of > >>>>>> moving pieces in count, such as usability, consistency, performance, > >>>>>> security, etc.. > >>>>>> > >>>>>> Let me point several major concerns I see at the moment: > >>>>>> > >>>>>> 1) Usability: INFORMATION_SCHEMA > >>>>>> This schema is part of SQL ANSI standard. When creating system > >>> views, > >>>>> some > >>>>>> vendors prefer to store them in completely different predefined > >>> schema > >>>>>> (Oracle, MS SQL). Others prefer to keep them in INFORMATION_SCHEMA > >>>>>> directly. Both approaches could work. However, the latter breaks > >>>>> separation > >>>>>> of concerns - we store typical metadata near to possibly sensitive > >>>> system > >>>>>> data. Also it makes security management more complex - system data > >>> is > >>>>> very > >>>>>> sensitive, and now we cannot simply grant access > >>> INFORMATIONAL_SCHEMA > >>>> to > >>>>>> user. Instead, we have to grant that access on per-view basis. For > >>> this > >>>>>> reason my preference is to store system tables in separate schema, > >>> not > >>>> in > >>>>>> INFORMATION_SCHEMA > >>>>>> > >>>>>> 2) Consistency: local data > >>>>>> One of implemented view GridH2SysViewImplInstance. Normally SQL > >>> users > >>>>>> communicate with Ignite through JDBC/ODBC drivers. These drivers are > >>>>>> connected to a single node, typically client node. Moreover, we will > >>>>>> introduce high-availability feature when drivers were able to > >>> connect > >>>> to > >>>>>> any address from a predefined list. It renders this view useless, as > >>>> you > >>>>> do > >>>>>> not know which node you connected to. Also, local-only data cannot > >>> be > >>>>>> joined in general case - you will receive different results on > >>>> different > >>>>>> nodes. The same goes for transactions, JVM info, etc. > >>>>>> > >>>>>> 3) Performance > >>>>>> Suppose we fixed consistency of transactions and now this view shows > >>>>>> transactions in the whole cluster with possibility to filter them by > >>>>> nodes > >>>>>> - this is what user would expect out of the box. Another problem > >>>> appears > >>>>>> then - performance. How would we collect necessary data? How would > >>> we > >>>>>> handle joins, when particular view could be scanned multiple times > >>>> during > >>>>>> query execution? How we achieve sensible consistency? Most probably > >>> we > >>>>>> would collect remote data once when query is started, cache it > >>> somehow > >>>> on > >>>>>> query session level, and then re-use during joins. But again, this > >>>> should > >>>>>> be discussed separately. > >>>>>> > >>>>>> 4) Security: JVM info > >>>>>> We should define clear boundaries of what info is exposed. JVM data > >>>> along > >>>>>> with running threads is critically sensitive information. We should > >>> not > >>>>>> expose it until we have authorization capabilities. > >>>>>> > >>>>>> In order to start moving this code from prototype to production > >>> state > >>>> we > >>>>>> should start with the most simple and consistent views. E.g. > >>>>> IGNITE_CACHES. > >>>>>> Let's move it to a separate PR, review infrastructure code, review > >>> view > >>>>>> implementation, agree on proper naming and placement, and merge it. > >>>> Then > >>>>>> each and every view (or group of related views) should be discussed > >>> and > >>>>>> reviewed separately. > >>>>>> > >>>>>> As far as node-local stuff, may be we should move it to a separate > >>>>> schema, > >>>>>> or mark with special prefix. E.g. "IGNITE.TRANSACTIONS" - all > >>>>> transactions > >>>>>> in the cluster, "IGNITE.LOCAL_TRANSACTIONS" - transactions on the > >>> local > >>>>>> node. In this case we will be able to merge "local" stuff shortly, > >>> and > >>>>>> implement more complex but at the same time much more useful > >>>> distributed > >>>>>> stuff later on. > >>>>>> > >>>>>> Makes sense? > >>>>>> > >>>>>> Vladimir. > >>>>>> > >>>>>> > >>>>>> On Tue, Jan 23, 2018 at 8:30 PM, Alex Plehanov < > >>>> plehanov.a...@gmail.com> > >>>>>> wrote: > >>>>>> > >>>>>>> Hello, Igniters! > >>>>>>> > >>>>>>> For Ignite diagnostic usually it’s helpful to get some Ignite > >>>> internals > >>>>>>> information. But currently, in my opinion, there are no convenient > >>>>> tools > >>>>>>> for this purpose: > >>>>>>> > >>>>>>> · Some issues can be solved by analyzing log files. Log > >>> files > >>>>> are > >>>>>>> useful for dumps, but sometimes they are difficult to read. Also > >>>>>>> interesting metrics can’t be received runtime by request, we need > >>> to > >>>>> wait > >>>>>>> until Ignite will write these metrics by timeout or other events. > >>>>>>> > >>>>>>> · JMX is useful for scalar metrics. Complex and table data > >>> can > >>>>>> also > >>>>>>> be received, but it’s difficult to read, filter and sort them > >>> without > >>>>>>> processing by specialized external tools. For most frequently used > >>>>> cases > >>>>>>> almost duplicating metrics are created to show data in an > >>>> easy-to-read > >>>>>>> form. > >>>>>>> > >>>>>>> · Web-console is able to show table and complex data. > >>> Perhaps, > >>>>>>> someday web-console will contain all necessary dashboards for > >>> most > >>>>>> problem > >>>>>>> investigation, but some non-trivial queries will not be covered > >>>> anyway. > >>>>>>> Also web-console needs additional infrastructure to work. > >>>>>>> > >>>>>>> · External “home-made” tools can be used for non-trivial > >>>> cases. > >>>>>> They > >>>>>>> cover highly specialized cases and usually can’t be used as > >>> general > >>>>>> purpose > >>>>>>> tools. > >>>>>>> > >>>>>>> Sometimes we are forced to use more than one tool and join data by > >>>>> hands > >>>>>>> (for example, current thread dump and data from logs). > >>>>>>> > >>>>>>> Often RDBMS for diagnostic purposes provides system views (for > >>>> example, > >>>>>>> DBA_% and V$% in Oracle), which can be queried by SQL. This > >>> solution > >>>>>> makes > >>>>>>> all internal diagnostic information available in a readable form > >>>> (with > >>>>>> all > >>>>>>> possible filters and projections) without using any other > >>> internal or > >>>>>>> external tools. My proposal is to create similar system views in > >>>>> Ignite. > >>>>>>> > >>>>>>> I implement working prototype (PR: [1]). It contains views: > >>>>>>> > >>>>>>> IGNITE_SYSTEM_VIEWS > >>>>>>> > >>>>>>> Registered system views > >>>>>>> > >>>>>>> IGNITE_INSTANCE > >>>>>>> > >>>>>>> Ignite instance > >>>>>>> > >>>>>>> IGNITE_JVM_THREADS > >>>>>>> > >>>>>>> JVM threads > >>>>>>> > >>>>>>> IGNITE_JVM_RUNTIME > >>>>>>> > >>>>>>> JVM runtime > >>>>>>> > >>>>>>> IGNITE_JVM_OS > >>>>>>> > >>>>>>> JVM operating system > >>>>>>> > >>>>>>> IGNITE_CACHES > >>>>>>> > >>>>>>> Ignite caches > >>>>>>> > >>>>>>> IGNITE_CACHE_CLUSTER_METRICS > >>>>>>> > >>>>>>> Ignite cache cluster metrics > >>>>>>> > >>>>>>> IGNITE_CACHE_NODE_METRICS > >>>>>>> > >>>>>>> Ignite cache node metrics > >>>>>>> > >>>>>>> IGNITE_CACHE_GROUPS > >>>>>>> > >>>>>>> Cache groups > >>>>>>> > >>>>>>> IGNITE_NODES > >>>>>>> > >>>>>>> Nodes in topology > >>>>>>> > >>>>>>> IGNITE_NODE_HOSTS > >>>>>>> > >>>>>>> Node hosts > >>>>>>> > >>>>>>> IGNITE_NODE_ADDRESSES > >>>>>>> > >>>>>>> Node addresses > >>>>>>> > >>>>>>> IGNITE_NODE_ATTRIBUTES > >>>>>>> > >>>>>>> Node attributes > >>>>>>> > >>>>>>> IGNITE_NODE_METRICS > >>>>>>> > >>>>>>> Node metrics > >>>>>>> > >>>>>>> IGNITE_TRANSACTIONS > >>>>>>> > >>>>>>> Active transactions > >>>>>>> > >>>>>>> IGNITE_TRANSACTION_ENTRIES > >>>>>>> > >>>>>>> Cache entries used by transaction > >>>>>>> > >>>>>>> IGNITE_TASKS > >>>>>>> > >>>>>>> Active tasks > >>>>>>> > >>>>>>> IGNITE_PART_ASSIGNMENT > >>>>>>> > >>>>>>> Partition assignment map > >>>>>>> > >>>>>>> IGNITE_PART_ALLOCATION > >>>>>>> > >>>>>>> Partition allocation map > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> There are much more useful views can be implemented (executors > >>>>>> diagnostic, > >>>>>>> SPIs diagnostic, etc). > >>>>>>> > >>>>>>> Some usage examples: > >>>>>>> > >>>>>>> Cache groups and their partitions, which used by transaction more > >>>> than > >>>>> 5 > >>>>>>> minutes long: > >>>>>>> > >>>>>>> SELECT cg.CACHE_OR_GROUP_NAME, te.KEY_PARTITION, count(*) AS > >>>>> ENTITIES_CNT > >>>>>>> FROM INFORMATION_SCHEMA.IGNITE_TRANSACTIONS t > >>>>>>> JOIN INFORMATION_SCHEMA.IGNITE_TRANSACTION_ENTRIES te ON t.XID = > >>>>> te.XID > >>>>>>> JOIN INFORMATION_SCHEMA.IGNITE_CACHES c ON te.CACHE_NAME = c.NAME > >>>>>>> JOIN INFORMATION_SCHEMA.IGNITE_CACHE_GROUPS cg ON c.GROUP_ID = > >>> cg.ID > >>>>>>> WHERE t.START_TIME < TIMESTAMPADD('MINUTE', -5, NOW()) > >>>>>>> GROUP BY cg.CACHE_OR_GROUP_NAME, te.KEY_PARTITION > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Average CPU load on server nodes grouped by operating system: > >>>>>>> > >>>>>>> SELECT na.VALUE, COUNT(n.ID), AVG(nm.AVG_CPU_LOAD) AVG_CPU_LOAD > >>>>>>> FROM INFORMATION_SCHEMA.IGNITE_NODES n > >>>>>>> JOIN INFORMATION_SCHEMA.IGNITE_NODE_ATTRIBUTES na ON na.NODE_ID = > >>>> n.ID > >>>>>> AND > >>>>>>> na.NAME = 'os.name' > >>>>>>> JOIN INFORMATION_SCHEMA.IGNITE_NODE_METRICS nm ON nm.NODE_ID = > >>> n.ID > >>>>>>> WHERE n.IS_CLIENT = false > >>>>>>> GROUP BY na.VALUE > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Top 5 nodes by puts to cache ‘cache’: > >>>>>>> > >>>>>>> SELECT cm.NODE_ID, cm.CACHE_PUTS FROM > >>>>>>> INFORMATION_SCHEMA.IGNITE_CACHE_NODE_METRICS cm > >>>>>>> WHERE cm.CACHE_NAME = 'cache' > >>>>>>> ORDER BY cm.CACHE_PUTS DESC > >>>>>>> LIMIT 5 > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Does this implementation interesting to someone else? Maybe any > >>> views > >>>>> are > >>>>>>> redundant? Which additional first-priority views must be > >>> implemented? > >>>>> Any > >>>>>>> other thoughts or proposal? > >>>>>>> > >>>>>>> [1] https://github.com/apache/ignite/pull/3413 > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > >> > >
