Vova,

Could you confirm https://issues.apache.org/jira/browse/IGNITE-7527 ready
to be merged?

On Wed, Feb 14, 2018 at 12:01 PM, Vladimir Ozerov <voze...@gridgain.com>
wrote:

> I would start with NODES and NODE_ATTRIBUTES as the most simple thing.
>
> On Tue, Feb 13, 2018 at 4:10 AM, Denis Magda <dma...@apache.org> wrote:
>
> > Alex P, sounds like a good plan for me.
> >
> > Vladimir, do you have any suggestions or corrections?
> >
> > —
> > Denis
> >
> > > On Feb 12, 2018, at 4:57 AM, Alex Plehanov <plehanov.a...@gmail.com>
> > wrote:
> > >
> > > The views engine and the first view are almost ready to merge (review
> > > comments are resolved). Which views should we take next? My proposal -
> > > NODES, NODE_ATTRIBUTES, NODE_METRICS, NODE_HOSTS and NODE_ADDRESSES,
> > since
> > > these views are clear and all topology data available on each node.
> > > Any objections?
> > >
> > > 2018-01-25 16:27 GMT+03:00 Alex Plehanov <plehanov.a...@gmail.com>:
> > >
> > >> Anton, Vladimir, I've made some fixes. There is only one view left and
> > >> it's renamed to 'IGNITE.LOCAL_TRANSACTIONS'.
> > >>
> > >> High level design of solution:
> > >> When IgniteH2Indexing is starting, it create and start
> > >> new GridH2SysViewProcessor, which create and register in H2 (via its
> own
> > >> table engine) all implementations of system views. Each system view
> > >> implementation extends base abstract class GridH2SysView. View
> > >> implementation describes columns, their types and indexes in
> constructor
> > >> and must override method getRows for data retrieval (this method
> called
> > by
> > >> H2-compatible table and index implementations for ignite system
> views).
> > >> Almost no fixes to existing parsing engine was made, except some
> places,
> > >> where GridH2Table instance was expected, but for system views there is
> > >> another class.
> > >>
> > >> New PR: [1].  Please have a look.
> > >>
> > >> [1] https://github.com/apache/ignite/pull/3433
> > >>
> > >> 2018-01-24 19:12 GMT+03:00 Anton Vinogradov <avinogra...@gridgain.com
> >:
> > >>
> > >>> I've created IEP-13 [1] to cover all cases.
> > >>> Feel free to create issues.
> > >>>
> > >>> [1]
> > >>> https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=75962769
> > >>>
> > >>> On Wed, Jan 24, 2018 at 6:10 PM, Vladimir Ozerov <
> voze...@gridgain.com
> > >
> > >>> wrote:
> > >>>
> > >>>> Let's start with a single and the most simple view, e.g.
> > >>>> LOCAL_TRANSACTIONS. We will review and merge it along with necessary
> > >>>> infrastructure. Then will handle the rest view in separate tickets
> and
> > >>>> separate focused discussions.
> > >>>>
> > >>>> On Wed, Jan 24, 2018 at 5:29 PM, Alex Plehanov <
> > plehanov.a...@gmail.com
> > >>>>
> > >>>> wrote:
> > >>>>
> > >>>>> 1) It’s not a principal point, I can change schema. The
> > >>>> INFORMATION_SCHEMA
> > >>>>> was used because it’s already exists and usually used for metadata
> > >>> tables
> > >>>>> and views. Your proposal is to use schema “IGNITE”, am I understand
> > >>> you
> > >>>>> right? BTW, for now, we can’t query another (H2) meta tables from
> the
> > >>>>> INFORMATION_SCHEMA, so, “Ignite system views” is only available
> views
> > >>> to
> > >>>>> query from this schema.
> > >>>>> 2) Exactly for this reason the IGNITE_INSTANCE view is useful: to
> > >>>> determine
> > >>>>> which node we are connected to.
> > >>>>> 3) As the first phase, in my opinion, local views will be enough.
> > >>>>> Performance and caching of distributed views should be discussed at
> > >>> next
> > >>>>> phases, when distributed views implementation will be planned. In
> > >>> current
> > >>>>> implementation I tried to use indexing for local views wherever
> it’s
> > >>>>> possible.
> > >>>>> 4) I don’t think, that JVM info is more critical information than,
> > for
> > >>>>> example, caches or nodes information. When authorization
> capabilities
> > >>>>> planned to implement?
> > >>>>>
> > >>>>> About local data: yes, we can rename all currently implemented
> views
> > >>> for
> > >>>>> the local node data as LOCAL_..., and create (someday) new whole
> > >>> cluster
> > >>>>> views (which use distributed requests) without prefix or, for
> > example,
> > >>>> with
> > >>>>> CLUSTER_ prefix. But some views can show all cluster information
> > using
> > >>>> only
> > >>>>> local node data, without distributed requests (for example
> > >>>>> IGNITE_NODE_METRICS, IGNITE_PART_ASSIGNMENT,
> IGNITE_PART_ALLOCATION,
> > >>>>> IGNITE_NODES, etc). Are they local or cluster views in this
> concept?
> > >>>> Which
> > >>>>> prefix should be used? And what about caches? Are they local or
> > >>> cluster?
> > >>>> On
> > >>>>> local node we can see cluster wide caches (replicated and
> > distributed)
> > >>>> and
> > >>>>> caches for current node only. Local caches list may differ from
> node
> > >>> to
> > >>>>> node. Which prefix should be used for this view? And one more,
> there
> > >>> is
> > >>>> no
> > >>>>> sense for some views to make them cluster wide (for example
> > >>>>> INGNITE_INSTANCE). Should we name it LOCAL_INSTANCE without
> creating
> > >>>>> INSTANCE view?
> > >>>>>
> > >>>>> So, next steps: split PR, change schema name (IGNITE?), change view
> > >>> name
> > >>>>> for caches (CACHES, LOCAL_CACHES?)
> > >>>>>
> > >>>>>
> > >>>>> 2018-01-24 13:03 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>:
> > >>>>>
> > >>>>>> Hi Alex,
> > >>>>>>
> > >>>>>> System views could be extremely valuable addition for Ignite.
> > >>> Ideally,
> > >>>>> user
> > >>>>>> should be able to monitor and manage state of the whole cluster
> > >>> with a
> > >>>>>> single SQL command line. We have plans to implement it for a very
> > >>> long
> > >>>>>> time. However, this is very sensitive task which should take a lot
> > >>> of
> > >>>>>> moving pieces in count, such as usability, consistency,
> performance,
> > >>>>>> security, etc..
> > >>>>>>
> > >>>>>> Let me point several major concerns I see at the moment:
> > >>>>>>
> > >>>>>> 1) Usability: INFORMATION_SCHEMA
> > >>>>>> This schema is part of SQL ANSI standard. When creating system
> > >>> views,
> > >>>>> some
> > >>>>>> vendors prefer to store them in completely different predefined
> > >>> schema
> > >>>>>> (Oracle, MS SQL). Others prefer to keep them in INFORMATION_SCHEMA
> > >>>>>> directly. Both approaches could work. However, the latter breaks
> > >>>>> separation
> > >>>>>> of concerns - we store typical metadata near to possibly sensitive
> > >>>> system
> > >>>>>> data. Also it makes security management more complex - system data
> > >>> is
> > >>>>> very
> > >>>>>> sensitive, and now we cannot simply grant access
> > >>> INFORMATIONAL_SCHEMA
> > >>>> to
> > >>>>>> user. Instead, we have to grant that access on per-view basis. For
> > >>> this
> > >>>>>> reason my preference is to store system tables in separate schema,
> > >>> not
> > >>>> in
> > >>>>>> INFORMATION_SCHEMA
> > >>>>>>
> > >>>>>> 2) Consistency: local data
> > >>>>>> One of implemented view GridH2SysViewImplInstance. Normally SQL
> > >>> users
> > >>>>>> communicate with Ignite through JDBC/ODBC drivers. These drivers
> are
> > >>>>>> connected to a single node, typically client node. Moreover, we
> will
> > >>>>>> introduce high-availability feature when drivers were able to
> > >>> connect
> > >>>> to
> > >>>>>> any address from a predefined list. It renders this view useless,
> as
> > >>>> you
> > >>>>> do
> > >>>>>> not know which node you connected to. Also, local-only data cannot
> > >>> be
> > >>>>>> joined in general case - you will receive different results on
> > >>>> different
> > >>>>>> nodes. The same goes for transactions, JVM info, etc.
> > >>>>>>
> > >>>>>> 3) Performance
> > >>>>>> Suppose we fixed consistency of transactions and now this view
> shows
> > >>>>>> transactions in the whole cluster with possibility to filter them
> by
> > >>>>> nodes
> > >>>>>> - this is what user would expect out of the box. Another problem
> > >>>> appears
> > >>>>>> then - performance. How would we collect necessary data? How would
> > >>> we
> > >>>>>> handle joins, when particular view could be scanned multiple times
> > >>>> during
> > >>>>>> query execution? How we achieve sensible consistency? Most
> probably
> > >>> we
> > >>>>>> would collect remote data once when query is started, cache it
> > >>> somehow
> > >>>> on
> > >>>>>> query session level, and then re-use during joins. But again, this
> > >>>> should
> > >>>>>> be discussed separately.
> > >>>>>>
> > >>>>>> 4) Security: JVM info
> > >>>>>> We should define clear boundaries of what info is exposed. JVM
> data
> > >>>> along
> > >>>>>> with running threads is critically sensitive information. We
> should
> > >>> not
> > >>>>>> expose it until we have authorization capabilities.
> > >>>>>>
> > >>>>>> In order to start moving this code from prototype to production
> > >>> state
> > >>>> we
> > >>>>>> should start with the most simple and consistent views. E.g.
> > >>>>> IGNITE_CACHES.
> > >>>>>> Let's move it to a separate PR, review infrastructure code, review
> > >>> view
> > >>>>>> implementation, agree on proper naming and placement, and merge
> it.
> > >>>> Then
> > >>>>>> each and every view (or group of related views) should be
> discussed
> > >>> and
> > >>>>>> reviewed separately.
> > >>>>>>
> > >>>>>> As far as node-local stuff, may be we should move it to a separate
> > >>>>> schema,
> > >>>>>> or mark with special prefix. E.g. "IGNITE.TRANSACTIONS" - all
> > >>>>> transactions
> > >>>>>> in the cluster, "IGNITE.LOCAL_TRANSACTIONS" - transactions on the
> > >>> local
> > >>>>>> node. In this case we will be able to merge "local" stuff shortly,
> > >>> and
> > >>>>>> implement more complex but at the same time much more useful
> > >>>> distributed
> > >>>>>> stuff later on.
> > >>>>>>
> > >>>>>> Makes sense?
> > >>>>>>
> > >>>>>> Vladimir.
> > >>>>>>
> > >>>>>>
> > >>>>>> On Tue, Jan 23, 2018 at 8:30 PM, Alex Plehanov <
> > >>>> plehanov.a...@gmail.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Hello, Igniters!
> > >>>>>>>
> > >>>>>>> For Ignite diagnostic usually it’s helpful to get some Ignite
> > >>>> internals
> > >>>>>>> information. But currently, in my opinion, there are no
> convenient
> > >>>>> tools
> > >>>>>>> for this purpose:
> > >>>>>>>
> > >>>>>>> ·        Some issues can be solved by analyzing log files. Log
> > >>> files
> > >>>>> are
> > >>>>>>> useful for dumps, but sometimes they are difficult to read. Also
> > >>>>>>> interesting metrics can’t be received runtime by request, we need
> > >>> to
> > >>>>> wait
> > >>>>>>> until Ignite will write these metrics by timeout or other events.
> > >>>>>>>
> > >>>>>>> ·        JMX is useful for scalar metrics. Complex and table data
> > >>> can
> > >>>>>> also
> > >>>>>>> be received, but it’s difficult to read, filter and sort them
> > >>> without
> > >>>>>>> processing by specialized external tools. For most frequently
> used
> > >>>>> cases
> > >>>>>>> almost duplicating metrics are created to show data in an
> > >>>> easy-to-read
> > >>>>>>> form.
> > >>>>>>>
> > >>>>>>> ·        Web-console is able to show table and complex data.
> > >>> Perhaps,
> > >>>>>>> someday  web-console will contain all necessary dashboards for
> > >>> most
> > >>>>>> problem
> > >>>>>>> investigation, but some non-trivial queries will not be covered
> > >>>> anyway.
> > >>>>>>> Also web-console needs additional infrastructure to work.
> > >>>>>>>
> > >>>>>>> ·        External “home-made” tools can be used for non-trivial
> > >>>> cases.
> > >>>>>> They
> > >>>>>>> cover highly specialized cases and usually can’t be used as
> > >>> general
> > >>>>>> purpose
> > >>>>>>> tools.
> > >>>>>>>
> > >>>>>>> Sometimes we are forced to use more than one tool and join data
> by
> > >>>>> hands
> > >>>>>>> (for example, current thread dump and data from logs).
> > >>>>>>>
> > >>>>>>> Often RDBMS for diagnostic purposes provides system views (for
> > >>>> example,
> > >>>>>>> DBA_% and V$% in Oracle), which can be queried by SQL. This
> > >>> solution
> > >>>>>> makes
> > >>>>>>> all internal diagnostic information available in a readable form
> > >>>> (with
> > >>>>>> all
> > >>>>>>> possible filters and projections) without using any other
> > >>> internal or
> > >>>>>>> external tools. My proposal is to create similar system views in
> > >>>>> Ignite.
> > >>>>>>>
> > >>>>>>> I implement working prototype (PR: [1]). It contains views:
> > >>>>>>>
> > >>>>>>> IGNITE_SYSTEM_VIEWS
> > >>>>>>>
> > >>>>>>> Registered system views
> > >>>>>>>
> > >>>>>>> IGNITE_INSTANCE
> > >>>>>>>
> > >>>>>>> Ignite instance
> > >>>>>>>
> > >>>>>>> IGNITE_JVM_THREADS
> > >>>>>>>
> > >>>>>>> JVM threads
> > >>>>>>>
> > >>>>>>> IGNITE_JVM_RUNTIME
> > >>>>>>>
> > >>>>>>> JVM runtime
> > >>>>>>>
> > >>>>>>> IGNITE_JVM_OS
> > >>>>>>>
> > >>>>>>> JVM operating system
> > >>>>>>>
> > >>>>>>> IGNITE_CACHES
> > >>>>>>>
> > >>>>>>> Ignite caches
> > >>>>>>>
> > >>>>>>> IGNITE_CACHE_CLUSTER_METRICS
> > >>>>>>>
> > >>>>>>> Ignite cache cluster metrics
> > >>>>>>>
> > >>>>>>> IGNITE_CACHE_NODE_METRICS
> > >>>>>>>
> > >>>>>>> Ignite cache node metrics
> > >>>>>>>
> > >>>>>>> IGNITE_CACHE_GROUPS
> > >>>>>>>
> > >>>>>>> Cache groups
> > >>>>>>>
> > >>>>>>> IGNITE_NODES
> > >>>>>>>
> > >>>>>>> Nodes in topology
> > >>>>>>>
> > >>>>>>> IGNITE_NODE_HOSTS
> > >>>>>>>
> > >>>>>>> Node hosts
> > >>>>>>>
> > >>>>>>> IGNITE_NODE_ADDRESSES
> > >>>>>>>
> > >>>>>>> Node addresses
> > >>>>>>>
> > >>>>>>> IGNITE_NODE_ATTRIBUTES
> > >>>>>>>
> > >>>>>>> Node attributes
> > >>>>>>>
> > >>>>>>> IGNITE_NODE_METRICS
> > >>>>>>>
> > >>>>>>> Node metrics
> > >>>>>>>
> > >>>>>>> IGNITE_TRANSACTIONS
> > >>>>>>>
> > >>>>>>> Active transactions
> > >>>>>>>
> > >>>>>>> IGNITE_TRANSACTION_ENTRIES
> > >>>>>>>
> > >>>>>>> Cache entries used by transaction
> > >>>>>>>
> > >>>>>>> IGNITE_TASKS
> > >>>>>>>
> > >>>>>>> Active tasks
> > >>>>>>>
> > >>>>>>> IGNITE_PART_ASSIGNMENT
> > >>>>>>>
> > >>>>>>> Partition assignment map
> > >>>>>>>
> > >>>>>>> IGNITE_PART_ALLOCATION
> > >>>>>>>
> > >>>>>>> Partition allocation map
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> There are much more useful views can be implemented (executors
> > >>>>>> diagnostic,
> > >>>>>>> SPIs diagnostic, etc).
> > >>>>>>>
> > >>>>>>> Some usage examples:
> > >>>>>>>
> > >>>>>>> Cache groups and their partitions, which used by transaction more
> > >>>> than
> > >>>>> 5
> > >>>>>>> minutes long:
> > >>>>>>>
> > >>>>>>> SELECT cg.CACHE_OR_GROUP_NAME, te.KEY_PARTITION, count(*) AS
> > >>>>> ENTITIES_CNT
> > >>>>>>> FROM INFORMATION_SCHEMA.IGNITE_TRANSACTIONS t
> > >>>>>>> JOIN INFORMATION_SCHEMA.IGNITE_TRANSACTION_ENTRIES te ON t.XID =
> > >>>>> te.XID
> > >>>>>>> JOIN INFORMATION_SCHEMA.IGNITE_CACHES c ON te.CACHE_NAME =
> c.NAME
> > >>>>>>> JOIN INFORMATION_SCHEMA.IGNITE_CACHE_GROUPS cg ON c.GROUP_ID =
> > >>> cg.ID
> > >>>>>>> WHERE t.START_TIME < TIMESTAMPADD('MINUTE', -5, NOW())
> > >>>>>>> GROUP BY cg.CACHE_OR_GROUP_NAME, te.KEY_PARTITION
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Average CPU load on server nodes grouped by operating system:
> > >>>>>>>
> > >>>>>>> SELECT na.VALUE, COUNT(n.ID), AVG(nm.AVG_CPU_LOAD) AVG_CPU_LOAD
> > >>>>>>> FROM INFORMATION_SCHEMA.IGNITE_NODES n
> > >>>>>>> JOIN INFORMATION_SCHEMA.IGNITE_NODE_ATTRIBUTES na ON na.NODE_ID
> =
> > >>>> n.ID
> > >>>>>> AND
> > >>>>>>> na.NAME = 'os.name'
> > >>>>>>> JOIN INFORMATION_SCHEMA.IGNITE_NODE_METRICS nm ON nm.NODE_ID =
> > >>> n.ID
> > >>>>>>> WHERE n.IS_CLIENT = false
> > >>>>>>> GROUP BY na.VALUE
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Top 5 nodes by puts to cache ‘cache’:
> > >>>>>>>
> > >>>>>>> SELECT cm.NODE_ID, cm.CACHE_PUTS FROM
> > >>>>>>> INFORMATION_SCHEMA.IGNITE_CACHE_NODE_METRICS cm
> > >>>>>>> WHERE cm.CACHE_NAME = 'cache'
> > >>>>>>> ORDER BY cm.CACHE_PUTS DESC
> > >>>>>>> LIMIT 5
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Does this implementation interesting to someone else? Maybe any
> > >>> views
> > >>>>> are
> > >>>>>>> redundant? Which additional first-priority views must be
> > >>> implemented?
> > >>>>> Any
> > >>>>>>> other thoughts or proposal?
> > >>>>>>>
> > >>>>>>> [1] https://github.com/apache/ignite/pull/3413
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >>
> >
> >
>

Reply via email to