Re: REST GridCacheCommandHandler writes ERROR in log in case of bad user input

2018-12-14 Thread Denis Magda
Hello Ilya,

It's fine to use ERROR level if an operation can't be completed due to
missing parameters. That's, in fact, an exception/error. What needs to be
changed is texts of messages so that everyone understands what exactly
happened and how to address a failure.

--
Denis


On Fri, Dec 14, 2018 at 5:45 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> It seems that we have subj behavior since the earliest days if Apache
> Ignite.
>
> If you send a REST command with error in it (such as missing "keys" for
> getAll), you will get a nice
> [2018-10-30 22:22:14,021][ERROR][rest-#61061][GridCacheCommandHandler]
> Failed to execute cache command: GridRestCacheRequest
> error in your logs.
>
> Which is probably an overkill since ERROR usually means unexpected and
> severe errors in application code as opposed to user input validation
> errors.
>
> This made worse by the fact that you can have some automatic REST client do
> a lot of incorrect requests, spam your logs with thousands of such ERRORs.
> The error is returned to client but it is also tee'd to log.
>
> What we could do:
> - Move log level from ERROR to WARN or even INFO.
> - Handle REST user input validation errors differently from Ignite internal
> errors by introducing new exception class, logging it as INFO or maybe just
> returning to user.
> - Third funny option?
>
> WDYT?
>
> Regards,
> --
> Ilya Kasnacheev
>


Re: Continuous queries and duplicates

2018-12-14 Thread Denis Magda
Vladimir,

Thanks for referring to the MVCC and Continuous Queries discussion, I knew
that saw us discussing a solution of the duplication problem. Let me copy
and paste it in here for others:

2) *Initial query*. We implemented it so that user can get some initial
> data snapshot and then start receiving events. Without MVCC we have no
> guarantees of visibility. E.g. if key is updated from V1 to V2, it is
> possible to see V2 in initial query and in event. With MVCC it is now
> technically possible to query data on certain snapshot and then receive
> only events happened after this snapshot. So that we never see V2 twice.
> Do
> you think we this feature will be interesting for our users?


Am I right that this would be a generic solution - whether you use Scan or
SQL query as an initial one? Have we planned it for the transactional SQL
GA or it's out of scope for now?

--
Denis

On Thu, Dec 13, 2018 at 12:40 PM Vladimir Ozerov 
wrote:

> [1]
>
> http://apache-ignite-developers.2346864.n4.nabble.com/Continuous-queries-and-MVCC-td33972.html
>
> On Thu, Dec 13, 2018 at 11:38 PM Vladimir Ozerov 
> wrote:
>
> > Denis,
> >
> > Not really. They are used to ensure that ordering of notifications is
> > consistent with ordering of updates, so that when a key K is updated to
> V1,
> > then V2, then V3, you never observe V1 -> V3 -> V2. It also solves
> > duplicate notification problem in case of node failures, when the same
> > update is delivered twice.
> >
> > However, partition counters are unable to solve duplicates problem in
> > general. Essentially, the question is how to get consistent view on some
> > data plus all notifications which happened afterwards. There are only two
> > ways to achieve this - either lock entries during initial query, or take
> a
> > kind of consistent data snapshot. The former was never implemented in
> > Ignite - our Scan and SQL queries do not user locking. The latter is
> > achievable in theory with MVCC. I raised that question earlier [1] (see
> > p.2), and we came to conclusion that it might be a good feature for the
> > product. It is not implemented that way for MVCC now, but most probably
> is
> > not extraordinary difficult to implement.
> >
> > Vladimir.
> >
> > [1]
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Continuous-queries-and-MVCC-td33972.html#a33998
> >
> > On Thu, Dec 13, 2018 at 11:17 PM Denis Magda  wrote:
> >
> >> Vladimir,
> >>
> >> The partition counter is supposed to be used internally to solve the
> >> duplication issue. Does it sound like a right approach then?
> >>
> >> What would be an approach for SQL queries? Not sure the partition
> counter
> >> is applicable.
> >>
> >> --
> >> Denis
> >>
> >> On Thu, Dec 13, 2018 at 11:16 AM Vladimir Ozerov 
> >> wrote:
> >>
> >> > Partition counter is internal implemenattion detail, which has no
> >> sensible
> >> > meaning to end users. It should not be exposed through public API.
> >> >
> >> > On Thu, Dec 13, 2018 at 10:14 PM Denis Magda 
> wrote:
> >> >
> >> > > Hello Piotr,
> >> > >
> >> > > That's a known problem and I thought a JIRA ticket already exists.
> >> > However,
> >> > > failed to locate it. The ticket for the improvement should be
> created
> >> as
> >> > a
> >> > > result of this conversation.
> >> > >
> >> > > Speaking of an initial query type, I would differentiate from
> >> ScanQueries
> >> > > and SqlQueries. For the former, it sounds reasonable to apply the
> >> > > partitionCounter logic. As for the latter, Vladimir Ozerov will it
> be
> >> > > addressed as part of MVCC/Transactional SQL activities?
> >> > >
> >> > > Btw, Piotr what's your initial query type?
> >> > >
> >> > > --
> >> > > Denis
> >> > >
> >> > > On Thu, Dec 13, 2018 at 3:28 AM Piotr Romański <
> >> piotr.roman...@gmail.com
> >> > >
> >> > > wrote:
> >> > >
> >> > > > Hi, as suggested by Ilya here:
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> http://apache-ignite-users.70518.x6.nabble.com/Continuous-queries-and-duplicates-td25314.html
> >> > > > I'm resending it to the developers list.
> >> > > >
> >> > >

Re: CONTRIBUTING.md first version

2018-12-14 Thread Denis Magda
>
> `Contributing.md` is more not about how to contribute in day by day basis,
> but a short step-by-step guide on how to join and start contributing.


Could we rework and refine the wiki version in a similar way - most
important stuff first, old is either removed and reworked.

--
Denis

On Fri, Dec 14, 2018 at 2:43 AM Dmitriy Pavlov  wrote:

> Thank you for sharing your vision, I also agree to have short in GH and
> full in the wiki.
>
> Denis, I see a number of outdated info in How To Contribute, but most
> sections there are for a strong reason. A lot of cases covered, and a lot
> of cases could be covered in addition to existing.
>
> `Contributing.md` is more not about how to contribute in day by day basis,
> but a short step-by-step guide on how to join and start contributing.
>
> Moreover, I would like to refactor website section
> https://ignite.apache.org/community/contribute.html#contribute in the same
> manner.
>
> пт, 14 дек. 2018 г. в 11:29, Petr Ivanov :
>
> > Agree. GitHub can have shorter quick notes with link to full
> documentation.
> >
> >
> > > On 14 Dec 2018, at 08:05, Alexey Kuznetsov 
> > wrote:
> > >
> > > Hi,
> > >
> > > From my experience it is better when all needed stuff available on
> > GitHub.
> > >
> > > It is very comfortable when thinks like, "read me", "change log", "how
> to
> > > build", "how to contribute" and so on available on GitHub.
> > >
> > > No need to open some external links.
> > >
> > > --
> > > Alexey Kuznetsov
> >
> >
>


Re: Continuous queries and duplicates

2018-12-13 Thread Denis Magda
Vladimir,

The partition counter is supposed to be used internally to solve the
duplication issue. Does it sound like a right approach then?

What would be an approach for SQL queries? Not sure the partition counter
is applicable.

--
Denis

On Thu, Dec 13, 2018 at 11:16 AM Vladimir Ozerov 
wrote:

> Partition counter is internal implemenattion detail, which has no sensible
> meaning to end users. It should not be exposed through public API.
>
> On Thu, Dec 13, 2018 at 10:14 PM Denis Magda  wrote:
>
> > Hello Piotr,
> >
> > That's a known problem and I thought a JIRA ticket already exists.
> However,
> > failed to locate it. The ticket for the improvement should be created as
> a
> > result of this conversation.
> >
> > Speaking of an initial query type, I would differentiate from ScanQueries
> > and SqlQueries. For the former, it sounds reasonable to apply the
> > partitionCounter logic. As for the latter, Vladimir Ozerov will it be
> > addressed as part of MVCC/Transactional SQL activities?
> >
> > Btw, Piotr what's your initial query type?
> >
> > --
> > Denis
> >
> > On Thu, Dec 13, 2018 at 3:28 AM Piotr Romański  >
> > wrote:
> >
> > > Hi, as suggested by Ilya here:
> > >
> > >
> >
> http://apache-ignite-users.70518.x6.nabble.com/Continuous-queries-and-duplicates-td25314.html
> > > I'm resending it to the developers list.
> > >
> > > From that thread we know that there might be duplicates between initial
> > > query results and listener entries received as part of continuous
> query.
> > > That means that users need to manually dedupe data.
> > >
> > > In my opinion the manual deduplication in some use cases may lead to
> > > possible memory problems on the client side. In order to remove
> > duplicated
> > > notifications which we are receiving in the local listener, we need to
> > keep
> > > all initial query results in memory (or at least their unique ids).
> > > Unfortunately, there is no way (is there?) to find a point in time when
> > we
> > > can be sure that no dups will arrive anymore. That would mean that we
> > need
> > > to keep that data indefinitely and use it every time a new notification
> > > arrives. In case of multiple continuous queries run from a single JVM,
> > this
> > > might eventually become a memory or performance problem. I can see the
> > > following possible improvements to Ignite:
> > >
> > > 1. The deduplication between initial query and incoming notification
> > could
> > > be done fully in Ignite. As far as I know there is already the
> > > updateCounter and partition id for all the objects so it could be used
> > > internally.
> > >
> > > 2. Add a guarantee that notifications arriving in the local listener
> > after
> > > query() method returns are not duplicates. This kind of functionality
> > would
> > > require a specific synchronization inside Ignite. It would also mean
> that
> > > the query() method cannot return before all potential duplicates are
> > > processed by a local listener what looks wrong.
> > >
> > > 3. Notify users that starting from a given notification they can be
> sure
> > > they will not receive any duplicates anymore. This could be an
> additional
> > > boolean flag in the CacheQueryEntryEvent.
> > >
> > > 4. CacheQueryEntryEvent already exposes the partitionUpdateCounter.
> > > Unfortunately we don't have this information for initial query results.
> > If
> > > we had, a client could manually deduplicate notifications and get rid
> of
> > > initial query results for a given partition after newer notifications
> > > arrive. Also it would be very convenient to expose partition id as well
> > but
> > > now we can figure it out using the affinity service. The assumption
> here
> > is
> > > that notifications are ordered by partitionUpdateCounter (is it true?).
> > >
> > > Please correct me if I'm missing anything.
> > >
> > > What do you think?
> > >
> > > Piotr
> > >
> >
>


Re: Continuous queries and duplicates

2018-12-13 Thread Denis Magda
Hello Piotr,

That's a known problem and I thought a JIRA ticket already exists. However,
failed to locate it. The ticket for the improvement should be created as a
result of this conversation.

Speaking of an initial query type, I would differentiate from ScanQueries
and SqlQueries. For the former, it sounds reasonable to apply the
partitionCounter logic. As for the latter, Vladimir Ozerov will it be
addressed as part of MVCC/Transactional SQL activities?

Btw, Piotr what's your initial query type?

--
Denis

On Thu, Dec 13, 2018 at 3:28 AM Piotr Romański 
wrote:

> Hi, as suggested by Ilya here:
>
> http://apache-ignite-users.70518.x6.nabble.com/Continuous-queries-and-duplicates-td25314.html
> I'm resending it to the developers list.
>
> From that thread we know that there might be duplicates between initial
> query results and listener entries received as part of continuous query.
> That means that users need to manually dedupe data.
>
> In my opinion the manual deduplication in some use cases may lead to
> possible memory problems on the client side. In order to remove duplicated
> notifications which we are receiving in the local listener, we need to keep
> all initial query results in memory (or at least their unique ids).
> Unfortunately, there is no way (is there?) to find a point in time when we
> can be sure that no dups will arrive anymore. That would mean that we need
> to keep that data indefinitely and use it every time a new notification
> arrives. In case of multiple continuous queries run from a single JVM, this
> might eventually become a memory or performance problem. I can see the
> following possible improvements to Ignite:
>
> 1. The deduplication between initial query and incoming notification could
> be done fully in Ignite. As far as I know there is already the
> updateCounter and partition id for all the objects so it could be used
> internally.
>
> 2. Add a guarantee that notifications arriving in the local listener after
> query() method returns are not duplicates. This kind of functionality would
> require a specific synchronization inside Ignite. It would also mean that
> the query() method cannot return before all potential duplicates are
> processed by a local listener what looks wrong.
>
> 3. Notify users that starting from a given notification they can be sure
> they will not receive any duplicates anymore. This could be an additional
> boolean flag in the CacheQueryEntryEvent.
>
> 4. CacheQueryEntryEvent already exposes the partitionUpdateCounter.
> Unfortunately we don't have this information for initial query results. If
> we had, a client could manually deduplicate notifications and get rid of
> initial query results for a given partition after newer notifications
> arrive. Also it would be very convenient to expose partition id as well but
> now we can figure it out using the affinity service. The assumption here is
> that notifications are ordered by partitionUpdateCounter (is it true?).
>
> Please correct me if I'm missing anything.
>
> What do you think?
>
> Piotr
>


Re: CONTRIBUTING.md first version

2018-12-13 Thread Denis Magda
The Github page looks like a shorter version of the main wiki page:
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute

Not sure we want to maintain both. If the wiki source is sluggish and
overloaded then I would rework it instead moving secondary stuff to
sub-pages. Think that the wiki needs the grooming regardless.

--
Denis



On Wed, Dec 12, 2018 at 2:23 PM Dmitriy Pavlov  wrote:

> There is nothing wrong with wiki (excepting it is sometimes way too slow
> and it may be annoying ).
>
> Github markdown is a way to
> 1. Keep documenting under version control system
> 2. Github recommends to provide this doc in the standard place.
>
> Of course, we can move this new doc content to wiki and just provide link
> from contributing.md to wiki. Should we?
>
> I can also add a copy to a wiki. Denis, folks,  please share your vision.
>
> чт, 13 дек. 2018 г., 0:32 Denis Magda :
>
> > What is wrong with Ignite wiki? It seems like the right place for the doc
> > like this.
> >
> > --
> > Denis
> >
> > On Wed, Dec 12, 2018 at 9:21 AM Dmitriy Pavlov 
> wrote:
> >
> > > Hi Igniters,
> > >
> > > Just to inform, the very first version of the Contribution step by step
> > > guide was added to GitHub. It is one more entry point for newcomers to
> > find
> > > out what and how should be done to contribute:
> > >
> > >
> >
> https://github.com/apache/ignite/blob/master/CONTRIBUTING.md#contributing-to-apache-ignite
> > >
> > >
> > > Feel free to share with newcomers and contribute.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> >
>


Re: CONTRIBUTING.md first version

2018-12-12 Thread Denis Magda
What is wrong with Ignite wiki? It seems like the right place for the doc
like this.

--
Denis

On Wed, Dec 12, 2018 at 9:21 AM Dmitriy Pavlov  wrote:

> Hi Igniters,
>
> Just to inform, the very first version of the Contribution step by step
> guide was added to GitHub. It is one more entry point for newcomers to find
> out what and how should be done to contribute:
>
> https://github.com/apache/ignite/blob/master/CONTRIBUTING.md#contributing-to-apache-ignite
>
>
> Feel free to share with newcomers and contribute.
>
> Sincerely,
> Dmitriy Pavlov
>


Re: Packt wants to feature books

2018-12-11 Thread Denis Magda
Hello Hishit,

Thanks for sharing the news, great!

Prachi, could you please add a reference to the to our website menu?

--
Denis

On Tue, Dec 11, 2018 at 10:14 AM Nishit Rajput 
wrote:

> Hi team,
>
>
> I am Nishit from Packt. We publish over 50 books related to trending and
> upcoming technologies every month. We also feature various books on Project
> websites.
>
>
> We would be interested in featuring our recently published book Apache
> Ignite Quick Start Guide<
> https://www.packtpub.com/big-data-and-business-intelligence/apache-ignite-quick-start-guide>
> by Sujoy Acharya
>
>
> The book takes you through the basics of Apache Ignite and in-memory
> technologies. You will learn about installation and clustering Ignite
> nodes, caching topologies, and various caching strategies, such as cache
> aside, read and write through, and write behind. Next, you will delve into
> detailed aspects of Ignite’s data grid: web session clustering and querying
> data.
>
> You will learn how to process large volumes of data using compute grid and
> Ignite’s map-reduce and executor service. You will learn about the memory
> architecture of Apache Ignite and monitoring memory and caches. You will
> use Ignite for complex event processing, event streaming, and the
> time-series predictions of opportunities and threats. Additionally, you
> will go through off-heap and on-heap caching, swapping, and native and
> Spring framework integration with Apache Ignite.
>
> By the end of this book, you will be confident with all the features of
> Apache Ignite 2.x that can be used to build a high-performance system
> architecture.
>
>
> We can shorten the description as per your website's requirement.
>
>
> I am certain this book will prove to be an insightful resource to the
> developers.
>
>
> Do let me know if we can feature this book on your website and if you can
> help us featuring it. If not, can you please direct me to the right person
> for this requirement.
>
>
> Looking forward to your reply.
>
>
> Thanks and regards,
>
> [https://s3-eu-west-1.amazonaws.com/email-signature-packt/128x128px.png]
>
>
>
>
>
> Nishit Rajput
>
> Author Marketing Executive
>
>
>
>  [https://s3-eu-west-1.amazonaws.com/email-signature-packt/ph.png]
>  022-28328148
>
>  [https://s3-eu-west-1.amazonaws.com/email-signature-packt/web.png]
> www.packt.com
>  [https://s3-eu-west-1.amazonaws.com/email-signature-packt/msg.png]
> nish...@packt.com
> [https://s3-eu-west-1.amazonaws.com/email-signature-packt/loc.png]  Plot
> No.103, Arena House, 3rd Floor,12th Road, MIDC, Andheri (E), Mumbai -
> 400093 Mumbai .
>
>
> [https://www.facebook.com/PacktPub/?ref=br_rs]<
> https://www.facebook.com/PacktPub/?ref=br_rs>  [
> https://twitter.com/PacktPub]    [
> https://www.linkedin.com/company/packt-publishing/] <
> https://www.linkedin.com/company/packt-publishing/>   [
> https://www.youtube.com/channel/UC3VydBGBl132baPCLeDspMQ] <
> https://www.youtube.com/channel/UC3VydBGBl132baPCLeDspMQ>
>
>
>
>
>
>
>
> Packt Publishing Private Limited. Registered Address: Plot No.103, Arena
> House, 3rd Floor,12th Road, MIDC, Andheri (E), Mumbai - 400093.
> CIN:U22100MH2005PTC152766
> This E-mail is confidential. It may also be legally privileged. If you are
> not the addressee you may not copy, forward, disclose or use any part of
> it. If you have received this message in error, please delete it and all
> copies from your system and notify the sender immediately by return E-mail.
> Whilst Packt Publishing Ltd take every reasonable precaution to avoid the
> transfer of software viruses or defects that may cause damage to computer
> systems; it is the responsibility of the recipient to ensure that all
> emails and attachments received are free from infection.
> The word 'Packt'® and the Packt logo are registered trademarks which
> belong to Packt Publishing Limited.
>


Re: [Fwd: Returned post for annou...@apache.org]

2018-12-10 Thread Denis Magda
Forwarding to the dev list.

*Prachi*, could you please help us with 3-5? It requires minor updates on
the webpage.

*Sergey K.*, *Peter*, could you assist with 1 and 2?

--
Denis

On Fri, Dec 7, 2018 at 11:19 PM Nikolay Izhikov  wrote:

> Hello, Igniters.
>
> Seems that our download page doesn't meet apache requirements.
> I think we should update the page according to apache rules.
>
> Who can do it?
>
>
>
> -- Forwarded message --
> From: announce-ow...@apache.org
> To: nizhi...@apache.org
> Cc:
> Bcc:
> Date: 6 Dec 2018 13:02:57 -
> Subject: Returned post for annou...@apache.org
>
> Hi! This is the ezmlm program. I'm managing the
> annou...@apache.org mailing list.
>
> I'm sorry, your message (enclosed) was not accepted by the moderator.
> If the moderator has made any comments, they are shown below.
>
> >  >
> Sorry, but the download page does not meet requirements.
>
> 1) There is no link to the KEYS file
> https://www.apache.org/dist/ignite/KEYS
> This is necessary for validating downloaded artifacts
>
> 2) No description of how to validate downloads
>
> 3) There are links to dist.apache.org - that host is only intended for use
> during release preparation and publication.
> Use https://www.apache.org/dist/ignite/... instead.
>
> 4) There is a link to nightly builds.
> That is not allowed on a public download page, as the builds have not been
> voted on.
>
> 5) The paragraph introducing the binary artifact says:
>
> "In order to verify the release, we recommend that you download the
> official source distribution and verify the signatures of the downloaded
> files before opening them."
>
> This does not make sense in the binary section (it belongs in the source
> section and/or needs rewording).
> Nor is there any description of how to perform the verification.
>
> Please correct the page and resubmit the announce when that has been done.
>
> Sebb
> <  <
>
>
>
>
> -- Forwarded message --
> From: Nikolay Izhikov 
> To: annou...@apache.org
> Cc:
> Bcc:
> Date: Thu, 6 Dec 2018 15:02:21 +0300
> Subject: Fwd: [ANNOUNCE] Apache Ignite 2.7.0 Released
> The Apache Ignite Community is pleased to announce the release of
> Apache Ignite 2.7.0.
>
> Apache Ignite [1] is a memory-centric distributed database, caching,
> and processing platform for transactional, analytical, and streaming
> workloads delivering in-memory speeds at petabyte scale.
>
> This release introduce several major features and fixes some critical
> issues
> https://ignite.apache.org/releases/2.7.0/release_notes.html
>
> Download the latest Ignite version from here:
> https://ignite.apache.org/download.cgi
>
> Please let us know [2] if you encounter any problems.
>
> Regards,
> Nikolay Izhikov on behalf of Apache Ignite community
>
> [1] https://ignite.apache.org
> [2] https://ignite.apache.org/community/resources.html#ask
>


Ignite cloud readiness and as a managed service

2018-12-07 Thread Denis Magda
Igniters,

Presently, there are no any doubts that the private/public cloud
deployments would be dominating soon. Personally, I already see even
conservative companies moving from on-prem to clouds. That migration is
impossible w/o essential support and integrations with tools/environments
like Kubernetes, Docker, AWS, Azure, OpenShift, VMWare to name few.

Luckily, Ignite is pretty well-equipped at the moment. However, there are
some gaps which I would encourage us to highlight. What should the
community do to make Ignite 100% ready for cloud deployments the next year?
For instance, that's my list of gaps:

   - Pivotal Cloud Foundry support
   - Helm integration - https://helm.sh
   - Prometheus integration - https://prometheus.io

Moreover, some of the companies moved forward and began using Ignite as a
managed service. There is nothing we can do on the community side except
for listing where Ignite users can get the service, which might be helpful.
I'm thinking to create a webpage that would list all the known solutions
like 2 below. Any recommendations on this?

   - https://www.nexaops.com/managed-services/managed-services-apache-ignite
   - https://gridgain.cloud

--
Denis


Re: Support service on inactive cluster

2018-12-07 Thread Denis Magda
Alex,

1. Why do you need an operational service if the cluster is deactivated (no
changes to data are allowed, right)?

2. That should be covered in the existing EIP

--
Denis

On Fri, Dec 7, 2018 at 6:20 AM Alexey Kuznetsov 
wrote:

> Igniters,
>
> I would like to discuss how we can support following features for Service:
>
> 1) Made  services work on deactivated cluster. What we need for that?
> Introduce some flag on service configuration?
>
> 2) Auto start service after full cluster restart if service was started at
> runtime.
> Can we have also add some flag on service configuration?
>  And store such services the same way as we store user information for
> authentication in metastorage?
>
> What do yo think?
>
> I can create issues in JIRA if we agreed to support them.
>
> --
> Alexey Kuznetsov
>


Re: [ANNOUNCE] Apache Ignite 2.7.0 Released

2018-12-06 Thread Denis Magda
Nikolay,

Could you confirm that all of the "Post-release steps" have been completed
and you don't need help with any? For instance, there is a WebConsole
related bullet point.
https://cwiki.apache.org/confluence/display/IGNITE/Release+Process

Prachi, please release the new website pages.

Dmitriy P, there were some security related changes. Please go ahead and
uncover them.

--
Denis

On Thu, Dec 6, 2018 at 4:02 AM Nikolay Izhikov  wrote:

> Done.
>
> чт, 6 дек. 2018 г. в 14:41, Dmitriy Pavlov :
>
> > According to previous
> >
> >
> https://lists.apache.org/thread.html/4dd772a607ba89b3aaa2084a1ebcd2f4e6a9790decad4d167b2dbd73@%3Cdev.ignite.apache.org%3E
> > it
> > seems to be To: annou...@apache.org, dev ,
> > u...@ignite.apache.org
> >
> > чт, 6 дек. 2018 г. в 14:30, Nikolay Izhikov :
> >
> > > Dmitriy.
> > >
> > > Yes, we should. Please, tell me mail of this channel.
> > >
> > > чт, 6 дек. 2018 г., 14:13 Dmitriy Pavlov dpav...@apache.org:
> > >
> > > > Nikolay, thank you for your efforts!
> > > >
> > > > Should we announce a release in a special Apache-wide channel?
> > > >
> > > > чт, 6 дек. 2018 г. в 10:36, Nikolay Izhikov :
> > > >
> > > > > The Apache Ignite Community is pleased to announce the release of
> > > > > Apache Ignite 2.7.0.
> > > > >
> > > > > Apache Ignite [1] is a memory-centric distributed database,
> caching,
> > > > > and processing platform for transactional, analytical, and
> streaming
> > > > > workloads delivering in-memory speeds at petabyte scale.
> > > > >
> > > > > This release introduce several major features and fixes some
> critical
> > > > > issues
> > > > > https://ignite.apache.org/releases/2.7.0/release_notes.html
> > > > >
> > > > > Download the latest Ignite version from here:
> > > > > https://ignite.apache.org/download.cgi
> > > > >
> > > > > Please let us know [2] if you encounter any problems.
> > > > >
> > > > > Regards,
> > > > > Nikolay Izhikov on behalf of Apache Ignite community
> > > > >
> > > > > [1] https://ignite.apache.org
> > > > > [2] https://ignite.apache.org/community/resources.html#ask
> > > > >
> > > >
> > >
> >
>


Re: [RESULT] [VOTE] Apache Ignite 2.7.0 Release (RC2)

2018-12-05 Thread Denis Magda
When are going to complete the post-release steps and announce the release?
Once the binaries are available on the website we need to release the docs.


I'm ready to prepare a blog post for blogs.apache.org.

--
Denis

On Mon, Dec 3, 2018 at 11:14 PM Nikolay Izhikov  wrote:

> Sorry, Alex.
>
> I miss your +1.
> Thank you, very much for checking RC artifacts.
>
> вт, 4 дек. 2018 г., 7:10 Alexey Kuznetsov akuznet...@apache.org:
>
> > Nikolay,
> >
> > Actually 4 "+1"  binding.
> >
> > You did not count my "+1".
> >
> > :)
> >
> >
> > On Tue, Dec 4, 2018 at 4:28 AM Nikolay Izhikov 
> > wrote:
> >
> > > Igniters,
> > >
> > > Apache Ignite 2.7.0 release (RC2) has been accepted.
> > >
> > > 3 "+1" binding votes received:
> > >
> > > - Pavel Tupitsyn
> > > - Dmitriy Pavlov
> > > - Nikolay Izhikov
> > >
> > > Vote thread:
> > >
> > >
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-Apache-Ignite-2-7-0-RC2-td38788.html
> > >
> >
> >
> > --
> > Alexey Kuznetsov
> >
>


Re: [NOTICE] List created: notificati...@ignite.apache.org

2018-11-30 Thread Denis Magda
Yes, the messages have to be redirected. Talk to INFRA please.

--
Denis

On Fri, Nov 30, 2018 at 7:12 AM Dmitriy Pavlov  wrote:

> Hi Eduard,
>
> could you please issue ticket similar to
> https://issues.apache.org/jira/browse/INFRA-17032
>  but referring to our new list?
>
> It seems notifications are still sent, at least from gitbox.
>
> Thank you in advance!
>
> Sincerely,
> Dmitriy Pavlov
>
> пт, 30 нояб. 2018 г. в 17:56, Dmitriy Pavlov :
>
> > Denis, thank you.
> >
> > Is it required to request INFRA to setup redirection of GitHub/GitBox?
> >
> > Or nothing is required now and we can subscribe to the new list?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пт, 30 нояб. 2018 г. в 17:16, Denis Magda :
> >
> >> The notifications list for GIT messages has been created.
> >>
> >> -- Forwarded message -
> >> From: ASF Self-Service Platform 
> >> Date: Fri, Nov 30, 2018 at 4:30 AM
> >> Subject: [NOTICE] List created: notificati...@ignite.apache.org
> >> To: , , ASF
> Infrastructure
> >> <
> >> priv...@infra.apache.org>
> >>
> >>
> >>
> >> As requested by dmagda, the following mailing list has been created:
> >> List name: notificati...@ignite.apache.org
> >> Moderators: dpav...@apache.org,dma...@apache.org
> >> Settings: Allow subscribers to post, moderate all others
> >> Reply-To: dev@ignite.apache.org (forced)
> >> This list is public.
> >>
> >> ---
> >>
> >> The list will start accepting mail in 60 minutes from now.  If it's a
> >> public
> >> list, it will appear on https://lists.apache.org/ within a few minutes
> of
> >> the first post to it.
> >>
> >>
> >> With regards,
> >> ASF Self-Service Platform, https://selfserve.apache.org
> >> For inquiries, please contact: us...@infra.apache.org
> >>
> >
>


Fwd: [NOTICE] List created: notificati...@ignite.apache.org

2018-11-30 Thread Denis Magda
The notifications list for GIT messages has been created.

-- Forwarded message -
From: ASF Self-Service Platform 
Date: Fri, Nov 30, 2018 at 4:30 AM
Subject: [NOTICE] List created: notificati...@ignite.apache.org
To: , , ASF Infrastructure <
priv...@infra.apache.org>



As requested by dmagda, the following mailing list has been created:
List name: notificati...@ignite.apache.org
Moderators: dpav...@apache.org,dma...@apache.org
Settings: Allow subscribers to post, moderate all others
Reply-To: dev@ignite.apache.org (forced)
This list is public.

---

The list will start accepting mail in 60 minutes from now.  If it's a public
list, it will appear on https://lists.apache.org/ within a few minutes of
the first post to it.


With regards,
ASF Self-Service Platform, https://selfserve.apache.org
For inquiries, please contact: us...@infra.apache.org


Re: IEP-24: SQL Partition Pruning

2018-11-29 Thread Denis Magda
Vladimir, thanks for the extensive description. Everything is clear for me
from a user perspective.

Hope Sergi, Taras, and other SQL experts would share their feedback.

--
Denis

On Mon, Nov 26, 2018 at 6:33 AM Vladimir Ozerov 
wrote:

> Igniters,
>
> I prepared and IEP-24 [1] for so-called "partition pruning" optimization
> for our SQL engine, which will allow us to determine target nodes
> containing query data prior to query execution. We already use this
> optimization for very simple scenarios - only one expression, no JOINs.
>
> The goals of this IEP:
> 1) Extract partitions from complex expressions
> 2) Support common JOIN scenarios
> 3) Allow calculation of target partitions on thin client to allow more
> efficient request routing
> 4) Introduce monitoring capabilities to let user know whether optimization
> is applicable to specific query or not
>
> IEP covers several complex architecture questions, which will be finalized
> during actual implementation:
> 1) Rules for partition extraction from complex AND and OR expressions, as
> well as from "IN (...)", "BETWEEN ... AND ...", and range expressions
> 2) Rules for partition extraction from JOINs
> 3) Several subquery rewrite rules which will allow to apply optimization to
> certain subqueries.
>
> Also this optimization will introduce some basic building blocks
> ("co-location tree") for further improvements of our distributed joins.
>
> Will appreciate your review and comments.
>
> Vladimir.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-24%3A+SQL+Partition+Pruning
>


Re: [Result][VOTE] Creation dedicated list for github notifiacations

2018-11-29 Thread Denis Magda
A request has been submitted.

--
Denis

On Thu, Nov 29, 2018 at 11:45 AM Dmitriy Pavlov  wrote:

> Denis, could you please create a new list for Apache Ignite, e.g.
> notificati...@ignite.apache.org  ?
>
> Only PMC Chair can create a list https://infra.apache.org/
>
> Create list feature has been restricted to ASF members and PMC chairs only.
> Thank you in advance.
>
> Sincerely,
> Dmitriy Pavlov
>
> чт, 29 нояб. 2018 г. в 21:44, Eduard Shangareev <
> eduard.shangar...@gmail.com>:
>
>> Igniters,
>> The result is successful.
>>
>> No "-1".
>> 11 "+1".
>> 2 "0".
>>
>> Vote thread:
>>
>> http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-Creation-dedicated-list-for-github-notifiacations-td38485.html
>>
>


Re: Apache Ignite 2.7. Last Mile

2018-11-29 Thread Denis Magda
I think that it's not a blocker since MVCC is in the beta state and some of
the APIs might not work well with it yet.

Apart from that, are we done with the stabilization and ready to start the
vote? What blocks us from that?

--
Denis


On Thu, Nov 29, 2018 at 7:43 AM Yakov Zhdanov  wrote:

> Vladimir, can you please take a look at
> https://issues.apache.org/jira/browse/IGNITE-10376?
>
> --Yakov
>


Re: proposed design for thin client SQL management and monitoring (view running queries and kill it)

2018-11-27 Thread Denis Magda
Vladimir,

Please see inline

On Mon, Nov 19, 2018 at 8:23 AM Vladimir Ozerov 
wrote:

> Denis,
>
> I partially agree with you. But there are several problem with syntax
> proposed by you:
> 1) This is harder to implement technically - more parsing logic to
> implement. Ok, this is our internal problem, users do not care about it
> 2) User will have to consult to docs in any case
>

Two of these are not a big deal. We just need to invest more time for
development and during the design phase so that people need to consult the
docs rarely.


> 3) "nodeId" is not really node ID. For Ignite users node ID is UUID. In our
> case this is node order, and we intentionally avoided any naming here.
>

Let's use a more loose name such as "node".


> Query is just identified by a string, no more than that
> 4) Proposed syntax is more verbose and open ways for misuse. E.g. what is
> "KILL QUERY WHERE queryId=1234"?
>
> I am not 100% satisfied with both variants, but the first one looks simpler
> to me. Remember, that user will not guess query ID. Instead, he will get
> the list of running queries with some other syntax. What we need to
> understand for now is how this syntax will look like. I think that we
> should implement getting list of running queries, and only then start
> working on cancellation.
>

That's a good point. Syntax of both running and killing queires commands
should be tightly coupled. We're going to name a column if running queries
IDs somehow anyway and that name might be resued in the WHERE clause of
KILL.

Should we discuss the syntax in a separate thread?

--
Denis

>
> Vladimir.
>
>
> On Mon, Nov 19, 2018 at 7:02 PM Denis Mekhanikov 
> wrote:
>
> > Guys,
> >
> > Syntax like *KILL QUERY '25.1234'* look a bit cryptic to me.
> > I'm going to look up in documentation, which parameter goes first in this
> > query every time I use it.
> > I like the syntax, that Igor suggested more.
> > Will it be better if we make *nodeId* and *queryId *named properties?
> >
> > Something like this:
> > KILL QUERY WHERE nodeId=25 and queryId=1234
> >
> > Denis
> >
> > пт, 16 нояб. 2018 г. в 14:12, Юрий :
> >
> > > I fully agree with last sentences and can start to implement this part.
> > >
> > > Guys, thanks for your productive participate at discussion.
> > >
> > > пт, 16 нояб. 2018 г. в 2:53, Denis Magda :
> > >
> > > > Vladimir,
> > > >
> > > > Thanks, make perfect sense to me.
> > > >
> > > >
> > > > On Thu, Nov 15, 2018 at 12:18 AM Vladimir Ozerov <
> voze...@gridgain.com
> > >
> > > > wrote:
> > > >
> > > > > Denis,
> > > > >
> > > > > The idea is that QueryDetailMetrics will be exposed through
> separate
> > > > > "historical" SQL view in addition to current API. So we are on the
> > same
> > > > > page here.
> > > > >
> > > > > As far as query ID I do not see any easy way to operate on a single
> > > > integer
> > > > > value (even 64bit). This is distributed system - we do not want to
> > have
> > > > > coordination between nodes to get query ID. And coordination is the
> > > only
> > > > > possible way to get sexy "long". Instead, I would propose to form
> ID
> > > from
> > > > > node order and query counter within node. This will be (int, long)
> > > pair.
> > > > > For use convenience we may convert it to a single string, e.g.
> > > > > "[node_order].[query_counter]". Then the syntax would be:
> > > > >
> > > > > KILL QUERY '25.1234'; // Kill query 1234 on node 25
> > > > > KILL QUERY '25.*; // Kill all queries on the node 25
> > > > >
> > > > > Makes sense?
> > > > >
> > > > > Vladimir.
> > > > >
> > > > > On Wed, Nov 14, 2018 at 1:25 PM Denis Magda 
> > wrote:
> > > > >
> > > > > > Yury,
> > > > > >
> > > > > > As I understand you mean that the view should contains both
> running
> > > and
> > > > > > > finished queries. If be honest for the view I was going to use
> > just
> > > > > > queries
> > > > > > > running right now. For finished queries I thought about another
> > > view
> > > > > with
> > > > > > > another 

Re: [VOTE] Creation dedicated list for github notifiacations

2018-11-26 Thread Denis Magda
0 - don't care

--
Denis

On Mon, Nov 26, 2018 at 10:54 AM Eduard Shangareev <
eduard.shangar...@gmail.com> wrote:

> This was discussed already [1].
>
> So, I want to complete this discussion with moving outside dev-list
> GitHub-notification to dedicated list.
>
> Please start voting.
>
> +1 - to accept this change.
> 0 - you don't care.
> -1 - to decline this change.
>
> This vote will go for 72 hours.
>
> [1]
>
> http://apache-ignite-developers.2346864.n4.nabble.com/Time-to-remove-automated-messages-from-the-devlist-td37484i20.html
>


Re: [IMPORTANT] Future of Binary Objects

2018-11-21 Thread Denis Magda
Vladimir,

Could you educate me a little bit, why the current format is bad for SQL
and why another one is more suitable?

Also, if we introduce the new format then why would we keep the binary one?
Is the new format just a next version of the binary one.

2.3) Remove restrictions on changing field type
> I do not know why we did that in the first place. This restriction prevents
> type evolution and confuses users.


That is a hot requirement shared by those who use Ignite SQL in production.
+1.

--
Denis

On Mon, Nov 19, 2018 at 11:05 PM Vladimir Ozerov 
wrote:

> Igniters,
>
> It is very likely that Apache Ignite 3.0 will be released next year. So we
> need to start thinking about major product improvements. I'd like to start
> with binary objects.
>
> Currently they are one of the main limiting factors for the product. They
> are fat - 30+ bytes overhead on average, high TCO of Apache Ignite
> comparing to other vendors. They are slow - not suitable for SQL at all.
>
> I would like to ask all of you who worked with binary objects to share your
> feedback and ideas, so that we understand how they should look like in AI
> 3.0. This is a brain storm - let's accumulate ideas first and minimize
> critics. Then we will work on ideas in separate topics.
>
> 1) Historical background
>
> BO were implemented around 2014 (Apache Ignite 1.5) when we started working
> on .NET and CPP clients. During design we had several ideas in mind:
> - ability to read object fields in O(1) without deserialization
> - interoperabillty between Java, .NET and CPP.
>
> Since then a number of other concepts were mixed to the cocktail:
> - Affinity key fields
> - Strict typing for existing fields (aka metadata)
> - Binary Object as storage format
>
> 2) My proposals
>
> 2.1) Introduce "Data Row Format" interface
> Binary Objects are terrible candidates for storage. Too fat, too slow.
> Efficient storage typically has <10 bytes overhead per row (no metadata, no
> length, no hash code, etc), allow supper-fast field access, support
> different string formats (ASCII, UTF-8, etc), support different temporal
> types (date, time, timestamp, timestamp with timezone, etc), and store
> these types as efficiently as possible.
>
> What we need is to introduce an interface which will convert a pair of
> key-value objects into a row. This row will be used to store data and to
> get fields from it. Care about memory consumption, need SQL and strict
> schema - use one format. Need flexibility and prefer key-value access - use
> another format which will store binary objects unchanged (current
> behavior).
>
> interface DataRowFormat {
> DataRow create(Object key, Object value); // primitives or binary
> objects
> DataRowMetadata metadata();
> }
>
> 2.2) Remove affinity field from metadata
> Affinity rules are governed by cache, not type. We should remove
> "affintiyFieldName" from metadata.
>
> 2.3) Remove restrictions on changing field type
> I do not know why we did that in the first place. This restriction prevents
> type evolution and confuses users.
>
> 2.4) Use bitmaps for "null" and default values and for fixed-length fields,
> put fixed-length fields before variable-length.
> Motivation: to save space.
>
> What else? Please share your ideas.
>
> Vladimir.
>


Re: proposed realization KILL QUERY command

2018-11-21 Thread Denis Magda
Vladimir,

All of the alternatives are reminiscent of mathematical operations. Don't
look like a SQL command. What if we use a SQL approach introducing named
parameters:

KILL QUERY query_id=10 [AND node_id=5]

--
Denis

On Wed, Nov 21, 2018 at 4:11 AM Vladimir Ozerov 
wrote:

> Denis,
>
> Space is bad candidate because it is a whitespace. Without whitespaces we
> can have syntax without quotes at all. Any non-whitespace delimiter will
> work, though:
>
> KILL QUERY 45.1
> KILL QUERY 45-1
> KILL QUERY 45:1
>
> On Wed, Nov 21, 2018 at 3:06 PM Юрий  wrote:
>
> > Denis,
> >
> > Let's consider parameter of KILL QUERY just a string with some query id,
> > without any meaning for user. User just need to get the id and pass as
> > parameter to KILL QUERY command.
> >
> > Even if query is distributed it have single query id from user
> perspective
> > and will killed on all nodes. User just need to known one global query
> id.
> >
> > How it can works.
> > 1)SELECT * from running_queries
> > result is
> >  query_id | node_id
> >   | sql   | schema_name | connection_id | duration
> > 123.33 | e0a69cb8-a1a8-45f6-b84d-ead367a0   | SELECT ...  | ...
> >   |   22 | 23456
> > 333.31 | aaa6acb8-a4a5-42f6-f842-ead111b00020 | UPDATE...  | ...
> >   |  321| 346
> > 2) KILL QUERY '123.33'
> >
> > So, user need select query_id from running_queries view and use it for
> KILL
> > QUERY command.
> >
> > I hope it became clearer.
> >
> >
> >
> > ср, 21 нояб. 2018 г. в 02:11, Denis Magda :
> >
> > > Folks,
> > >
> > > The decimal syntax is really odd - KILL QUERY
> > > '[node_order].[query_counter]'
> > >
> > > Confusing, let's use a space to separate parameters.
> > >
> > > Also, what if I want to halt a specific query with certain ID? Don't
> know
> > > the node number, just know that the query is distributed and runs
> across
> > > several machines. Sounds like the syntax still should consider
> > > [node_order/id] as an optional parameter.
> > >
> > > Probably, if you explain to me how an end user will use this command
> from
> > > the very beginning (how do I look for a query id and node id, etc) then
> > the
> > > things get clearer.
> > >
> > > --
> > > Denis
> > >
> > > On Tue, Nov 20, 2018 at 1:03 AM Юрий 
> > wrote:
> > >
> > > > Hi Vladimir,
> > > >
> > > > Thanks for your suggestion to use MANAGEMENT_POOL for processing
> > > > cancellation requests.
> > > >
> > > > About your questions.
> > > > 1) I'm going to implements SQL view to provide list of running
> queries.
> > > The
> > > > SQL VIEW has been a little bit discussed earlier. Proposed name is
> > > > *running_queries* with following columns: query_id, node_id, sql,
> > > > schema_name, connection_id, duration. Currently most of the
> information
> > > can
> > > > be  retrieved through cache API, however it doesn't matter, any case
> we
> > > > need to expose SQL VIEW. Seem's you are right - the part should be
> > > > implemented firstly.
> > > > 2) Fully agree that we need to support all kind of SQL queries
> > > > (SLECT/DML/DDL, transactional, non transnational, local,
> distributed).
> > I
> > > > definitely sure that it will possible for all of above, however I'm
> not
> > > > sure about DDL - need to investigate it deeper. Also need to
> understand
> > > > that canceled DML operation can lead to partially updated data for
> non
> > > > transational caches.
> > > >
> > > >
> > > >
> > > > пн, 19 нояб. 2018 г. в 19:17, Vladimir Ozerov  >:
> > > >
> > > > > Hi Yuriy,
> > > > >
> > > > > I think we can use MANAGEMENT_POOL for this. It is already used for
> > > some
> > > > > internal Ignite tasks, and it appears to be a good candidate to
> > process
> > > > > cancel requests.
> > > > >
> > > > > But there are several things which are not clear enough for me at
> the
> > > > > moment:
> > > > > 1) How user is going to get the list of running queries in the
> first
> > > > place?
> > > > > Do we already have any SQL commands/vi

Re: proposed realization KILL QUERY command

2018-11-20 Thread Denis Magda
Folks,

The decimal syntax is really odd - KILL QUERY '[node_order].[query_counter]'

Confusing, let's use a space to separate parameters.

Also, what if I want to halt a specific query with certain ID? Don't know
the node number, just know that the query is distributed and runs across
several machines. Sounds like the syntax still should consider
[node_order/id] as an optional parameter.

Probably, if you explain to me how an end user will use this command from
the very beginning (how do I look for a query id and node id, etc) then the
things get clearer.

--
Denis

On Tue, Nov 20, 2018 at 1:03 AM Юрий  wrote:

> Hi Vladimir,
>
> Thanks for your suggestion to use MANAGEMENT_POOL for processing
> cancellation requests.
>
> About your questions.
> 1) I'm going to implements SQL view to provide list of running queries. The
> SQL VIEW has been a little bit discussed earlier. Proposed name is
> *running_queries* with following columns: query_id, node_id, sql,
> schema_name, connection_id, duration. Currently most of the information can
> be  retrieved through cache API, however it doesn't matter, any case we
> need to expose SQL VIEW. Seem's you are right - the part should be
> implemented firstly.
> 2) Fully agree that we need to support all kind of SQL queries
> (SLECT/DML/DDL, transactional, non transnational, local, distributed). I
> definitely sure that it will possible for all of above, however I'm not
> sure about DDL - need to investigate it deeper. Also need to understand
> that canceled DML operation can lead to partially updated data for non
> transational caches.
>
>
>
> пн, 19 нояб. 2018 г. в 19:17, Vladimir Ozerov :
>
> > Hi Yuriy,
> >
> > I think we can use MANAGEMENT_POOL for this. It is already used for some
> > internal Ignite tasks, and it appears to be a good candidate to process
> > cancel requests.
> >
> > But there are several things which are not clear enough for me at the
> > moment:
> > 1) How user is going to get the list of running queries in the first
> place?
> > Do we already have any SQL commands/views to get this information?
> > 2) We need to ensure that KILL command will be processed properly by all
> > kinds of SQL queries - SELECT/DML/DDL, non-transactional or
> transactional,
> > local queries and distributed queries. Will we be able to support all
> these
> > modes?
> >
> > Vladimir.
> >
> > On Mon, Nov 19, 2018 at 6:37 PM Юрий 
> wrote:
> >
> > > Hi Igniters,
> > >
> > > Earlier we agreed about syntax KILL QUERY
> '[node_order].[query_counter]',
> > > e.g. KILL QUERY '25.123' for single query  or KILL QUERY '25.*' for all
> > > queries on the node. Which is part of IEP-29
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-29%3A+SQL+management+and+monitoring
> > > >
> > > .
> > >
> > > Now I want to discuss internal realization of KILL query feature.
> > >
> > > My current vision is following:
> > > After parsing, Ignite create KILL query command with two parameters:
> > > nodeOrderId, nodeQryId. To determine that need to kill all queries on a
> > > node we can use negative value of query id, due to qry id always have
> > > positive values.
> > > The command process at IgniteH2Indexing as native command.
> > > By nodeOrderId we find node which initial for the query and send to the
> > > node new GridQueryKillRequest with nodeQryId to TOPIC_QUERY with not
> > QUERY
> > > POOL executor.
> > > At GridReduceQueryExecutor we add support of processing new
> > > GridQueryKillRequest
> > > which just run already exists cancelQueries method with given qryId or
> > with
> > > all qryIds which currently running at the node in case at initial KILL
> > > QUERY parameters used star symbol.
> > >
> > > I have a doubt which of thread pool we should use to process
> > > GridQueryKillRequest.
> > > My opinion it shouldn't be QUERY pool, due to the pool can be fully
> used
> > by
> > > executing queries, it such case we can't cancel query immediately. May
> we
> > > use one of already existed pool or create new one? Or may be I'm
> mistaken
> > > and it should use QUERY pool.
> > >
> > > What do you think about proposed plan of implementation?
> > >
> > > And please give comments about which of thread pool will be better to
> use
> > > for kill query requests. It's small, but really important part of the
> > > realization.
> > >
> > >
> > > Thanks.
> > >
> > >
> > > --
> > > Живи с улыбкой! :D
> > >
> >
>
>
> --
> Живи с улыбкой! :D
>


New PMC Member: Nikolay Izhikov

2018-11-20 Thread Denis Magda
The Project Management Committee (PMC) for Apache Ignite
has invited Nikolay Izhikov to become a PMC member and we are pleased
to announce that he has accepted.

Being a PMC member enables assistance with the management
and to guide the direction of the project.

Please welcome Nikolay! Great progress so far and we believe that more to
come.

Denis,
On behalf of Ignite PMC


Re: Spring Update and delete queries

2018-11-20 Thread Denis Magda
Excellent,

Please get to know basics about Ignite's contribution process, create a
ticket in JIRA and propose a design here.

--
Denis



On Tue, Nov 20, 2018 at 11:02 AM Jonathan Camargo 
wrote:

> Hi,
>
> Sure, I'm ready to start working on the DELETE/UPDATE.
>
> El mar., 20 nov. 2018 a las 13:10, Denis Magda ()
> escribió:
>
> > Hello Jonathan,
> >
> > Thanks for the feedback. Are you ready to work on missing DELETE/UPDATE
> > capabilities? Our community experts can guide you and do reviews.
> >
> > --
> > Denis
> >
> > On Tue, Nov 20, 2018 at 7:34 AM Jonathan Camargo 
> > wrote:
> >
> > > Hi community,
> > >
> > > First let me introduce myself, My name is Jonathan Camargo, I'm a Java
> > > developer at a software outsourcing company, I'm currently working for
> a
> > > legacy project.
> > >
> > > I'm currently learning new technologies and concepts such as IMDG, I
> have
> > > been reading concepts and code about that mainly because I have find
> this
> > > concept fascinating.
> > >
> > > On the test project I'm working on I am using Spring Boot 2, Ignite
> > Spring
> > > Data 2.0 and a H2 DB (It could be any other but seems easier this way)
> > and
> > > aside from the already documented issues I have find that the
> > > IgniteRepository does not support the Update and Delete queries, on the
> > > code even is throwing a controlled exception:
> > >
> > > if (parts.isDelete())
> > > throw new UnsupportedOperationException("DELETE clause is
> not
> > > supported now.");
> > > else {
> > > sql.append("SELECT ");
> > >
> > > if (parts.isDistinct())
> > > throw new UnsupportedOperationException("DISTINCT
> clause
> > in
> > > not supported.");
> > >
> > > And I would like to point that the lib is validating if if is going to
> > > generate the query or if it is going to use the @Query SQL that is
> being
> > > sent, and that SQL is executed through the SqlFieldsQuery of the core
> > lib.
> > >
> > > So my suggestion would be to add the Update and Delete query to the
> > > validation so those functions will run over the core.
> > >
> > > I would not like to Inject the Ignite object, get or create the cache
> and
> > > run those statements over there since that way the Ignite Repository
> > would
> > > not be needed and I would prefer just to remove the Ignite Spring lib
> and
> > > work directly with the client and the server on other applications.
> > >
> > > If is there any issue I could help with I would be there.
> > >
> > > And I would like to know if is there any filter on Jira over the issues
> > > that I could look.
> > >
> > > Thanks.
> > >
> > > --
> > > Jonathan Giovanny Camargo Sanabria
> > >
> >
>
>
> --
> Jonathan Giovanny Camargo Sanabria
>


Re: Spring Update and delete queries

2018-11-20 Thread Denis Magda
Hello Jonathan,

Thanks for the feedback. Are you ready to work on missing DELETE/UPDATE
capabilities? Our community experts can guide you and do reviews.

--
Denis

On Tue, Nov 20, 2018 at 7:34 AM Jonathan Camargo 
wrote:

> Hi community,
>
> First let me introduce myself, My name is Jonathan Camargo, I'm a Java
> developer at a software outsourcing company, I'm currently working for a
> legacy project.
>
> I'm currently learning new technologies and concepts such as IMDG, I have
> been reading concepts and code about that mainly because I have find this
> concept fascinating.
>
> On the test project I'm working on I am using Spring Boot 2, Ignite Spring
> Data 2.0 and a H2 DB (It could be any other but seems easier this way) and
> aside from the already documented issues I have find that the
> IgniteRepository does not support the Update and Delete queries, on the
> code even is throwing a controlled exception:
>
> if (parts.isDelete())
> throw new UnsupportedOperationException("DELETE clause is not
> supported now.");
> else {
> sql.append("SELECT ");
>
> if (parts.isDistinct())
> throw new UnsupportedOperationException("DISTINCT clause in
> not supported.");
>
> And I would like to point that the lib is validating if if is going to
> generate the query or if it is going to use the @Query SQL that is being
> sent, and that SQL is executed through the SqlFieldsQuery of the core lib.
>
> So my suggestion would be to add the Update and Delete query to the
> validation so those functions will run over the core.
>
> I would not like to Inject the Ignite object, get or create the cache and
> run those statements over there since that way the Ignite Repository would
> not be needed and I would prefer just to remove the Ignite Spring lib and
> work directly with the client and the server on other applications.
>
> If is there any issue I could help with I would be there.
>
> And I would like to know if is there any filter on Jira over the issues
> that I could look.
>
> Thanks.
>
> --
> Jonathan Giovanny Camargo Sanabria
>


Re: Network partitioning

2018-11-19 Thread Denis Magda
Hi,

Ignite comes with partition loss policies that might be useful:
https://apacheignite.readme.io/docs/partition-loss-policies

As for the network segmentation, check out GridGain that provides this for
Ignite users.

Denis


On Sat, Nov 17, 2018 at 7:39 AM Vj Anand 
wrote:

> Hi Ignite Developers,
>
> Here is the scenario that I am trying to  mitigate – I have a two node
> ignite cluster as cache; I am using Partitioning as the cache policy. I
> need to address network partitioning and avoid the split-brain scenario.
> What are the options?
> Are there hooks into the Ignite cache that I can use to be notified when,
> there is a partition, or when nodes rejoin? Any pointers? Code Snippets? I
> am trying to avoid external quorum mechanism.
>
> Thanks
> VJ
>


Re: Disk page compression for Ignite persistent store

2018-11-19 Thread Denis Magda
Hi Sergi,

Didn't know you were cooking this dish in the background ) Excellent.  Just
to be sure, that's part of this IEP, right?
https://cwiki.apache.org/confluence/display/IGNITE/IEP-20%3A+Data+Compression+in+Ignite#IEP-20:DataCompressioninIgnite-Withoutin-memorycompression

Since we can release only full file system blocks which are typically 4k
> size, user must configure page size to be at least multiple FS blocks, e.g.
> 8k or 16k. It also means that max compression ratio here is fsBlockSize /
> pageSize = 4k / 16k = 0.25


How to we handle the case if the page size is not a multiple of 4K? What is
the most optimal page size if the user wants to get the best compression?
Probably, we can adjust the default page size automatically if it's a clean
deployment.

There will be 2 new properties on CacheConfiguration
> (setDiskPageCompression and setDiskPageCompressionLevel) to setup disk page
> compression.


How about setting it at DataRegionConfiguration level as well so that it's
applied for all the caches/tables from there?

--
Denis

On Mon, Nov 19, 2018 at 2:01 AM Sergi Vladykin 
wrote:

> Folks,
>
> I've implemented page compression for persistent store and going to merge
> it to master.
>
> https://github.com/apache/ignite/pull/5200
>
> Some design notes:
>
> It employs "hole punching" approach, it means that the pages are kept
> uncompressed in memory,
> but when they get written to disk, they will be compressed and all the
> extra file system blocks for the page will be released. Thus the storage
> files become sparse.
>
> Right now we will support 4 compression methods: ZSTD, LZ4, SNAPPY and
> SKIP_GARBAGE. All of them are self-explaining except SKIP_GARBAGE, which
> basically just takes only meaningful data from half-filled pages but does
> not apply any compression. It is easy to add more if needed.
>
> Since we can release only full file system blocks which are typically 4k
> size, user must configure page size to be at least multiple FS blocks, e.g.
> 8k or 16k. It also means that max compression ratio here is fsBlockSize /
> pageSize = 4k / 16k = 0.25
>
> It is possible to enable compression for existing databases if they were
> configured for large enough page size. In this case pages will be written
> to disk in compressed form when updated, and the database will become
> compressed gradually.
>
> There will be 2 new properties on CacheConfiguration
> (setDiskPageCompression and setDiskPageCompressionLevel) to setup disk page
> compression.
>
> Compression dictionaries are not supported at the time, but may in the
> future. IMO it should be added as a separate feature if needed.
>
> The only supported platform for now is Linux. Since all popular file
> systems support sparse files, it must be  relatively easy to support more
> platforms.
>
> Please take a look and provide your thoughts and suggestions.
>
> Thanks!
>
> Sergi
>


Re: Apache Ignite 2.7. Last Mile

2018-11-18 Thread Denis Magda
 > > > > > > > > assignment
> > > > > > > > > > > support.
> > > > > > > > > > > IGNITE-9985 Igor SeliverstovMVCC TX: fix
> backup mappings
> > > > > > > > > > > IGNITE-9982 Ivan Pavlukhin  SQLLine: can't
> run with
> > > > > > > >
> > > > > > > > option
> > > > > > > > > > > --autoCommit=false or true
> > > > > > > > > > > IGNITE-10010Unassigned  Node halted if
> table was
> > > > > > > > >
> > > > > > > > > dropped
> > > > > > > > > > > IGNITE-10013Unassigned  Node restart
> may lead to NPE
> > > > > > > >
> > > > > > > > in
> > > > > > > > > > > GridDhtPartitionsExchangeFuture
> > > > > > > > > > > IGNITE-9996 Unassigned  Investigate
> possible
> > > > > > > > >
> > > > > > > > > performance
> > > > > > > > > > > drop in FSYNC mode for ignite-2.7 compared to
> ignite-2.6
> > > > > > > > > > > IGNITE-10007Unassigned  Deactivation
> hangs if an open
> > > > > > > > > > > transaction exists
> > > > > > > > > > > IGNITE-10004Unassigned  Parse error
> leads to leave
> > > > > > > >
> > > > > > > > the
> > > > > > > > > > > transaction
> > > > > > > > > > >
> > > > > > > > > > > В Ср, 24/10/2018 в 12:30 +0300, Nikolay Izhikov пишет:
> > > > > > > > > > > > Hello, Igniters.
> > > > > > > > > > > >
> > > > > > > > > > > > We have 3 ticket mapped to 2.7 today:
> > > > > > > > > > > >
> > > > > > > > > > > > Igor Seliverstov - IGNITE-9892 - MVCC: Exchange
> hangs on mvcc
> > > > > > > > > > >
> > > > > > > > > > > coordinator fail
> > > > > > > > > > > > Roman Kondakov   - IGNITE-9828 - MVCC: Continuous
> query failover.
> > > > > > > > > > > > Roman Kondakov   - IGNITE-9928 - MVCC TX: Bug in SQL
> query mapping.
> > > > > > > > > > > >
> > > > > > > > > > > > В Вт, 23/10/2018 в 15:01 +0300, Nikolay Izhikov
> пишет:
> > > > > > > > > > > > > Hello, Dmitriy.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm OK with including this patch to 2.7.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Can you ensure it completely fix the issue?
> > > > > > > > > > > > > I left comment in ticket.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > >
> > > > > > > >
> https://issues.apache.org/jira/browse/IGNITE-9854?focusedCommentId=16660516=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16660516
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > В Вт, 23/10/2018 в 13:10 +0300, Dmitriy Govorukhin
> пишет:
> > > > > > > > > > > > > > Nikolay,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have an issue which I want to include in 2.7
> release,
> > > > > > > > > > > > > >
> https://issues.apache.org/jira/browse/IGNITE-9854
> > > > > > > > > > > > > > It is a very small fix but very important, it
> protects us from
> > > > > > > > >
> > > > > > > > > NPE
> > > > > > > > > > > in some
> > > > > > > > > > > > > > rac

Re: proposed design for thin client SQL management and monitoring (view running queries and kill it)

2018-11-15 Thread Denis Magda
Vladimir,

Thanks, make perfect sense to me.


On Thu, Nov 15, 2018 at 12:18 AM Vladimir Ozerov 
wrote:

> Denis,
>
> The idea is that QueryDetailMetrics will be exposed through separate
> "historical" SQL view in addition to current API. So we are on the same
> page here.
>
> As far as query ID I do not see any easy way to operate on a single integer
> value (even 64bit). This is distributed system - we do not want to have
> coordination between nodes to get query ID. And coordination is the only
> possible way to get sexy "long". Instead, I would propose to form ID from
> node order and query counter within node. This will be (int, long) pair.
> For use convenience we may convert it to a single string, e.g.
> "[node_order].[query_counter]". Then the syntax would be:
>
> KILL QUERY '25.1234'; // Kill query 1234 on node 25
> KILL QUERY '25.*; // Kill all queries on the node 25
>
> Makes sense?
>
> Vladimir.
>
> On Wed, Nov 14, 2018 at 1:25 PM Denis Magda  wrote:
>
> > Yury,
> >
> > As I understand you mean that the view should contains both running and
> > > finished queries. If be honest for the view I was going to use just
> > queries
> > > running right now. For finished queries I thought about another view
> with
> > > another set of fields which should include I/O related ones. Is it
> works?
> >
> >
> > Got you, so if only running queries are there then your initial proposal
> > makes total sense. Not sure we need a view of the finished queries. It
> will
> > be possible to analyze them through the updated DetailedMetrics approach,
> > won't it?
> >
> > For "KILL QUERY node_id query_id"  node_id required as part of unique key
> > > of query and help understand Ignite which node start the distributed
> > query.
> > > Use both parameters will allow cheap generate unique key across all
> > nodes.
> > > Node which started a query can cancel it on all nodes participate
> nodes.
> > > So, to stop any queries initially we need just send the cancel request
> to
> > > node who started the query. This mechanism is already in Ignite.
> >
> >
> > Can we locate node_id behind the scenes if the user supplies query_id
> only?
> > A query record in the view already contains query_id and node_id and it
> > sounds like an extra work for the user to fill in all the details for us.
> > Embed node_id into query_id if you'd like to avoid extra network hops for
> > query_id to node_id mapping.
> >
> > --
> > Denis
> >
> > On Wed, Nov 14, 2018 at 1:04 AM Юрий 
> wrote:
> >
> > > Denis,
> > >
> > > Under the hood 'time' will be as startTime, but for system view I
> planned
> > > use duration which will be simple calculated as now - startTime. So,
> > there
> > > is't a performance issue.
> > > As I understand you mean that the view should contains both running and
> > > finished queries. If be honest for the view I was going to use just
> > queries
> > > running right now. For finished queries I thought about another view
> with
> > > another set of fields which should include I/O related ones. Is it
> works?
> > >
> > > For "KILL QUERY node_id query_id"  node_id required as part of unique
> key
> > > of query and help understand Ignite which node start the distributed
> > query.
> > > Use both parameters will allow cheap generate unique key across all
> > nodes.
> > > Node which started a query can cancel it on all nodes participate
> nodes.
> > > So, to stop any queries initially we need just send the cancel request
> to
> > > node who started the query. This mechanism is already in Ignite.
> > >
> > > Native SQL APIs will automatically support the futures after
> implementing
> > > for thin clients. So we are good here.
> > >
> > >
> > >
> > > вт, 13 нояб. 2018 г. в 18:52, Denis Magda :
> > >
> > > > Yury,
> > > >
> > > > Please consider the following:
> > > >
> > > >- If we record the duration instead of startTime, then the former
> > has
> > > to
> > > >be updated frequently - sounds like a performance red flag. Should
> > we
> > > > store
> > > >startTime and endTime instead? This way a query record will be
> > updated
> > > >twice - when the query is started and terminated.
> > > >- In the IEP you've mentioned I/O re

Re: Time to remove automated messages from the devlist?

2018-11-15 Thread Denis Magda
Let's move git notifications to a separate list. As for JIRA, not sure it
bothers me, it took me several minutes to set up all the filters to spread
the messages out across specific folders. Otherwise, some of us might
ignore subscribing to jira-list and would miss notifications when their
input is needed.

--
Denis

On Thu, Nov 15, 2018 at 12:03 PM Vladimir Ozerov 
wrote:

> Dmitry,
>
> I am not referring to some "authoritative ASF member" as a guide for us. We
> are on our own. What I meant is that at some point in time we were pointed
> to an idea, that tons of automated messages has nothing to do with healthy
> community. Which seems pretty reasonable to me.
>
> On Thu, Nov 15, 2018 at 10:15 PM Dmitriy Pavlov 
> wrote:
>
> > What incubator mentor do you refer to? Incubator member are asf members
> as
> > well.
> >
> > I was involved at least to 3 discussions at the list started from Jira
> > issue created.
> >
> > If others were not involved, it do not convince me its is not useful to
> > keep forwarding.
> >
> > чт, 15 нояб. 2018 г., 21:23 Vladimir Ozerov :
> >
> > > Dmitry,
> > >
> > > What Apache member do you refer to?
> > >
> > > чт, 15 нояб. 2018 г. в 21:10, Dmitriy Pavlov :
> > >
> > > > How do you know what to watch if new tickets are not forwarded?
> > > >
> > > > Again, PRs are ok to remove since it is duplicate to jira, but jira
> > > removal
> > > > does not make any sense for me.
> > > >
> > > > Com dev folks instead suggest to forward all comments and all
> activity
> > > from
> > > > github to the list. So if Apache member will confirm it is not useful
> > to
> > > > allow dev. list watchers see new issues on the list we can continue
> > > > discussion. Openness is needed not for veterans but for all community
> > > > members and users who is subscribed to the list.
> > > >
> > > > чт, 15 нояб. 2018 г., 21:00 Pavel Tupitsyn :
> > > >
> > > > > Personal emails for _watched_ JIRA tickets are very useful.
> > > > > Emails to everyone are not.
> > > > >
> > > > > +1 for separate mailing list for all automated emails.
> > > > > I don't think we can avoid automated emails completely, but dev
> list
> > > > should
> > > > > be human-only.
> > > > > So separate list is the only way.
> > > > >
> > > > >
> > > > > On Thu, Nov 15, 2018 at 8:11 PM Vladimir Ozerov <
> > voze...@gridgain.com>
> > > > > wrote:
> > > > >
> > > > > > Completely agree with Denis. Tons of generated messages and
> > community
> > > > > > health are not relevant. Currently we obviously have too much
> > tickets
> > > > and
> > > > > > too little communications. This is bad. But whether we accumulate
> > > > > generated
> > > > > > stuff here or in some other place is not important at all,
> provided
> > > > that
> > > > > we
> > > > > > can point dev-list readers to JIRA channel. And as far as
> generated
> > > > > stuff,
> > > > > > this was one of very serious concerns of our mentors during
> > > incubation
> > > > > > phase - too many tickets, too little real communications.
> Splitting
> > > > > message
> > > > > > flows will help us understand where we are.
> > > > > >
> > > > > > And another very interesting thing is how PMCs treat all these
> > > > messages -
> > > > > > they ignore them. When I come with that problem, one PMC proposed
> > > > > solution
> > > > > > - "just filter them like I do". Then I, another PMC, answered -
> "I
> > do
> > > > not
> > > > > > know how to filter them". Finally, third PMC, who also filters
> > these
> > > > > > messages, helped me create proper filter in GMail.
> > > > > >
> > > > > > Isn't it demonstrative enough that so many PMC, who are expected
> to
> > > > > > understand project very well and follow a lot of activities, find
> > it
> > > > > useful
> > > > > > to *remove* JIRA emails from their inboxes in order to ... well
> ...
> > > > > > understand what is going on. If Ignite veterans do not find these
> > > > > generated
> > > > > > emails useful, then I do not know who else can benefit from them.
> > > > > >
> > > > > > On Thu, Nov 15, 2018 at 7:06 PM Denis Mekhanikov <
> > > > dmekhani...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Dmitriy,
> > > > > > >
> > > > > > > > I do believe Igniters can set up a filter.
> > > > > > > I doesn't mean we should make them do it.
> > > > > > >
> > > > > > > How do JIRA messages help?
> > > > > > > If you want do discuss something – write to dev list.
> > > > > > > If you want a code review – write to dev list.
> > > > > > > If you have an announcement – write to dev list.
> > > > > > > I don't see, how JIRA messages can replace any of these points.
> > > > > > >
> > > > > > > Literally nobody ever answered a message from JIRA bot.
> > > > > > > I think, that only watchers of JIRA tickets should be notified
> > > about
> > > > > > > updates.
> > > > > > > There is no point in sending messages to everyone.
> > > > > > >
> > > > > > > Denis
> > > > > > >
> > > > > > > чт, 15 нояб. 2018 г. в 18:50, Dmitriy Pavlov <
> 

Re: proposed design for thin client SQL management and monitoring (view running queries and kill it)

2018-11-14 Thread Denis Magda
Yury,

As I understand you mean that the view should contains both running and
> finished queries. If be honest for the view I was going to use just queries
> running right now. For finished queries I thought about another view with
> another set of fields which should include I/O related ones. Is it works?


Got you, so if only running queries are there then your initial proposal
makes total sense. Not sure we need a view of the finished queries. It will
be possible to analyze them through the updated DetailedMetrics approach,
won't it?

For "KILL QUERY node_id query_id"  node_id required as part of unique key
> of query and help understand Ignite which node start the distributed query.
> Use both parameters will allow cheap generate unique key across all nodes.
> Node which started a query can cancel it on all nodes participate nodes.
> So, to stop any queries initially we need just send the cancel request to
> node who started the query. This mechanism is already in Ignite.


Can we locate node_id behind the scenes if the user supplies query_id only?
A query record in the view already contains query_id and node_id and it
sounds like an extra work for the user to fill in all the details for us.
Embed node_id into query_id if you'd like to avoid extra network hops for
query_id to node_id mapping.

--
Denis

On Wed, Nov 14, 2018 at 1:04 AM Юрий  wrote:

> Denis,
>
> Under the hood 'time' will be as startTime, but for system view I planned
> use duration which will be simple calculated as now - startTime. So, there
> is't a performance issue.
> As I understand you mean that the view should contains both running and
> finished queries. If be honest for the view I was going to use just queries
> running right now. For finished queries I thought about another view with
> another set of fields which should include I/O related ones. Is it works?
>
> For "KILL QUERY node_id query_id"  node_id required as part of unique key
> of query and help understand Ignite which node start the distributed query.
> Use both parameters will allow cheap generate unique key across all nodes.
> Node which started a query can cancel it on all nodes participate nodes.
> So, to stop any queries initially we need just send the cancel request to
> node who started the query. This mechanism is already in Ignite.
>
> Native SQL APIs will automatically support the futures after implementing
> for thin clients. So we are good here.
>
>
>
> вт, 13 нояб. 2018 г. в 18:52, Denis Magda :
>
> > Yury,
> >
> > Please consider the following:
> >
> >- If we record the duration instead of startTime, then the former has
> to
> >be updated frequently - sounds like a performance red flag. Should we
> > store
> >startTime and endTime instead? This way a query record will be updated
> >twice - when the query is started and terminated.
> >- In the IEP you've mentioned I/O related fields that should help to
> >grasp why a query runs that slow. Should they be stored in this view?
> >- "KILL QUERY query_id" is more than enough. Let's not add "node_id"
> >unless it's absolutely required. Our queries are distributed and
> > executed
> >across several nodes that's why the node_id parameter is redundant.
> >- This API needs to be supported across all our interfaces. We can
> start
> >with JDBC/ODBC and thin clients and then support for the native SQL
> APIs
> >(Java, Net, C++)
> >- Please share examples of SELECTs in the IEP that would show how to
> >find long running queries, queries that cause a lot of I/O troubles.
> >
> > --
> > Denis
> >
> > On Tue, Nov 13, 2018 at 1:15 AM Юрий 
> wrote:
> >
> > > Igniters,
> > >
> > > Some comments for my original email's.
> > >
> > > The proposal related to part of IEP-29
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-29%3A+SQL+management+and+monitoring
> > > >
> > > .
> > >
> > > What purpose are we pursuing of the proposal?
> > > We want to be able check which queries running right now through thin
> > > clients. Get some information related to the queries and be able to
> > cancel
> > > a query if it required for some reasons.
> > > So, we need interface to get a running queries. For the goal we propose
> > > running_queries system view. The view contains unique query identifier
> > > which need to pass to kill query command to cancel the query.
> > >
> > > What do you think about fields of the running queries view? May be some
> > > us

Re: proposed design for thin client SQL management and monitoring (view running queries and kill it)

2018-11-13 Thread Denis Magda
Yury,

Please consider the following:

   - If we record the duration instead of startTime, then the former has to
   be updated frequently - sounds like a performance red flag. Should we store
   startTime and endTime instead? This way a query record will be updated
   twice - when the query is started and terminated.
   - In the IEP you've mentioned I/O related fields that should help to
   grasp why a query runs that slow. Should they be stored in this view?
   - "KILL QUERY query_id" is more than enough. Let's not add "node_id"
   unless it's absolutely required. Our queries are distributed and executed
   across several nodes that's why the node_id parameter is redundant.
   - This API needs to be supported across all our interfaces. We can start
   with JDBC/ODBC and thin clients and then support for the native SQL APIs
   (Java, Net, C++)
   - Please share examples of SELECTs in the IEP that would show how to
   find long running queries, queries that cause a lot of I/O troubles.

--
Denis

On Tue, Nov 13, 2018 at 1:15 AM Юрий  wrote:

> Igniters,
>
> Some comments for my original email's.
>
> The proposal related to part of IEP-29
> <
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-29%3A+SQL+management+and+monitoring
> >
> .
>
> What purpose are we pursuing of the proposal?
> We want to be able check which queries running right now through thin
> clients. Get some information related to the queries and be able to cancel
> a query if it required for some reasons.
> So, we need interface to get a running queries. For the goal we propose
> running_queries system view. The view contains unique query identifier
> which need to pass to kill query command to cancel the query.
>
> What do you think about fields of the running queries view? May be some
> useful fields we could easy add to the view.
>
> Also let's discuss syntax of cancellation of query. I propose to use MySQL
> like syntax as easy to understand and shorter then Oracle and Postgres
> syntax ( detailed information in IEP-29
> <
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-29%3A+SQL+management+and+monitoring
> >
> ).
>
>
>
> пн, 12 нояб. 2018 г. в 19:28, Юрий :
>
> > Igniters,
> >
> > Below is a proposed design for thin client SQL management and monitoring
> > to cancel a queries.
> >
> > 1) Ignite expose system SQL view with name *running_queries*
> > proposed columns: *node_id, query_id, sql, schema_name, connection_id,
> > duration*.
> >
> > node_id - initial node of request
> > query_id - unique id of query on node
> > sql - text of query
> > schema name - name of sql schema
> > connection_id - id of client connection from
> ClientListenerConnectionContext
> > class
> > duration - duration in millisecond from start of query
> >
> >
> > Ignite will gather info about running queries from each of nodes and
> > collect it during user query. We already have most of the information at
> GridRunningQueryInfo
> > on each of nodes.
> >
> > Instead of duration we can use start_time, but I think duration will be
> > simple to use due to it not depend on a timezone.
> >
> >
> > 2) Propose to use following syntax to kill a running query:
> >
> > *KILL QUERY node_Id query_id*
> >
> >
> > Both parameters node_id and query_id can be get through running_queries
> > system view.
> >
> > When a node receive such request it can be run locally in case node have
> > given node_id or send message to node with given id. Because node have
> > information about local running queries then can cancel it - it already
> > implemented in GridReduceQueryExecutor.cancelQueries(qryId) method.
> >
> > Comments are welcome.
> > --
> > Живи с улыбкой! :D
> >
>
>
> --
> Живи с улыбкой! :D
>


Re: Hello, Ignite Community

2018-11-09 Thread Denis Magda
Welcome, Albert! You are in, look forward to your contribution!

--
Denis

On Thu, Nov 8, 2018 at 10:39 PM Альберт Исхаков 
wrote:

> Hello, Ignite Community!
>
> My name is Albert Iskhakov. I want to contribute to Apache Ignite and want
> to start with this issue [1].
>
> JIRA username is aliskhakov.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-10184
>


Re: Service grid redesign

2018-11-09 Thread Denis Magda
Vyacheslav,

What are the cases when the service can be redeployed? Affinity, failure,
etc., right. It would be good to list all the cases on the wiki and then
our tech writers will get everything documented.

--
Denis

On Thu, Nov 8, 2018 at 11:06 PM Vyacheslav Daradur 
wrote:

> Denis,
>
> Services reassignment process takes into account previous assignments
> to avoid redundant redeployments.
> So, in the described case, ServiceA won't be moved from node1 to node2.
> On Fri, Nov 9, 2018 at 4:41 AM Denis Magda  wrote:
> >
> > Vyacheslav,
> >
> > First of all, thanks for archiving this milestone and rolling out these
> new
> > capabilities.
> >
> > Speaking of the topology change events [1], does the new architecture
> avoid
> > a running service redeployment when a new node joins? For instance, let's
> > say I have ServiceA running node1, then node2 joins and I don't want the
> > service to be redeployed to any other node.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95654584#ServiceGridredesign.Phase1.Implementationdetails.-Topologychange
> >
> > --
> > Denis
> >
> > On Wed, Nov 7, 2018 at 7:04 AM Vyacheslav Daradur 
> > wrote:
> >
> > > Dmitriy, I published documentation in wiki:
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95654584
> > >
> > > Thank you!
> > > On Wed, Nov 7, 2018 at 5:10 PM Dmitriy Pavlov 
> > > wrote:
> > > >
> > > > Hi I think wiki is better than any attached docs. Could you please
> > > create a
> > > > page?
> > > >
> > > > ср, 7 нояб. 2018 г., 14:39 Vyacheslav Daradur :
> > > >
> > > > > I prepared a description of the implemented solution and attached
> it
> > > > > to the issue [1].
> > > > >
> > > > > This should help during a review. Should I post the document into
> wiki
> > > or
> > > > > IEP?
> > > > >
> > > > > I'd like to ask Ignite's experts review the solution [1] [2],
> please?
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9607
> > > > > [2] https://github.com/apache/ignite/pull/4434
> > > > > On Wed, Oct 31, 2018 at 5:04 PM Vyacheslav Daradur <
> > > daradu...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Hi, Igniters! Good news!
> > > > > >
> > > > > > Service Grid Redesign Phase 1 - is in Patch Available now.
> > > > > >
> > > > > > Nikolay Izhikov has reviewed implementation.
> > > > > >
> > > > > > However, we need additional review from other Ignite experts.
> > > > > >
> > > > > > Here is an umbrella ticket [1] and PR [2].
> > > > > >
> > > > > > Could someone step in and do the review?
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9607
> > > > > > [2] https://github.com/apache/ignite/pull/4434
> > > > > > On Sat, Aug 18, 2018 at 11:44 AM Denis Mekhanikov <
> > > dmekhani...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > Pavel, could you assist?
> > > > > > >
> > > > > > > Does it make sense for .Net to specify service class name
> instead
> > > of
> > > > > its
> > > > > > > implementation?
> > > > > > >
> > > > > > > I think, it shouldn't be a problem.
> > > > > > >
> > > > > > > Denis
> > > > > > >
> > > > > > > On Sat, Aug 18, 2018, 11:33 Vyacheslav Daradur <
> > > daradu...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > I think that the replacement of serialized instance makes
> sense
> > > to me
> > > > > > > > for Java part.
> > > > > > > >
> > > > > > > > But how it should work for .NET client?
> > > > > > > >
> > > > > > > > On Tue, Aug 14, 2018 at 4:07 PM Dmitriy Setrakyan <
> > > > > dsetrak...@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > 

Re: Service grid redesign

2018-11-08 Thread Denis Magda
Vyacheslav,

First of all, thanks for archiving this milestone and rolling out these new
capabilities.

Speaking of the topology change events [1], does the new architecture avoid
a running service redeployment when a new node joins? For instance, let's
say I have ServiceA running node1, then node2 joins and I don't want the
service to be redeployed to any other node.

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95654584#ServiceGridredesign.Phase1.Implementationdetails.-Topologychange

--
Denis

On Wed, Nov 7, 2018 at 7:04 AM Vyacheslav Daradur 
wrote:

> Dmitriy, I published documentation in wiki:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95654584
>
> Thank you!
> On Wed, Nov 7, 2018 at 5:10 PM Dmitriy Pavlov 
> wrote:
> >
> > Hi I think wiki is better than any attached docs. Could you please
> create a
> > page?
> >
> > ср, 7 нояб. 2018 г., 14:39 Vyacheslav Daradur :
> >
> > > I prepared a description of the implemented solution and attached it
> > > to the issue [1].
> > >
> > > This should help during a review. Should I post the document into wiki
> or
> > > IEP?
> > >
> > > I'd like to ask Ignite's experts review the solution [1] [2], please?
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-9607
> > > [2] https://github.com/apache/ignite/pull/4434
> > > On Wed, Oct 31, 2018 at 5:04 PM Vyacheslav Daradur <
> daradu...@gmail.com>
> > > wrote:
> > > >
> > > > Hi, Igniters! Good news!
> > > >
> > > > Service Grid Redesign Phase 1 - is in Patch Available now.
> > > >
> > > > Nikolay Izhikov has reviewed implementation.
> > > >
> > > > However, we need additional review from other Ignite experts.
> > > >
> > > > Here is an umbrella ticket [1] and PR [2].
> > > >
> > > > Could someone step in and do the review?
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-9607
> > > > [2] https://github.com/apache/ignite/pull/4434
> > > > On Sat, Aug 18, 2018 at 11:44 AM Denis Mekhanikov <
> dmekhani...@gmail.com>
> > > wrote:
> > > > >
> > > > > Pavel, could you assist?
> > > > >
> > > > > Does it make sense for .Net to specify service class name instead
> of
> > > its
> > > > > implementation?
> > > > >
> > > > > I think, it shouldn't be a problem.
> > > > >
> > > > > Denis
> > > > >
> > > > > On Sat, Aug 18, 2018, 11:33 Vyacheslav Daradur <
> daradu...@gmail.com>
> > > wrote:
> > > > >
> > > > > > I think that the replacement of serialized instance makes sense
> to me
> > > > > > for Java part.
> > > > > >
> > > > > > But how it should work for .NET client?
> > > > > >
> > > > > > On Tue, Aug 14, 2018 at 4:07 PM Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > On Tue, Aug 14, 2018 at 6:10 AM, Nikita Amelchev <
> > > nsamelc...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello, Igniters.
> > > > > > > >
> > > > > > > > I am working on task [1] that would replace serialized
> service's
> > > > > > instance
> > > > > > > > by service's class name and properties map in
> > > {ServiceConfiguration}.
> > > > > > > >
> > > > > > > > The task describes that we should use
> > > > > > > > {String className} + {Map properties} instead
> > > {Service
> > > > > > > > srvc}.
> > > > > > > >
> > > > > > > > I'd like to clarify the following questions:
> > > > > > > >
> > > > > > > > 1. What about public methods?
> > > > > > > > I suggest to mark them as deprecated and use class name of
> > > provided
> > > > > > > > instance.
> > > > > > > > Also to add deploying methods with new parameters:
> > > > > > > >
> > > > > > > > @Deprecated
> > > > > > > > public IgniteInternalFuture
> deployNodeSingleton(ClusterGroup
> > > prj,
> > > > > > > > String
> > > > > > > > name, Service svc)
> > > > > > > >
> > > > > > > > public IgniteInternalFuture
> deployNodeSingleton(ClusterGroup
> > > prj,
> > > > > > > > String
> > > > > > > > name, String srvcClsName, Map prop)
> > > > > > > >
> > > > > > >
> > > > > > > I think this makes sense, but I would like other committers to
> > > confirm.
> > > > > > > Perhaps Vladimir Ozerov should comment here.
> > > > > > >
> > > > > > >
> > > > > > > > 2. Is {Map properties} parameter mandatory
> when
> > > > > > deploying a
> > > > > > > > service?
> > > > > > > > Is it make sense to add deploying methods without it? For
> > > example:
> > > > > > > >
> > > > > > > > public IgniteInternalFuture
> deployNodeSingleton(ClusterGroup
> > > prj,
> > > > > > > > String
> > > > > > > > name, String srvcClsName)
> > > > > > > >
> > > > > > > > public IgniteInternalFuture
> deployNodeSingleton(ClusterGroup
> > > prj,
> > > > > > > > String
> > > > > > > > name, String srvcClsName, Map prop)
> > > > > > > >
> > > > > > >
> > > > > > > I would always ask the user to pass the property map, but would
> > > allow it
> > > > > > to
> > > > > > > be null.
> > > > > > >
> > > > > > > D.
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best Regards, Vyacheslav D.
> > > 

Re: SQL management and monitoring improvements

2018-11-07 Thread Denis Magda
Yuri,

That's an excellent idea, thank you for driving it.

What is not explained is how to leverage from all that stat. For instance,
how can I know a total number of SELECTs, or a total number of SELECTs
happened yesterday for specific data sets. Do believe that it's feasible
and just not covered.

--
Denis



On Wed, Nov 7, 2018 at 2:01 AM Юрий  wrote:

> Hi Igniters!
>
> I think we can improve Ignite management and monitoring instruments related
> to SQL.
> I've prepared draft of IEP-29: SQL management and monitoring
> <
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-29%3A+SQL+management+and+monitoring
> >
> .
>
> What do you think about it? May be do you have some additional suggestions?
>
>
> --
> Живи с улыбкой! :D
>


Re: Becoming a contributor

2018-11-07 Thread Denis Magda
Hi Stephen,

This is good news! Added you to the list, please move on ;)

--
Denis

On Wed, Nov 7, 2018 at 2:31 AM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> Hi,
>
> I’ve started to work on a couple of tickets:
>
> https://issues.apache.org/jira/browse/IGNITE-10155
> https://issues.apache.org/jira/browse/IGNITE-1436
>
> It would be great if I could be added to the list of contributors so I can
> assign the tickets to myself.
>
> Thanks,
> Stephen
>
>
>
>


Re: destroy cache holding residual metadata in memory (2.7)

2018-11-02 Thread Denis Magda
+ dev list

Vladimir, Igniters,

Don't we wipe out the metadata on a cache destroy? Is it an issue or done
on purpose?

--
Denis


On Wed, Oct 31, 2018 at 11:28 AM wt  wrote:

> i am testing code and part of my tests is adding\removing tables. In one of
> the tests i add a table then destroy it and add it again but with an
> additional column. When i try load the table i am getting a data type
> mismatch and it is referring to the previous version of the table
>
> in the work directory there is a folder for the original table but it is
> empty. here is the error i get when trying to flush the loader. If i stop
> and start ignite then adding the table with the new column data type works
> so there is some residual metadata that isn't being cleaned up by
> destroycache of the client.
>
> Apache.Ignite.Core.Binary.BinaryObjectException
>   HResult=0x80131500
>   Message=Binary type has different field types
> [typeName=Tables.csvCurrencyRates, fieldName=id, fieldTypeName1=UUID,
> fieldTypeName2=String]
>   Source=Apache.Ignite.Core
>   StackTrace:
>at Apache.Ignite.Core.Impl.PlatformJniTarget.InStreamOutLong(Int32 type,
> Action`1 writeAction)
>at Apache.Ignite.Core.Impl.PlatformTargetAdapter.DoOutOp(Int32 type,
> Action`1 action)
>at Apache.Ignite.Core.Impl.Binary.Marshaller.FinishMarshal(BinaryWriter
> writer)
>at Apache.Ignite.Core.Impl.PlatformJniTarget.InStreamOutLong(Int32 type,
> Action`1 writeAction)
>at Apache.Ignite.Core.Impl.PlatformTargetAdapter.DoOutOp(Int32 type,
> Action`1 action)
>at Apache.Ignite.Core.Impl.Datastream.DataStreamerImpl`2.Update(Action`1
> action)
>at
>
> Apache.Ignite.Core.Impl.Datastream.DataStreamerBatch`2.Send(DataStreamerImpl`2
> ldr, Int32 plc)
>at
>
> Apache.Ignite.Core.Impl.Datastream.DataStreamerImpl`2.Flush0(DataStreamerBatch`2
> curBatch, Boolean wait, Int32 plc)
>at Apache.Ignite.Core.Impl.Datastream.DataStreamerImpl`2.Flush()
>at ClusterTool.classes.DbProviderCSV.StreamCsvData(List`1 headers,
> String
> tablename, ICsvLine[] lines, Object _class, Type type, IIgnite
> igniteclient,
> Boolean hashashid, CsvNewId csvid) in
> C:\temp\IgniteTool\ClusterTool\classes\DbProviderCSV.cs:line 236
>at ClusterTool.classes.DbProviderCSV.LoadFromCsvFiles(String tablename)
> in C:\temp\IgniteTool\ClusterTool\classes\DbProviderCSV.cs:line 104
>at ClusterTool.DataLoaderForm.<>c__DisplayClass7_0.b__0() in
> C:\temp\IgniteTool\ClusterTool\DataLoaderForm.cs:line 94
>at System.Threading.ExecutionContext.RunInternal(ExecutionContext
> executionContext, ContextCallback callback, Object state, Boolean
> preserveSyncCtx)
>at System.Threading.ExecutionContext.Run(ExecutionContext
> executionContext, ContextCallback callback, Object state, Boolean
> preserveSyncCtx)
>at System.Threading.ExecutionContext.Run(ExecutionContext
> executionContext, ContextCallback callback, Object state)
>at System.Threading.ThreadHelper.ThreadStart()
>
> Inner Exception 1:
> JavaException: class org.apache.ignite.binary.BinaryObjectException: Binary
> type has different field types [typeName=Tables.csvCurrencyRates,
> fieldName=id, fieldTypeName1=UUID, fieldTypeName2=String]
> at
>
> org.apache.ignite.internal.binary.BinaryUtils.mergeMetadata(BinaryUtils.java:1047)
> at
>
> org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.addMeta(CacheObjectBinaryProcessorImpl.java:480)
> at
>
> org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl$2.addMeta(CacheObjectBinaryProcessorImpl.java:207)
> at
>
> org.apache.ignite.internal.binary.BinaryContext.updateMetadata(BinaryContext.java:1332)
> at
>
> org.apache.ignite.internal.processors.platform.PlatformContextImpl.processMetadata(PlatformContextImpl.java:336)
> at
>
> org.apache.ignite.internal.processors.platform.binary.PlatformBinaryProcessor.processInStreamOutLong(PlatformBinaryProcessor.java:70)
> at
>
> org.apache.ignite.internal.processors.platform.PlatformAbstractTarget.processInStreamOutLong(PlatformAbstractTarget.java:87)
> at
>
> org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:67)
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>


Re: Partition Loss Policies issues

2018-11-02 Thread Denis Magda
Hi Stan,

Thanks for the extensive analysis. Alex G. could you please step in and
share your thoughts. Seems it's time to revisit IEP-4 and prioritize the
gaps.

--
Denis

On Wed, Oct 31, 2018 at 7:56 AM Stanislav Lukyanov 
wrote:

> Hi Igniters,
>
> I've been looking into various scenarios of Partition Loss Policies usage
> recently,
> and found a number of issues in the current implementation.
>
> I'll start with an overview, but if you'd like to dive to a proposal I
> have right now then please
> feel free to scroll down to TLDR.
>
> The list of issues is below:
> https://issues.apache.org/jira/browse/IGNITE-10041: Partition loss
> policies work incorrectly with BLT
> https://issues.apache.org/jira/browse/IGNITE-10043: Lost partitions list
> is reset when only one server is alive in the cluster
> https://issues.apache.org/jira/browse/IGNITE-9841: SQL doesn't take lost
> partitions into account when persistence is enabled
> https://issues.apache.org/jira/browse/IGNITE-10057: SQL queries hang
> during rebalance if there are LOST partitions
> https://issues.apache.org/jira/browse/IGNITE-9902: ScanQuery doesn't take
> lost partitions into account
> https://issues.apache.org/jira/browse/IGNITE-10059: Local scan query
> against LOST partition fails
> https://issues.apache.org/jira/browse/IGNITE-10044: LOST partition is
> marked as OWNING after the owner rejoins with existing persistent data
> https://issues.apache.org/jira/browse/IGNITE-10058: resetLostPartitions()
> leaves an additional copy of a partition in the cluster
>
> I'm sure this is not a complete list, but this is what I could find by
> tackling how queries and
> persistence interact with current handling of partition loss.
>
> It seems that the issues - from this list and some other fixed recently -
> can be split into three categories
> - corner case bugs - there are always some, and we can fix them as they
> show up
> - handling of lost partitions by different APIs - while JCache API handles
> lost partitions fine,
> SQL and Scan queries have known issues; other APIs, such different types
> of queries, DataStreamer,
> etc probably need to have more testing
> - Partition Loss Policices + BLT (
> https://issues.apache.org/jira/browse/IGNITE-10041) - BLT seems
> to be fundametally conflicting with the pre-existing semantics of
> partition loss
>
> While the former two categories can be solved case-by-case, the last one
> needs a wider design effort.
> We need to reimagine our partition loss semantics for BLT, and change
> behavior accordingly.
> For now, most of the features don't really work for a cache with BLT, with
> only READ_WRITE_SAFE and
> READ_ONLY_SAFE working correctly (good thing - these two are the most
> useful policies anyway).
>
> TLDR: we have issues with partition loss policices, and the largest one is
> that BLT semantics
> conflict with most partition lost policices, and we need to address this
> somehow.
>
> What I suggest to do right now:
> 1. Deny the configurations that don't work - e.g. just throw an exception
> if a cache starts
> with BLT and PartitionLossPolicy.IGNORE or others.
> 2. Change default PartitionLossPolicy to READ_WRITE_SAFE *for persistent
> caches only*.
> This is what effectively in place for the persistent caches already (since
> IGNORE semantics are
> not supported), so this shouldn't bring a lot of compatibility issues.
>
> I believe doing this will at least help us to protect the users from
> unexpected/inconsistent behavior.
> Actual design changes can be done later, e.g. as a part of IEP 4 Phase 2/3.
>
> WDYT?
>
> Thanks,
> Stan
>
>


Re: Ignite documentation process

2018-11-01 Thread Denis Magda
Artem,

Thanks for the update. Basically, it would be much better if you list the
proposed changes here, so that everyone can share feedback without clicking
the links )

--
Denis



On Thu, Nov 1, 2018 at 7:36 AM Artem Budnikov 
wrote:

> Hi Igniters,
>
> Please take a look at the changes in the "How to Contribute" procedure
> [1] related to Ignite documentation. I just reflected there what we
> discussed earlier about our documentation process. Please see this
> thread for more information [2].
>
>
> [1]:
>
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-Documentingaticket
>
> [2]:
>
> apache-ignite-developers.2346864.n4.nabble.com/Re-Documenting-Ignite-tt32746.html
>
>
> Cheers,
> Artem
>
>


Re: Legal advise on including Visual C++ Redistributable package into ODBC installer

2018-10-25 Thread Denis Magda
Dmitry,

Thanks for the pointers!

Would you be interested in sorting this out for the community by talking to
our ASF-mates who should know?

--
Denis

On Thu, Oct 25, 2018 at 3:21 AM Dmitriy Pavlov 
wrote:

> Hi Igor,
>
> I'm not sure if someone in the community can know an answer to this
> question.
>
> I can share a page with resolved legal questions here:
> https://www.apache.org/legal/resolved.html#category-b
>
> But I think it may require confirmation from Apache Legal
> http://www.apache.org/legal/
> https://issues.apache.org/jira/projects/LEGAL/summary
> http://www.apache.org/foundation/mailinglists.html#foundation-legal
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 24 окт. 2018 г. в 18:40, Igor Sapego :
>
> > Hi guys,
> >
> > I need your advice here. As some of you probably know we have
> > Windows installers for ODBC in our binary releases, so users won't need
> > to build Ignite C++ if all they want is to install and use our ODBC
> driver.
> > However, as we build driver with MSVC 10 when we are preparing our
> > binary release, user should have Visual C++ 2010 Redistributable package
> > for it to work properly. If they do not, it just do not work, not giving
> > any
> > human-readable errors, which is very confusing to a user.
> >
> > So my idea was to check for required package during installation and
> > give user a download link to a proper package. However, on the WiX
> > website there are articles on how to include the package into installer
> > [1].
> > So I've thought that maybe this is even the better way to solve a
> problem.
> >
> > So thus is my question, is it OK from the legal standpoint, if we will
> > distribute
> > installer, that includes Visual C++ 2010 Redistributable package? We are
> > not
> > going to put any binaries under the source control, of course.
> >
> > Can anyone help?
> >
> > [1] -
> >
> >
> http://wixtoolset.org/documentation/manual/v3/howtos/redistributables_and_install_checks/install_vcredist.html
> >
> > Best Regards,
> > Igor
> >
>


Re: Pre-touch for Ignite off-heap memory

2018-10-25 Thread Denis Magda
In my understanding, that's an easy operation to support - we just need to
allocate requested space and pre-touch it (iterate and fill out with 0).
Yes, it will increase the launch time but it's a valid use case. Some
companies do memory pre-touching on purpose.

--
Denis



On Thu, Oct 25, 2018 at 7:41 AM Gerus  wrote:

> Hi, Ivan
> Let me try to explain. I think that there are 2 goals that are mostly
> applicable for prod systems:
> 1. I can agree with Dave that swap is not a good case for performance and
> disk resource. It is possible that other applications can consume memory
> that is free after Ignite was started. In this case if Ignite will be
> creating new pages in runtime, it can face with OOM.
>
> 2. Ignite user can create wrong configuration. For example, data region can
> exceed available memory by mistake or Ignite can be started on another
> server with less RAM. It can lead to catch OOM in runtime
>
> To summarize: Pre-allocation can detect listed issues on Ignite start by
> allocating OS RAM
>
> Im sure that this option have to be disabled by default, but user should
> have a choice for startup
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>


Re: hello

2018-10-25 Thread Denis Magda
Hi Scott,

Thanks, the contributions looks valuable to me. As I see a community member
is already working with you. Hope the code will be merged soon!

--
Denis

On Thu, Oct 25, 2018 at 10:59 AM Scott Feldstein  wrote:

> Hi Denis,
> Thanks!
>
> This is what I'm trying to contribute
>   jira: https://issues.apache.org/jira/browse/IGNITE-9893
>   pullRequest: https://github.com/apache/ignite/pull/5008
>
> WRT the blog, we actually discussed this at the in-memory conference.  At
> the beginning of the year we'll start to make our usage more public
> internally and externally if all goes well :)
>
> Scott
>
> On Wed, Oct 24, 2018 at 7:44 PM Denis Magda  wrote:
>
> > Hey Scott,
> >
> > Great to know, welcome to the community!
> >
> > What exactly do you like to contribute back? Please start a separate
> > discussion with the most meaningful topic.
> >
> > Also, if you can post a blog post about your architecture at VMWare will
> be
> > glad to mention it on the "Ignite in Production" page:
> > https://ignite.apache.org/provenusecases.html
> >
> > --
> > Denis
> >
> > On Wed, Oct 24, 2018 at 7:39 PM Scott Feldstein 
> > wrote:
> >
> > > Hi Everyone,
> > > I'm following the instructions from
> > > https://ignite.apache.org/community/contribute.html. I'd like to
> > > contribute
> > > some of the code I've been working on in order to continue evolving
> > Ignite.
> > >
> > > I've been evangelizing Ignite at VMware and am in the process of
> locking
> > > down our next-gen service architecture.  The plan is to use Ignite as a
> > > multi-faceted caching server on steriods!
> > >
> > > Please feel free to reach out to me and say hi as well :)
> > >
> > > Scott
> > >
> >
>


Re: hello

2018-10-24 Thread Denis Magda
Hey Scott,

Great to know, welcome to the community!

What exactly do you like to contribute back? Please start a separate
discussion with the most meaningful topic.

Also, if you can post a blog post about your architecture at VMWare will be
glad to mention it on the "Ignite in Production" page:
https://ignite.apache.org/provenusecases.html

--
Denis

On Wed, Oct 24, 2018 at 7:39 PM Scott Feldstein  wrote:

> Hi Everyone,
> I'm following the instructions from
> https://ignite.apache.org/community/contribute.html. I'd like to
> contribute
> some of the code I've been working on in order to continue evolving Ignite.
>
> I've been evangelizing Ignite at VMware and am in the process of locking
> down our next-gen service architecture.  The plan is to use Ignite as a
> multi-faceted caching server on steriods!
>
> Please feel free to reach out to me and say hi as well :)
>
> Scott
>


Re: Pre-touch for Ignite off-heap memory

2018-10-23 Thread Denis Magda
Alex,

Correct me if I'm wrong, but even if an OS runs out of physical memory (X
GB in total) an Ignite node process still can request the X GB from virtual
memory. Yes, virtual memory can involve swapping and disk to satisfy your
request but this shouldn't be a reason of the OOM. Shouldn't OOM happen if
you're trying to allocate beyond the virtual memory capacity (beyond X GB)?

Denis

On Tue, Oct 23, 2018 at 12:08 PM Gerus  wrote:

> Hi *Igniters*,
> Some time ago I've raised a suggestion for product improvement
> https://issues.apache.org/jira/browse/IGNITE-9112
>   .  It's all about
> off-heap memory allocation. Current implementation can have some
> improvements for failure critical systems. Ignite can have OOM in runtime,
> because RAM can be used by OS, if it will not be pre-booked by operation
> system and this proposal is to address that. Common case is offheap and
> thats why memory segment cannot be managed by JVM that has +AlwaysPreTouch
> option
> Obviously this implementation will make startup longer and thats why it is
> proposed to use configuration flag to manage this feature
> I think, it will be useful to have this option. Are you supporting this?
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>


Re: Apache Ignite 2.7. Last Mile

2018-10-20 Thread Denis Magda
Nikolay,

Where do you track those 8 blockers? Can't find them on this wiki page:
https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7

--
Denis

On Sat, Oct 20, 2018 at 11:31 AM Nikolay Izhikov 
wrote:

> Hello, Denis.
>
> As a first time release manager I'm trying to rely on Ignite veterans
> opinion.
> Guys told me that we must fix all blockers and only after it make the
> release.
>
> Let's fix them all.
> What do you think?
>
> > When are we sending a release candidate for vote?
>
> When all blocker bugs will be fixed.
> Currently, we have *8*.
>
> В Пт, 19/10/2018 в 13:50 -0700, Denis Magda пишет:
> > Guys, as a side observer of the current release, this all looks like a
> > never ending story :)
> >
> > When are we sending a release candidate for vote?
> >
> > --
> > Denis
> >
> > On Fri, Oct 19, 2018 at 4:39 AM Nikolay Izhikov 
> wrote:
> >
> > > Hello, Igniters.
> > >
> > > We have 6 tickets for 2.7
> > >
> > > Roman Kondakov - IGNITE-9892, IGNITE-9663, IGNITE-9928
> > > Igor Seliverstov - IGNITE-9911
> > > Ivan Pavlukhin - IGNITE-5935, IGNITE-9944
> > >
> > > В Чт, 18/10/2018 в 15:40 +0300, Andrey Kuznetsov пишет:
> > > > I have got one more potential 2.7 blocker [1] with straightforward
> fix. I
> > > > beleive it will not break any production use case, but it leads to
> test
> > > > suite hang, thus affecting other urgent issues.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-9932
> > > >
> > > > чт, 18 окт. 2018 г. в 14:59, Ivan Daschinsky :
> > > >
> > > > > Hi! Is it possible to merge IGNITE-9854? Fix is pretty simple, but
> > >
> > > quite
> > > > > important.
> > > > >
> > > > > ср, 17 окт. 2018 г. в 17:49, Andrey Gura :
> > > > >
> > > > > > JFYI
> > > > > >
> > > > > > IGNITE-9737 and IGNITE-9710 are merged to release branch.
> > > > > > On Wed, Oct 17, 2018 at 5:41 PM Pavel Tupitsyn <
> ptupit...@apache.org
> > > > > > wrote:
> > > > > > >
> > > > > > > Thank you. Fix has been merged to master and cherry-picked to
> > > > >
> > > > > ignite-2.7.
> > > > > > >
> > > > > > > On Wed, Oct 17, 2018 at 1:26 PM Nikolay Izhikov <
> > >
> > > nizhi...@apache.org>
> > > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Pavel.
> > > > > > > >
> > > > > > > > Ok, I agree to include this ticket into 2.7
> > > > > > > > Let's do it.
> > > > > > > >
> > > > > > > > В Ср, 17/10/2018 в 13:20 +0300, Pavel Tupitsyn пишет:
> > > > > > > > > Nikolay,
> > > > > > > > >
> > > > > > > > > It completely breaks a major feature under certain
> conditions.
> > >
> > > I
> > > > > >
> > > > > > would
> > > > > > > > > consider it a blocker.
> > > > > > > > >
> > > > > > > > > On Wed, Oct 17, 2018 at 1:00 PM Nikolay Izhikov <
> > > > >
> > > > > nizhi...@apache.org
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello, Pavel.
> > > > > > > > > >
> > > > > > > > > > Is it a blocker?
> > > > > > > > > >
> > > > > > > > > > В Ср, 17/10/2018 в 12:58 +0300, Pavel Tupitsyn пишет:
> > > > > > > > > > > Hi Igniters,
> > > > > > > > > > >
> > > > > > > > > > > I'd like to include IGNITE-9877 in 2.7, can we do that?
> > > > > > > > > > > The fix is ready, I'm waiting for TC run.
> > > > > > > > > > >
> > > > > > > > > > > Pavel
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Oct 17, 2018 at 11:45 AM Павлухин Иван <
> > > > > >
> > > > > > vololo...@gmail.com&g

Re: [DISCUSSION] Spark Data Frame through Thin Client

2018-10-20 Thread Denis Magda
Hello Nikolay,

Your proposal sounds reasonable. However, I would suggest us to wait while
partition-awareness is supported for Java thin client first. With that
feature, the client can connect to any node directly while presently all
the communication goes through a proxy (a node the client is connected to).
All of that is bad for performance.


Vladimir, how hard would it be to support the partition-awareness for Java
client? Probably, Nikolay can take over.

--
Denis


On Sat, Oct 20, 2018 at 2:09 PM Nikolay Izhikov  wrote:

> Hello, Igniters.
>
> Currently, Spark Data Frame integration implemented via client node
> connection.
> Whenever we need to retrieve some data into Spark worker(or master) from
> Ignite we start a client node.
>
> It has several major disadvantages:
>
> 1. We should copy whole Ignite distribution on to each Spark
> worker [1]
> 2. We should copy whole Ignite distribution on to Spark master to
> get catalogue works.
> 3. We should have the same absolute path to Ignite configuration
> file on every worker and provide it during data frame construction [2]
> 4. We should additionally configure Spark workerks classpath to
> include Ignite libraries.
>
> For now, almost all operation we need to do in Spark Data Frame
> integration is supported by Java Thin Client.
> * obtain the list of caches.
> * get cache configuration.
> * execute SQL query.
> * stream data to the table - don't support by the thin client for
> now, but can be implemented using simple SQL INSERT statements.
>
> Advantages of usage Java Thin Client in Spark integration(they all known
> from Java Thin Client advantages):
> 1. Easy to configure: only IP addresses of server nodes are
> required.
> 2. Easy to deploy: only 1 additional jar required. No server
> side(Ignite worker) configuration required.
>
> I propose to implement Spark Data Frame integration through Java Thin
> Client.
>
> Thoughts?
>
> [1] https://apacheignite-fs.readme.io/docs/installation-deployment
> [2]
> https://apacheignite-fs.readme.io/docs/ignite-data-frame#section-ignite-dataframe-options
>


Re: Apache Ignite 2.7. Last Mile

2018-10-19 Thread Denis Magda
Guys, as a side observer of the current release, this all looks like a
never ending story :)

When are we sending a release candidate for vote?

--
Denis

On Fri, Oct 19, 2018 at 4:39 AM Nikolay Izhikov  wrote:

> Hello, Igniters.
>
> We have 6 tickets for 2.7
>
> Roman Kondakov - IGNITE-9892, IGNITE-9663, IGNITE-9928
> Igor Seliverstov - IGNITE-9911
> Ivan Pavlukhin - IGNITE-5935, IGNITE-9944
>
> В Чт, 18/10/2018 в 15:40 +0300, Andrey Kuznetsov пишет:
> > I have got one more potential 2.7 blocker [1] with straightforward fix. I
> > beleive it will not break any production use case, but it leads to test
> > suite hang, thus affecting other urgent issues.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-9932
> >
> > чт, 18 окт. 2018 г. в 14:59, Ivan Daschinsky :
> >
> > > Hi! Is it possible to merge IGNITE-9854? Fix is pretty simple, but
> quite
> > > important.
> > >
> > > ср, 17 окт. 2018 г. в 17:49, Andrey Gura :
> > >
> > > > JFYI
> > > >
> > > > IGNITE-9737 and IGNITE-9710 are merged to release branch.
> > > > On Wed, Oct 17, 2018 at 5:41 PM Pavel Tupitsyn  >
> > > > wrote:
> > > > >
> > > > > Thank you. Fix has been merged to master and cherry-picked to
> > >
> > > ignite-2.7.
> > > > >
> > > > > On Wed, Oct 17, 2018 at 1:26 PM Nikolay Izhikov <
> nizhi...@apache.org>
> > > >
> > > > wrote:
> > > > >
> > > > > > Pavel.
> > > > > >
> > > > > > Ok, I agree to include this ticket into 2.7
> > > > > > Let's do it.
> > > > > >
> > > > > > В Ср, 17/10/2018 в 13:20 +0300, Pavel Tupitsyn пишет:
> > > > > > > Nikolay,
> > > > > > >
> > > > > > > It completely breaks a major feature under certain conditions.
> I
> > > >
> > > > would
> > > > > > > consider it a blocker.
> > > > > > >
> > > > > > > On Wed, Oct 17, 2018 at 1:00 PM Nikolay Izhikov <
> > >
> > > nizhi...@apache.org
> > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hello, Pavel.
> > > > > > > >
> > > > > > > > Is it a blocker?
> > > > > > > >
> > > > > > > > В Ср, 17/10/2018 в 12:58 +0300, Pavel Tupitsyn пишет:
> > > > > > > > > Hi Igniters,
> > > > > > > > >
> > > > > > > > > I'd like to include IGNITE-9877 in 2.7, can we do that?
> > > > > > > > > The fix is ready, I'm waiting for TC run.
> > > > > > > > >
> > > > > > > > > Pavel
> > > > > > > > >
> > > > > > > > > On Wed, Oct 17, 2018 at 11:45 AM Павлухин Иван <
> > > >
> > > > vololo...@gmail.com>
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi NIkolay,
> > > > > > > > > >
> > > > > > > > > > Thank you for keeping everybody focused! Regarding to my
> > >
> > > ticket
> > > > > > > > > > IGNITE-5935.
> > > > > > > > > > It is in final stage now. Tests look good. I believe
> that it
> > > >
> > > > will
> > > > > > be
> > > > > > > >
> > > > > > > > merged
> > > > > > > > > > in couple of days (at most).
> > > > > > > > > >
> > > > > > > > > > ср, 17 окт. 2018 г. в 11:39, Nikolay Izhikov <
> > > >
> > > > nizhi...@apache.org
> > > > > > > :
> > > > > > > > > >
> > > > > > > > > > > Hello, Igniters.
> > > > > > > > > > >
> > > > > > > > > > > 9 tickets to go!
> > > > > > > > > > >
> > > > > > > > > > > Alexey Goncharuk - IGNITE-9784
> > > > > > > > > > > Dmitriy Govorukhin - IGNITE-9898
> > > > > > > > > > > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > > > > > > > > > > Taras Ledkov - IGNITE-9882
> > > > > > > > > > > Petr Ivanov - IGNITE-9852
> > > > > > > > > > > Ivan Pavlukhin - IGNITE-5935
> > > > > > > > > > > Roman Kondakov - IGNITE-9663
> > > > > > > > > > > Alexey Stelmak - IGNITE-9776
> > > > > > > > > > >
> > > > > > > > > > > В Вт, 16/10/2018 в 16:20 +0300, Andrey Gura пишет:
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > I've found that IGNITE-9723 was resolved but didn't
> > >
> > > cherry
> > > > > > picked
> > > > > > > >
> > > > > > > > to
> > > > > > > > > > > > ignite-2.7 branch. So I'll do it.
> > > > > > > > > > > > On Tue, Oct 16, 2018 at 2:30 PM Nikolay Izhikov <
> > > > > > > >
> > > > > > > > nizhi...@apache.org>
> > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hello, Igniters.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We have 13 tickets mapped to 2.7.
> > > > > > > > > > > > > All tickets assigned to some contributor.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Alexey Gonchruk - IGNITE-9784, IGNITE-9895
> > > > > > > > > > > > > Vladimir Ozerov - IGNITE-9887
> > > > > > > > > > > > > Dmitriy Govorukhin - IGNITE-9898
> > > > > > > > > > > > > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > > > > > > > > > > > > Petr Ivanov - IGNITE-9852
> > > > > > > > > > > > > Ivan Pavlukhin - IGNITE-5935
> > > > > > > > > > > > > Alexey Stelmak - IGNITE-9776
> > > > > > > > > > > > > Roman Kondakov - IGNITE-9663
> > > > > > > > > > > > > Taras Ledkov - IGNITE-9864
> > > > > > > > > > > > > Igor Seliverstov   - IGNITE-9292
> > > > > > > > > > > > >
> > > > > > > > > > > > > В Пн, 15/10/2018 в 17:49 +0300, Andrey Gura 

Re: [DISCUSS] Dropping hadoop-accelerator download

2018-10-19 Thread Denis Magda
Here it is: https://issues.apache.org/jira/browse/IGNITE-9953

--
Denis

On Fri, Oct 19, 2018 at 2:34 AM Petr Ivanov  wrote:

> Can you file an issue then, please?
>
>
> > On 19 Oct 2018, at 04:04, Denis Magda  wrote:
> >
> > I would do this if it doesn't impact the release time.
> >
> > --
> > Denis
> >
> > On Thu, Oct 18, 2018 at 2:34 AM Petr Ivanov  wrote:
> >
> >> Can we drop Hadoop binaries from delivery this release (2.7) or should
> it
> >> be moved to next?
> >>
> >>
> >>> On 18 Oct 2018, at 04:38, Denis Magda  wrote:
> >>>
> >>> No objections from my side.
> >>>
> >>> --
> >>> Denis
> >>>
> >>>
> >>> On Tue, Oct 16, 2018 at 4:45 PM Dmitriy Setrakyan <
> dsetrak...@apache.org
> >>>
> >>> wrote:
> >>>
> >>>> Igniters,
> >>>>
> >>>> I would like to suggest dropping the hadoop accelerator distribution
> of
> >>>> Apache Ignite. Note, that it does not mean dropping any features. We
> are
> >>>> still going to support IGFS, in-memory map reduce, spark integration,
> >> etc.
> >>>> However, the hadoop-accelerator downloadable is putting extra pressure
> >> on
> >>>> the community to test and document, which can be avoided. All the
> >> features
> >>>> will still be supported through regular Ignite distribution.
> >>>>
> >>>> Any objections?
> >>>>
> >>>> D.
> >>>>
> >>
> >>
>
>


[jira] [Created] (IGNITE-9953) Dropping hadoop accelerator downloads

2018-10-19 Thread Denis Magda (JIRA)
Denis Magda created IGNITE-9953:
---

 Summary: Dropping hadoop accelerator downloads
 Key: IGNITE-9953
 URL: https://issues.apache.org/jira/browse/IGNITE-9953
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Reporter: Denis Magda
Assignee: Peter Ivanov
 Fix For: 2.7


The community agreed to drop the Hadoop distributions of Apache Ignite.

Note, that it does not mean dropping any features. We are
still going to support IGFS, in-memory map reduce, spark integration, etc.
However, the hadoop-accelerator downloadable is putting extra pressure on
the community to test and document, which can be avoided. All the features
will still be supported through regular Ignite distribution.

Discussion: 
http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Dropping-hadoop-accelerator-download-td36617.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: [DISCUSS] Dropping hadoop-accelerator download

2018-10-18 Thread Denis Magda
I would do this if it doesn't impact the release time.

--
Denis

On Thu, Oct 18, 2018 at 2:34 AM Petr Ivanov  wrote:

> Can we drop Hadoop binaries from delivery this release (2.7) or should it
> be moved to next?
>
>
> > On 18 Oct 2018, at 04:38, Denis Magda  wrote:
> >
> > No objections from my side.
> >
> > --
> > Denis
> >
> >
> > On Tue, Oct 16, 2018 at 4:45 PM Dmitriy Setrakyan  >
> > wrote:
> >
> >> Igniters,
> >>
> >> I would like to suggest dropping the hadoop accelerator distribution of
> >> Apache Ignite. Note, that it does not mean dropping any features. We are
> >> still going to support IGFS, in-memory map reduce, spark integration,
> etc.
> >> However, the hadoop-accelerator downloadable is putting extra pressure
> on
> >> the community to test and document, which can be avoided. All the
> features
> >> will still be supported through regular Ignite distribution.
> >>
> >> Any objections?
> >>
> >> D.
> >>
>
>


Re: [DISCUSS] Dropping hadoop-accelerator download

2018-10-17 Thread Denis Magda
No objections from my side.

--
Denis


On Tue, Oct 16, 2018 at 4:45 PM Dmitriy Setrakyan 
wrote:

> Igniters,
>
> I would like to suggest dropping the hadoop accelerator distribution of
> Apache Ignite. Note, that it does not mean dropping any features. We are
> still going to support IGFS, in-memory map reduce, spark integration, etc.
> However, the hadoop-accelerator downloadable is putting extra pressure on
> the community to test and document, which can be avoided. All the features
> will still be supported through regular Ignite distribution.
>
> Any objections?
>
> D.
>


Re: Applicability of term 'cache' to Apache Ignite

2018-10-17 Thread Denis Magda
Key-value calls are just primary key based calls. From a user perspective,
it's the same as "SELECT * FROM table WHERE primary_idx = X", just
different API.

--
Denis

On Wed, Oct 17, 2018 at 5:04 PM Dmitriy Setrakyan 
wrote:

> On Wed, Oct 17, 2018 at 4:58 PM Denis Magda  wrote:
>
> > I've been calling everything "tables" instead of "caches" for a while.
> The
> > main reason is the maturity of our SQL engine - seeing more SQL users and
> > deployments which talk "tables" language.
> >
> >
> I think "IgniteTable" only implies SQL, not key-value. We need both.
>


Re: Applicability of term 'cache' to Apache Ignite

2018-10-17 Thread Denis Magda
I've been calling everything "tables" instead of "caches" for a while. The
main reason is the maturity of our SQL engine - seeing more SQL users and
deployments which talk "tables" language.


On Wed, Oct 17, 2018 at 3:55 PM Dmitriy Setrakyan 
wrote:

> If dataset cannot be used, can we still consider "IgniteData"?
>
> D.
>
> On Wed, Oct 17, 2018 at 5:06 AM Ilya Lantukh 
> wrote:
>
> > As I see, many people agree that the term *"cache"* is outdated, but
> > consider these changes too disruptive.
> >
> > For me, keeping terminology up-to-date is important part of project
> > development. If we change some of our core terms with more relevant ones,
> > it indeed might cause confusion for current users, but in long term it
> will
> > help new users to understand what Ignite is and what it isn't. And most
> > short-term problems can easily be avoided by keeping @Deprecated
> > IgniteCache.
> >
> > On Wed, Oct 17, 2018 at 2:59 PM Ilya Lantukh 
> > wrote:
> >
> > > Unfortunately, we already use the word *"store"* for many other
> concepts,
> > > like CacheStore and PageStore. I'd prefer to avoid giving it one more
> > > meaning.
> > >
> > > As already mentioned, *"dataset"* has special meaning for ML folks.
> > >
> > > *"Bucket" *might give wrong association with bucket in a hash table.
> > >
> > > On Wed, Oct 17, 2018 at 1:49 PM Igor Sapego 
> wrote:
> > >
> > >> Well, the obvious term for me is a "Store" or "MemoryStore", as we
> > already
> > >> have persistence store.
> > >>
> > >> Best Regards,
> > >> Igor
> > >>
> > >>
> > >> On Wed, Oct 17, 2018 at 1:19 PM Andrey Kuznetsov 
> > >> wrote:
> > >>
> > >> > I'm not an ML expert, so 'dataset' term just reminds me of various
> > >> client
> > >> > drivers to access tables from RDBM servers. For me, the only common
> > >> trait
> > >> > of all kinds of Ignite caches is their asociativity. So if we rename
> > >> them
> > >> > I'd suggest something like KVStore.
> > >> >
> > >> > ср, 17 окт. 2018 г. в 12:56, Alexey Zinoviev <
> zaleslaw@gmail.com
> > >:
> > >> >
> > >> > > From my perspective, the main goal is to make easy the explanation
> > >> what
> > >> > is
> > >> > > Ignite on conferences, marketing deals, in papers, in
> documentation.
> > >> And
> > >> > > the
> > >> > > /cache/ term really reduces the area of Ignite usage in users
> minds.
> > >> > >
> > >> > > I don't support the critical changes in code base, but I support
> all
> > >> > > changes
> > >> > > that helps the goal described above in this letter.
> > >> > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> > Best regards,
> > >> >   Andrey Kuznetsov.
> > >> >
> > >>
> > >
> > >
> > > --
> > > Best regards,
> > > Ilya
> > >
> >
> >
> > --
> > Best regards,
> > Ilya
> >
>


[jira] [Created] (IGNITE-9798) Add TensorFlow Integration Page to Ignite website

2018-10-04 Thread Denis Magda (JIRA)
Denis Magda created IGNITE-9798:
---

 Summary: Add TensorFlow Integration Page to Ignite website
 Key: IGNITE-9798
 URL: https://issues.apache.org/jira/browse/IGNITE-9798
 Project: Ignite
  Issue Type: Task
  Components: site
Reporter: Denis Magda
Assignee: Prachi Garg
 Fix For: 2.7


We need to create a dedicated page for Ignite and TensorFlow integration. 
Please put it under Machine Learning item of the Features menu.

[~abchaudhri], will provide a reference to the readme.io page with in-depth 
integration description.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (IGNITE-9797) Refer to PHP, Python and Node.JS getting started guides from the website

2018-10-04 Thread Denis Magda (JIRA)
Denis Magda created IGNITE-9797:
---

 Summary: Refer to PHP, Python and Node.JS getting started guides 
from the website
 Key: IGNITE-9797
 URL: https://issues.apache.org/jira/browse/IGNITE-9797
 Project: Ignite
  Issue Type: Task
  Components: site
Affects Versions: 2.7
Reporter: Denis Magda
Assignee: Prachi Garg


This page includes a section with the list of references to getting started 
guides:
https://ignite.apache.org/features/multilanguage.html

Add references to Python, PHP and Node.JS docs on readme.io (Instantiation and 
Configuration pages on readme).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: [ML] New features and improvement of ML module for 2.7 release

2018-10-03 Thread Denis Magda
>
> It will be great if you can share your presentation/video after this summit
> in dev-list.


The video should be posted on this page in a couple of weeks:
https://www.imcsummit.org/2018/us/session/scalable-machine-and-deep-learning-apache-ignite

Did you use a TensorFlow integration stand in your presentation?


Yes, could present and announce it. Yuri and Andrey helped me with the demo
but, unfortunately, could show it because the organizers failed to set up
my laptop and projector.

--
Denis

On Wed, Oct 3, 2018 at 2:35 AM Alexey Zinoviev 
wrote:

> It will be great if you can share your presentation/video after this summit
> in dev-list.
> Did you use a TensorFlow integration stand in your presentation?
>
> Good news about potential users, it will be great to contact with somebody
> who are going to use ML in production to discuss possible cases
>
> ср, 3 окт. 2018 г. в 6:29, Denis Magda :
>
> > Alexey,
> >
> > Thanks for spreading the word about the ML capabilities! *Prachi*, please
> > help us to add the talks Alexey is going to give to Ignite events page:
> > https://ignite.apache.org/events.html
> >
> > Btw, I gave a presentation about Ignite ML + TensorFlow integration today
> > at IMC Summit in the US. It was perceived really well, was bombarded with
> > many questions after the talk and think we've got some potential users ;)
> >
> > --
> > Denis
> >
> > On Tue, Oct 2, 2018 at 8:54 AM Alexey Zinoviev 
> > wrote:
> >
> > > Currently, in release 2.7, the ignite ML has a parity with a Spark ML
> by
> > ML
> > > algorithms, feature preprocessing and other capabilities.
> > >
> > > I'm going to talk about that in October on two conferences
> > >
> > > 1) [Ru] Yaroslavl, Open Source Distributed Machine Learning Library for
> > > Apache Ignite https://yappidays.ru/talks.html#zinovev
> > >
> > > 2) [En] Minsk, Nuances of Machine Learning with Ignite ML,
> > > https://jfuture.by/#talkbyAlexeyZinoviev
> > >
> > > After my previous event, JUG MSK, the new contributor @Ravil Galeyev
> > joined
> > > to our community, hope for new members from Yaroslavl and Minsk soon
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> > >
> >
>


Re: [ML] New features and improvement of ML module for 2.7 release

2018-10-02 Thread Denis Magda
Alexey,

Thanks for spreading the word about the ML capabilities! *Prachi*, please
help us to add the talks Alexey is going to give to Ignite events page:
https://ignite.apache.org/events.html

Btw, I gave a presentation about Ignite ML + TensorFlow integration today
at IMC Summit in the US. It was perceived really well, was bombarded with
many questions after the talk and think we've got some potential users ;)

--
Denis

On Tue, Oct 2, 2018 at 8:54 AM Alexey Zinoviev 
wrote:

> Currently, in release 2.7, the ignite ML has a parity with a Spark ML by ML
> algorithms, feature preprocessing and other capabilities.
>
> I'm going to talk about that in October on two conferences
>
> 1) [Ru] Yaroslavl, Open Source Distributed Machine Learning Library for
> Apache Ignite https://yappidays.ru/talks.html#zinovev
>
> 2) [En] Minsk, Nuances of Machine Learning with Ignite ML,
> https://jfuture.by/#talkbyAlexeyZinoviev
>
> After my previous event, JUG MSK, the new contributor @Ravil Galeyev joined
> to our community, hope for new members from Yaroslavl and Minsk soon
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>


Re: Apache Ignite 2.7 release

2018-09-28 Thread Denis Magda
t a bit for other
> cool
> > > > > >
> > > > > > features
> > > > > > > > > ready
> > > > > > > > > > > by that time, then again and again, and release will
> never
> > > > > >
> > > > > > happen.
> > > > > > > > And
> > > > > > > > > > > while we are waiting for new features to come, already
> > > > >
> > > > > completerd
> > > > > > > > > > features
> > > > > > > > > > > cannot be used by anyone.
> > > > > > > > > > >
> > > > > > > > > > > This is why we have an agreement that if feature is not
> > > >
> > > > ready,
> > > > > it
> > > > > > > > > should
> > > > > > > > > > be
> > > > > > > > > > > moved to future release, instead of shifting release.
> The
> > > >
> > > > sole
> > > > > > > reason
> > > > > > > > > to
> > > > > > > > > > > have strict dates when decisions are made is to let
> release
> > > > > >
> > > > > > happen.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Sep 28, 2018 at 2:22 AM Dmitriy Pavlov <
> > > > > > > >
> > > > > > > > dpavlov@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Vladimir,  I'm not searching for enemy, and not
> fighting
> > > >
> > > > with
> > > > > > > you.
> > > > > > > > > I'm
> > > > > > > > > > > not
> > > > > > > > > > > > happy about cases when we are hurrying.
> > > > > > > > > > > >
> > > > > > > > > > > > We can't fix test, fill ticket details, can't wait
> for
> > > > > > > >
> > > > > > > > contributions
> > > > > > > > > to
> > > > > > > > > > > > finish their tasks.  It is not best idea to use
> > >
> > > experience
> > > > > from
> > > > > > > > > > > commercial
> > > > > > > > > > > > companies in open source. Are there any pressure
> outside
> > > > > > >
> > > > > > > community?
> > > > > > > > > Did
> > > > > > > > > > > > someone promised rest of features to be released at
> 30
> > > > > >
> > > > > > September?
> > > > > > > > > > > >
> > > > > > > > > > > > Let's remember principle do-orcracy, power of those
> who
> > >
> > > do.
> > > > > If
> > > > > > > > > > contribor
> > > > > > > > > > > > does change and reviewer does review, let's give
> right of
> > > > > >
> > > > > > making
> > > > > > > > > > decision
> > > > > > > > > > > > to them, but not to some closed club of people who
> > > >
> > > > privately
> > > > > > > > discuss
> > > > > > > > > > > > something.
> > > > > > > > > > > >
> > > > > > > > > > > > Sincerely
> > > > > > > > > > > > Dmitriy Pavlov
> > > > > > > > > > > >
> > > > > > > > > > > > чт, 27 сент. 2018 г., 23:42 Vyacheslav Daradur <
> > > > > > > >
> > > > > > > > daradu...@gmail.com
> > > > > > > > > > :
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Igniters!
> > > > > > > > > > > > >
> > > > > > > > > > > > > As I have written about

Re: Apache Ignite 2.7 release

2018-09-28 Thread Denis Magda
ce Grid before [1] I'm
> > > > finalizing
> > > > > > the
> > > > > > > > > > > solution to be sure that implementation is reliable.
> > > > > > > > > > >
> > > > > > > > > > > About including it in 2.7, if we talk that code freeze
> > > > tomorrow
> > > > > > > then
> > > > > > > > > > > the solution is not ready to merge yet.
> > > > > > > > > > > I hope that prereviewers Anton Vinogradov and Nikolay
> > > Izhikov
> > > > > > will
> > > > > > > be
> > > > > > > > > > > able to answer if solution out of scope or not in a
> > couple
> > > of
> > > > > > days.
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Apache-Ignite-2-7-release-td34076.html#a34485
> > > > > > > > > > > On Thu, Sep 27, 2018 at 11:14 PM Dmitriy Pavlov <
> > > > > > > > dpavlov@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, I agree, NPE during WAL flush is definitely a
> > > blocker.
> > > > > > > > > > > >
> > > > > > > > > > > > It is strange why the current test set did not fail
> > after
> > > > > > commit.
> > > > > > > > > > > >
> > > > > > > > > > > > чт, 27 сент. 2018 г. в 21:45, Andrey Kuznetsov <
> > > > > > > stku...@gmail.com
> > > > > > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > > > Igniters,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I've bumped into a new bug in WAL manager recently,
> > see
> > > > > [1].
> > > > > > It
> > > > > > > > > looks
> > > > > > > > > > > > > critical enough, and can be a good candidate for
> > fixing
> > > > > > before
> > > > > > > > 2.7
> > > > > > > > > > > release.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Do you agree?
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1]
> > https://issues.apache.org/jira/browse/IGNITE-9731
> > > > > > > > > > > > >
> > > > > > > > > > > > > чт, 27 сент. 2018 г. в 19:45, Dmitriy Pavlov <
> > > > > > > > > dpavlov@gmail.com
> > > > > > > > > > >:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I need Vhyacheslav's opinion to be absolutely
> sure
> > > what
> > > > > > > status
> > > > > > > > is
> > > > > > > > > > > now.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We never committed to dates of release, as well.
> I
> > > > don't
> > > > > > > quite
> > > > > > > > > > > understand
> > > > > > > > > > > > > > what can mean 'the community committed to
> > > > doing/releasing
> > > > > > > > > > something'.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > About SG, I also concerned why such a big feature
> > has
> > > > > > quite a
> > > > > > > > few
> > > > > > > > > > > > > > discussions o

Re: ML examples wrap logic in IgniteThread. Why?

2018-09-27 Thread Denis Magda
Thanks Yury, good to know about that!

--
Denis

On Wed, Sep 26, 2018 at 3:49 PM Юрий Бабак  wrote:

> Denis,
>
> Thanks for this notice, actually this is some kind of atavism. Run this
> code inside IgniteThread was a requirement when we had a distributed
> matrices. But now all our algorithms builds over distributed datasets and
> we don't need it anymore.
>
> I created JIRA ticket <https://issues.apache.org/jira/browse/IGNITE-9711>
> for this.
>
> Thanks,
> Yuriy
>
> чт, 27 сент. 2018 г. в 0:20, Denis Magda :
>
> > Yury, ML folks,
> >
> > I've mentioned a strange thing. Looks like every example we have wraps up
> > its logic in the following block
> >
> > IgniteThread igniteThread = new
> > IgniteThread(ignite.configuration().getIgniteInstanceName(),
> > KMeansClusterizationExample.class.getSimpleName(), () -> {
> >
> >
> > //ML specific stuff (training, predicting, calculations, etc.)
> >
> > });
> >
> > igniteThread.start();
> > igniteThread.join();
> >
> >
> > Why do we do that?
> >
> > Denis
> >
>


Re: Apache Ignite 2.7 release

2018-09-27 Thread Denis Magda
>
> Denis, as PMC Chair, could you please control, that Service Grid
> inclusion/exclusion is discussed properly according to the Apache Way.


It's fine when committers/contributors have private discussions related to
a feature they've been working on. Not everything should go through the dev
list. Otherwise, it will be inundated.  However, agree, that architectural
and release decisions need to be done publicly.

Speaking about Service Grid, there was a discussion where I saw that it was
questionable whether it gets added to the release or not.

*Vladislav*, could you please shed some light on the current status of the
service grid?

On Thu, Sep 27, 2018 at 9:12 AM Dmitriy Pavlov 
wrote:

> Ok, let's wait for feedback from SG Author(s)/Reviewer(s) first. If it is
> not ready, ok. But I thought it is almost done.
>
> I apologize if I missed some discussion (it can happen), but
> According to the statement "our current agreement"
> I can suspect some members are making some sort of private agreements, and
> do not to discuss it on the list.
>
> Let's build consensus here first, and then name an agreement.
>
> Denis, as PMC Chair, could you please control, that Service Grid
> inclusion/exclusion is discussed properly according to the Apache Way.
>
> чт, 27 сент. 2018 г. в 18:55, Vladimir Ozerov :
>
>> Dmitriy,
>>
>> This is an outcome of current state of Service Grid - it is not ready. We
>> never committed to have it to 2.7. Our goal was to try to include it into
>> 2.7.
>>
>> On Thu, Sep 27, 2018 at 6:48 PM Dmitriy Pavlov 
>> wrote:
>>
>> > Could you please provide a reference to some thread? Probably I missed
>> it.
>> >
>> > чт, 27 сент. 2018 г. в 18:46, Vladimir Ozerov :
>> >
>> > > Our current agreement is that Service Grid is out of scope. This is a
>> > huge
>> > > feature, which hasn't entered review stage so far, We will not be
>> able to
>> > > review/fix/test it properly.
>> > >
>> > > On Thu, Sep 27, 2018 at 6:32 PM Dmitriy Pavlov > >
>> > > wrote:
>> > >
>> > > > I agree, and I prefer four weeks for stabilization* (1 Oct - 29 Oct)
>> > > >
>> > > > Do I understand it correctly: Service Grid is still in scope, isn't
>> > it? I
>> > > > find it very important.
>> > > >
>> > > > чт, 27 сент. 2018 г. в 18:28, Nikolay Izhikov > >:
>> > > >
>> > > > > Hello, Vova.
>> > > > >
>> > > > > Thank you for clear release status.
>> > > > > I'm +1 for your proposal.
>> > > > >
>> > > > > чт, 27 сент. 2018 г., 18:25 Alexey Kuznetsov <
>> akuznet...@apache.org
>> > >:
>> > > > >
>> > > > > > Vova,
>> > > > > >
>> > > > > > Huge +1 to do a stabilization.
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Alexey Kuznetsov
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>


ML examples wrap logic in IgniteThread. Why?

2018-09-26 Thread Denis Magda
Yury, ML folks,

I've mentioned a strange thing. Looks like every example we have wraps up
its logic in the following block

IgniteThread igniteThread = new
IgniteThread(ignite.configuration().getIgniteInstanceName(),
KMeansClusterizationExample.class.getSimpleName(), () -> {


//ML specific stuff (training, predicting, calculations, etc.)

});

igniteThread.start();
igniteThread.join();


Why do we do that?

Denis


Re: Apache Ignite project summary at people.apache.org

2018-09-26 Thread Denis Magda
Hi Dmitriy and thanks for putting this out,

Ignite definition needs to be changed to this one



*Memory-centric distributed database, caching, and processing platform
fortransactional, analytical, and streaming workloads delivering
in-memoryspeeds at petabyte scale*

Have no idea how to update it here: http://people.apache.org/phonebook.html

Please try to reach out to gene...@incubator.apache.org

--
Denis

On Wed, Sep 26, 2018 at 9:04 AM Dmitriy Pavlov 
wrote:

> Hi Igniters,
>
> It seems to me, that the project description at
> http://people.apache.org/phonebook.html is a little bit outdated.
>
> It says
>   High-performance, integrated and distributed in-memory platform for
> computing and transacting on large-scale data sets in real-time
>
> But our site says
>  Memory-centric distributed database, caching, and processing platform for
> transactional, analytical, and streaming workloads delivering in-memory
> speeds at petabyte scale
>
> After persistent storage donation I think the second one is more correct,
> so I suggest to update Apache info.
>
> Who can advise me how to update product description at the Apache site. Is
> it easy to do?
>
> The same issue is related to
> https://projects.apache.org/project.html?ignite
> but here I can ask at the ComDev list to update.
>
> Sincerely,
> Dmitriy Pavlov
>


Re: Wrong off-heap size is reported for a node

2018-09-26 Thread Denis Magda
Thanks, Pavel and the rest of the Igniters involved.

That simple usability improvement is a big deal for those who use Ignite in
production.

Are we getting it in 2.7?

--
Denis

On Wed, Sep 26, 2018 at 10:11 AM Dmitriy Pavlov 
wrote:

> Hi Igniters,
>
> Thanks to everyone, who has participated in this discussion and shared
> their view and ideas.
>
> I've merged fix of changes related to logging only. Fixing of cluster
> metrics can be done in a separate ticket/discussion.
>
> Pavel, thank you for your contribution and for answering my questions.
>
> Sincerely,
> Dmitriy Pavlov
>
> пн, 24 сент. 2018 г. в 18:52, Pavel Pereslegin :
>
> > Andrei,
> > I totally agree with you and I think that "ClusterMetrics" should also
> > be fixed, I'm just not sure that we should include this change in the
> > same ticket.
> > пн, 24 сент. 2018 г. в 18:43, aealexsandrov :
> > >
> > > Hi,
> > >
> > > OK, the user can use it to calculate the off-heap. But I think that the
> > > reason for your changes to fix the calculation of the nonHeap used in
> > Ignite
> > > now. For example now REST return "-1" for nonHeapMemoryMaximum. I think
> > that
> > > it can't be used somehow. So REST possible should be updated as you did
> > for
> > > log metrics and it will require for the same logic.
> > >
> > > BR,
> > > Andrei
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>


Re: [ML] New features and improvement of ML module for 2.7 release

2018-09-26 Thread Denis Magda
Enormous and outstanding addition!

Yuriy, I've talked to Akmal and he is happy to help with the documentation.
Please start documenting everything and reach out Akmal directly.

--
Denis

On Wed, Sep 26, 2018 at 10:31 AM Юрий Бабак  wrote:

> Hello Igniters,
>
> I want to make up some overview of all features and major improvement of ML
> module for this release.
>
> So let me start from the one of our main feature for this release:
>
> *TensorFlow integration* <
> https://issues.apache.org/jira/browse/IGNITE-8670>
>
> This integration allows us to use Apache Ignite as a data source for
> TensorFlow. Also, this integration will allow creating and maintaining
> TensorFlow clusters over Apache Ignite and submit TF jobs to those
> clusters. More details in the related umbrella ticket.
>
> Also, for this release we have some new algorithms:
>
> * Random forest  
> * Gradient boosted trees <
> https://issues.apache.org/jira/browse/IGNITE-7149>
> * Logistic regression[binary
> ][multi-class
> ]
> * ANN 
>
> New features related with data preprocessing:
>
> * Pipeline 
> * L1,L2 normalization 
> * Data filtering for new datasets
> 
> * Encoding categorical features [OneHotEncoder
> ][OneOfKEncoder
> ]
> * Imputer and Binarizer  >
> * MaxAbsScaler 
> * Dataset splitting 
>
> New features for a model validation:
>
> * Model estimator 
> * k-fold cross-validation
> 
> * Param grid for tuning hyper-parameters in cross-validation
> 
>
> Other features and improvements:
>
> * Model updating 
> * ML tutorial 
> * Optional indexing for decision trees
> 
> * Learning context for trainers(local parallelizing and logging of training
> process) 
> * Unification of API for feature extractor
> 
> * Several tickets for removing old unused classes and improvements for code
> coverage and examples [1 <
> https://issues.apache.org/jira/browse/IGNITE-9124>
> ][2 ][3
> ][4
> ][5
> ][6
> ]
>
> Sincerely,
> Yuriy Babak
>


Re: Volunteer needed: AWS Elastic Load Balancer IP Finders implemented

2018-09-26 Thread Denis Magda
Igniters,

Don't we have experts in our networking component? I do believe that's a
small improvement that can be verified and merged quickly.

--
Denis

On Wed, Sep 26, 2018 at 8:50 AM Dmitriy Pavlov 
wrote:

> Igniters, Friendly reminder.
>
> Denis, could you advice how to proceed in this case? It seems feature can
> be useful, but committers are not involved in a review.
>
> Should we send a proposal with lazy consensus and automatically merge? Any
> alternative ideas on how to proceed with issues stuck in the review process
> are strongly appreciated.
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 15 авг. 2018 г. в 19:38, Dmitriy Pavlov :
>
> > Hi Igniters,
> >
> > I took a look at the proposed contribution
> > https://issues.apache.org/jira/browse/IGNITE-8617 and PR
> > https://github.com/apache/ignite/pull/4076.
> >
> > I found this change reasonable and I can update code according to code
> > style.
> >
> > But I would ask someone from the community with experience with AWS ELB
> to
> > join the review. It would also benefit us if someone could try to run a
> > cluster with AWS ELB IP Finder.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
>


Re: [ML] TensorFlow intergration module release

2018-09-25 Thread Denis Magda
Great addition, Yuriy!

Adding the "ignite-tensorflow" module makes total sense to me.

--
Denis

On Tue, Sep 25, 2018 at 12:53 PM Юрий Бабак  wrote:

> Hello, Igniters.
>
> For release 2.7 we will introduce integration between TensorFlow and Apache
> Ignite. This integration contains changes on Apache Ignite side and on
> TensorFlow side.
>
> Apache Ignite part is the command line tool which allows create and
> maintain TensorFlow clusters over Apache Ignite and submit TF jobs to those
> clusters.
>
> For TensorFlow we implemented "ignite dataset". More details in related PR
> [1]
>
> As Apache Ignite part is done and TensorFlow part is ready for the merge I
> suggest to add module "ignite-tensorflow" to other Ignite deliverables. So
> I've created the ticket in JIRA for this [2]. In that case, we will be able
> to release this feature with Apache Ignite binary release includes deb/rpm
> packages.
>
> [1] https://github.com/tensorflow/tensorflow/pull/22210
> [2] https://issues.apache.org/jira/browse/IGNITE-9685
>
> Regards,
> Yury
>


Re: Spring Session Ignite plugin

2018-09-24 Thread Denis Magda
Hello Anton,

I would merge the contribution to "ignite-spring-session" module. Create
it. We already have "ignite-spring", "ignite-spring-date" modules and the
addition of one more looks natural.

--
Denis

On Mon, Sep 24, 2018 at 3:31 AM Dmitriy Pavlov 
wrote:

> Hi Anton,
>
> There is GitHub mirror of Apache Ignite codebase here:
> https://github.com/apache/ignite
>
> You can create JIRA ticket and pull request from your Apache Ignite fork.
>
> Another option is to donate to some supplementary repository, but this way
> is much more complex and requires discussion from all community and grant
> agreement signing.
>
> So I suggest creating PR.
>
> Sincerely,
> Dmitriy Pavlov
>
> пн, 24 сент. 2018 г. в 13:03, akurbanov :
>
> > Hello Igniters,
> >
> > I have implemented small community extension to use Apache Ignite as
> > backing
> > repository for session clustering with Spring Session:  github link
> >   . I was wondering what
> > would be the best way to share this with community, is there any Ignite
> > repository existing for community integrations?
> >
> > Regards,
> > anton
> >
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>


Re: Critical worker threads liveness checking drawbacks

2018-09-24 Thread Denis Magda
Andrey K. and G.,

Thanks, do we have a documentation ticket created? Prachi (copied) can help
with the documentation.

--
Denis

On Mon, Sep 24, 2018 at 5:51 AM Andrey Gura  wrote:

> Andrey,
>
> finally your change is merged to master branch. Congratulations and
> thank you very much! :)
>
> I think that the next step is feature that will allow signal about
> blocked threads to the monitoring tools via MXBean.
>
> I hope you will continue development of this feature and provide your
> vision in new JIRA issue.
>
>
> On Tue, Sep 11, 2018 at 6:54 PM Andrey Kuznetsov 
> wrote:
> >
> > David, Maxim!
> >
> > Thanks a lot for you ideas. Unfortunately, I can't adopt all of them
> right
> > now: the scope is much broader than the scope of the change I implement.
> I
> > have had a talk to a group of Ignite commiters, and we agreed to complete
> > the change as follows.
> > - Blocking instructions in system-critical which may resonably last long
> > should be explicitly excluded from the monitoring.
> > - Failure handlers should have a setting to suppress some failures on
> > per-failure-type basis.
> > According to this I have updated the implementation: [1]
> >
> > [1] https://github.com/apache/ignite/pull/4089
> >
> > пн, 10 сент. 2018 г. в 22:35, David Harvey :
> >
> > > When I've done this before,I've needed to find the oldest  thread, and
> kill
> > > the node running that.   From a language standpoint, Maxim's "without
> > > progress" better than "heartbeat".   For example, what I'm most
> interested
> > > in on a distributed system is which thread started the work it has not
> > > completed the earliest, and when did that thread last make forward
> > > process. You don't want to kill a node because a thread is waiting
> on a
> > > lock held by a thread that went off-node and has not gotten a response.
> > > If you don't understand the dependency relationships, you will make
> > > incorrect recovery decisions.
> > >
> > > On Mon, Sep 10, 2018 at 4:08 AM Maxim Muzafarov 
> > > wrote:
> > >
> > > > I think we should find exact answers to these questions:
> > > >  1. What `critical` issue exactly is?
> > > >  2. How can we find critical issues?
> > > >  3. How can we handle critical issues?
> > > >
> > > > First,
> > > >  - Ignore uninterruptable actions (e.g. worker\service shutdown)
> > > >  - Long I/O operations (should be a configurable timeout for each
> type of
> > > > usage)
> > > >  - Infinite loops
> > > >  - Stalled\deadlocked threads (and\or too many parked threads,
> exclude
> > > I/O)
> > > >
> > > > Second,
> > > >  - The working queue is without progress (e.g. disco, exchange
> queues)
> > > >  - Work hasn't been completed since the last heartbeat (checking
> > > > milestones)
> > > >  - Too many system resources used by a thread for the long period of
> time
> > > > (allocated memory, CPU)
> > > >  - Timing fields associated with each thread status exceeded a
> maximum
> > > time
> > > > limit.
> > > >
> > > > Third (not too many options here),
> > > >  - `log everything` should be the default behaviour in all these
> cases,
> > > > since it may be difficult to find the cause after the restart.
> > > >  - Wait some interval of time and kill the hanging node (cluster
> should
> > > be
> > > > configured stable enough)
> > > >
> > > > Questions,
> > > >  - Not sure, but can workers miss their heartbeat deadlines if CPU
> loads
> > > up
> > > > to 80%-90%? Bursts of momentary overloads can be
> > > > expected behaviour as a normal part of system operations.
> > > >  - Why do we decide that critical thread should monitor each other?
> For
> > > > instance, if all the tasks were blocked and unable to run,
> > > > node reset would never occur. As for me, a better solution is to
> use
> > > a
> > > > separate monitor thread or pool (maybe both with software
> > > > and hardware checks) that not only checks heartbeats but
> monitors the
> > > > other system as well.
> > > >
> > > > On Mon, 10 Sep 2018 at 00:07 David Harvey 
> wrote:
> > > >
> > > > > It would be safer to restart the entire cluster than to remove the
> last
> > > > > node for a cache that should be redundant.
> > > > >
> > > > > On Sun, Sep 9, 2018, 4:00 PM Andrey Gura  wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I agree with Yakov that we can provide some option that manage
> worker
> > > > > > liveness checker behavior in case of observing that some worker
> is
> > > > > > blocked too long.
> > > > > > At least it will  some workaround for cases when node fails is
> too
> > > > > > annoying.
> > > > > >
> > > > > > Backups count threshold sounds good but I don't understand how it
> > > will
> > > > > > help in case of cluster hanging.
> > > > > >
> > > > > > The simplest solution here is alert in cases of blocking of some
> > > > > > critical worker (we can improve WorkersRegistry for this purpose
> and
> > > > > > expose list of blocked workers) and optionally call system
> configured
> > > > > > 

Re: IgniteEvents for MVCC caches

2018-09-22 Thread Denis Magda
Vladimir,

That's a good question. I confused _ENTRY_ events with those mentioned by
you.

Anyway, are you saying that the _ENTRY_ events are workable but have no
value or they are not workable any longer with the MVCC?

--
Denis



On Sat, Sep 22, 2018 at 4:10 AM Vladimir Ozerov 
wrote:

> Denis,
>
> What is the point of these events, provided that we already has CACHE_PUT
> and CACHE_REMOVED?
>
> On Sat, Sep 22, 2018 at 1:01 AM Denis Magda  wrote:
>
> > Hello Ivan,
> >
> > First of all, it looks like that EVT_CACHE_ENTRY_CREATED,
> > > EVT_CACHE_ENTRY_DESTROYED are legacy came from Ignite 1.x time and
> have a
> > > little value nowadays. Does anyone have something against deprecating
> > them?
> >
> >
> > These events are used in a pure caching or in-memory data grid use cases
> > where key-value is a primary access pattern. I personally know several
> > companies who rely on those events. So, we can't discontinue them, not
> > everyone uses SQL. What's a hurdle of making them working with MVCC?
> >
> > --
> > Denis
> >
> > On Fri, Sep 21, 2018 at 6:44 AM Павлухин Иван 
> wrote:
> >
> > > Hi Igniters,
> > >
> > > As you might know MVCC was introduced in Apache Ignite. We started
> > > IgniteEvents support implementation for MVCC caches and faced some
> > > obstacles. I would like to start a discussion about next steps which
> > should
> > > be done to deal with current problems. The main theme is about
> > > EventType.EVTS_CACHE and key-value API.
> > >
> > > First of all, it looks like that EVT_CACHE_ENTRY_CREATED,
> > > EVT_CACHE_ENTRY_DESTROYED are legacy came from Ignite 1.x time and
> have a
> > > little value nowadays. Does anyone have something against deprecating
> > them?
> > >
> > > Second, there could be problems with EVT_CACHE_OBJECT_UNLOCKED event,
> > > because for MVCC all locks are released on transaction end. And it does
> > not
> > > sound good idea to track all locked entries during transaction
> execution,
> > > in MVCC we are not keeping entries modified participating in
> transaction
> > in
> > > private working set in memory. One possible solution is deprecation of
> > > lock, unlock events. Another one is introducing special lock event for
> > > MVCC, but it will be confusing. Also, I see an option of firing only
> > > EVT_CACHE_OBJECT_LOCKED.
> > >
> > > Last, is different number of events fired for similar scenarios and
> > > different cache modes. Let's consider "put, remove, put" for the same
> key
> > > and different modes. For ATOMIC 2 put and 1 remove event will be fired.
> > For
> > > TRANSACTIONAL 1 put will be fired in case of commit and nothing in case
> > of
> > > rollback. For MVCC in current vision 2 put will be fired regardless
> > whether
> > > transaction was committed and rolled back. Currently I do not see
> options
> > > how to overcome it.
> > >
> > > Also, I hardly imagine current use cases for cache events. I think that
> > > understanding them is the best way for developing working solution for
> > > MVCC.
> > >
> > > I need your opinions.
> > >
> > > 2018-09-21 12:54 GMT+03:00 Павлухин Иван :
> > >
> > > > Hi Igniters,
> > > >
> > > > As you might know MVCC was introduced in Apache Ignite. We started
> > > > IgniteEvents implementation for MVCC caches and faced some
> obstacles. I
> > > > would like to start a discussion about next steps which should be
> done
> > to
> > > > deal with current problems.
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
> >
>


Fwd: .NET java thread count keeps growing

2018-09-21 Thread Denis Magda
Pavel,

Do you think we can get it fixed in 2.7 time frame?

--
Denis

-- Forwarded message -
From: Ilya Kasnacheev 
Date: Tue, Sep 18, 2018 at 9:30 AM
Subject: Re: .NET java thread count keeps growing
To: 


Hello!

I can observe the problem that you are describing. I have created a ticket
https://issues.apache.org/jira/browse/IGNITE-9638

Regards,
-- 
Ilya Kasnacheev


пн, 17 сент. 2018 г. в 20:21, Alew :

> Hi, I found a way to reproduce the issue.
>
> I think it is related to .net thread identity.
>
> New .net thread leads to new java thread.
>
>
>
> On 17/09/2018 15:04, Ilya Kasnacheev wrote:
>
> Hello!
>
> You seem to have 3000 threads of form "Thread-", which is not what
> Ignite usually uses, and they're all empty.
> This is mysterious so I urge you to share the sample reproducer (or,
> barring that, code snippet) that leads to such behavior.
>
> By the way, do you observe any errors in the log?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вс, 16 сент. 2018 г. в 4:09, Alew :
>
>> Hi!
>>
>> I have growing java thread count. The application crashes with OOME. The
>> application is very simple, only reading values in a cache.
>>
>> Any suggestion to debug this issue?
>>
>> Ignite 2.4
>>
>>
>>
>


Re: Python thin client installation instructions

2018-09-21 Thread Denis Magda
I would add a disclaimer or a prerequisite step. That what other companies
do if a user needs to do some basic installation steps. At least mention it.

--
Denis

On Fri, Sep 21, 2018 at 3:04 PM Prachi Garg  wrote:

> Hi Dmitry,
>
> Thank you for taking the time to explain me everything in such detail :)
>
> I am trying to do this because I have to document. In general, I am
> assuming that a Python thin client user would already have Python installed
> and be using it. So, I would not suggest adding any disclaimers regarding
> Python installation.
>
> The example works with python3 command :)
>
>
> -P
>
>
>
> On Thu, Sep 20, 2018 at 8:31 PM, Dmitry Melnichuk <
> dmitry.melnic...@nobitlost.com> wrote:
>
> > Prachi,
> >
> > I feel your struggle. It is easier for end user to perceive Python 2 and
> > Python 3 as different languages, not as versions of one language. They
> > usually installed alongside each other; their updates are handled
> > separately. On most systems they have their respective shell commands:
> > `python2` and `python3`.
> >
> > Shell command `python` can be viewed as an alias of either `python2` or
> > `python3`. I use an Arch Linux derivative, where `python` is `python3`.
> > Most other GNU/Linux OSes use `python` as an alias of `python2`. I am not
> > sure about MacOS. On Windows the latest Python distribution installed
> > overrule PATH environment variables, so it impossible to predict the
> > “default” Python version (2 or 3).
> >
> > Luckily, virtualenv was introduced to leverage all these issues. It is
> > able to handle multiple isolated Python environments, where the `python`
> > command is set upon the creation of the environment, while the
> > environment-specific package dependencies are handled transparently with
> > pip.
> >
> > But the use of pyignite should not be limited to virtualenv. There are
> > many cases when the use of virtualenv is discouraged or even impossible.
> > For example, when deploying Python app into an OS-level container or
> > similar isolating environment, virtualenv would be just a useless
> overhead.
> > There are also non-standard Python distributions (used mostly on Windows)
> > that do not support virtualenv.
> >
> > I am sorry, that users who are not proficient in Python can have so many
> > problems with following my documentation. But still it seems obvious for
> > me, that all the details of organizing user's own Python environment are
> > out of pyignite documentation's scope. The only thing I can suggest to
> > improve my documentation in this regard is putting a big bold foreword
> like
> > this:
> >
> >   It is assumed in this document that you know how to install
> >   and use Python 3 on your system. Please consult your OS manual pages
> >   or documentations of your specific Python 3 distribution regarding
> >   the details of organizing your Python 3 environment. The use of
> >   virualenv for development with pyignite is highly recommended.
> >
> > But, frankly, I have not seen such disclaimers in the wild and not sure
> if
> > it would be useful. It is very vague and do not cover any of the
> potential
> > pitfalls.
> >
> > I am sorry for giving such a lengthy explanation here, though I've been
> > asked a very specific question. I understand you may not have time to
> > invest in learning virtualenv. If so, you did everything right, just use
> > `python3` command for launching examples:
> >
> > ```
> > $ python3 get_and_put.py
> > ```
> >
> > On 9/21/18 10:34 AM, Prachi Garg wrote:
> >
> >> Hi Dmitry,
> >>
> >> Sorry, I am not familiar with Python.
> >>
> >> So there are more issues...
> >>
> >> 1. The version on my mac remains 2.7.10 even though I tried to link to
> >> the new version.
> >>
> >> ~$ python --version
> >> Python 2.7.10
> >>
> >> ~$ brew unlink python && brew link --overwrite python3
> >> Unlinking /usr/local/Cellar/python/3.7.0. .. 25 symlinks
> >> removed
> >> Linking /usr/local/Cellar/python/3.7.0. .. 25 symlinks
> >> created
> >>
> >> ~$ python --version
> >> Python 2.7.10
> >>
> >> 2. Then I tried to update /pip/, uninstall and re-install /pyignite/
> >>
> >> ~$ pip install -U pip
> >> -bash: /Library/Frameworks/Python.framework/Versions/3.7/bin/pip: No
> >> such file or directory
> >> ~$ pip3 install -U pip
> >> Requirement already up-to-date: pip in /Library/Frameworks/Python.fra
> >> mework/Versions/3.7/lib/python3.7/site-packages (18.0)
> >>
> >> ~$ pip3 uninstall pyignite
> >> Uninstalling pyignite-0.3.0:
> >>Would remove:
> >>  /Library/Frameworks/Python.framework/Versions/3.7/lib/pytho
> >> n3.7/site-packages/pyignite-0.3.0.dist-info/*
> >>  /Library/Frameworks/Python.framework/Versions/3.7/lib/pytho
> >> n3.7/site-packages/pyignite/*
> >> Proceed (y/n)? y
> >>Successfully uninstalled pyignite-0.3.0
> >>
> >> ~$ pip3 install pyignite
> >> Collecting pyignite
> >>Using cached https://files.pythonhosted.org
> >> 

Re: Page IO statistics for Ignite

2018-09-21 Thread Denis Magda
Hello Yuri,

I might give useful feedback if see how the metrics will look like from the
API standpoint. If it's not difficult please create a draft.

AS for the interface, in addition to JMX and SQL we need to ensure Visor
CMD and Web Console gets updated. *Alex K.*, please join the thread and
share your requirements.

--
Denis

On Fri, Sep 21, 2018 at 8:16 AM Юрий  wrote:

> Hi Igniters,
>
> I started IGNITE-8580
>  ticket
> related to print page read/write metrics and did some investigation what
> other databases provide for the similar purposes.
>
> Based on the investigation I want to propose my raw vision of how to IGNITE
> can be more transparent from performance perspective.
>
> Need to collect statistics for logical (from memory) and physical (from
> storage) page reads/writes. All these metrics should be separated by next
> dimensions:
> 1) index/cache
> 2) query level
> 3) node/cluster
> ...
>
> Seems the statistics should be limited by time.
>
> If we will have such statistics we could realize such things as:
> 1) Get IO statistics per SQL query, global or/and splitted by
> indexes/caches.
> 2) Have ability to understand why performance goes down in case it related
> to IO. For example on concrete node or cache.
> 3) Evaluate effectiveness of use indexes. Find unused indexes.
> 4) Keep TOP queries with aggressive physical reads
> 
>
>
> Such statistics could be available least at JMX and SQL interfaces.
>
> Let's discuss. In case it will be interested for you I can dig deeper into
> the area and prepare IEP based on our discussion.
>
>
> Igniters, what do you think?
>
>
>
>
> --
> Живи с улыбкой! :D
>


Re: IgniteEvents for MVCC caches

2018-09-21 Thread Denis Magda
Hello Ivan,

First of all, it looks like that EVT_CACHE_ENTRY_CREATED,
> EVT_CACHE_ENTRY_DESTROYED are legacy came from Ignite 1.x time and have a
> little value nowadays. Does anyone have something against deprecating them?


These events are used in a pure caching or in-memory data grid use cases
where key-value is a primary access pattern. I personally know several
companies who rely on those events. So, we can't discontinue them, not
everyone uses SQL. What's a hurdle of making them working with MVCC?

--
Denis

On Fri, Sep 21, 2018 at 6:44 AM Павлухин Иван  wrote:

> Hi Igniters,
>
> As you might know MVCC was introduced in Apache Ignite. We started
> IgniteEvents support implementation for MVCC caches and faced some
> obstacles. I would like to start a discussion about next steps which should
> be done to deal with current problems. The main theme is about
> EventType.EVTS_CACHE and key-value API.
>
> First of all, it looks like that EVT_CACHE_ENTRY_CREATED,
> EVT_CACHE_ENTRY_DESTROYED are legacy came from Ignite 1.x time and have a
> little value nowadays. Does anyone have something against deprecating them?
>
> Second, there could be problems with EVT_CACHE_OBJECT_UNLOCKED event,
> because for MVCC all locks are released on transaction end. And it does not
> sound good idea to track all locked entries during transaction execution,
> in MVCC we are not keeping entries modified participating in transaction in
> private working set in memory. One possible solution is deprecation of
> lock, unlock events. Another one is introducing special lock event for
> MVCC, but it will be confusing. Also, I see an option of firing only
> EVT_CACHE_OBJECT_LOCKED.
>
> Last, is different number of events fired for similar scenarios and
> different cache modes. Let's consider "put, remove, put" for the same key
> and different modes. For ATOMIC 2 put and 1 remove event will be fired. For
> TRANSACTIONAL 1 put will be fired in case of commit and nothing in case of
> rollback. For MVCC in current vision 2 put will be fired regardless whether
> transaction was committed and rolled back. Currently I do not see options
> how to overcome it.
>
> Also, I hardly imagine current use cases for cache events. I think that
> understanding them is the best way for developing working solution for
> MVCC.
>
> I need your opinions.
>
> 2018-09-21 12:54 GMT+03:00 Павлухин Иван :
>
> > Hi Igniters,
> >
> > As you might know MVCC was introduced in Apache Ignite. We started
> > IgniteEvents implementation for MVCC caches and faced some obstacles. I
> > would like to start a discussion about next steps which should be done to
> > deal with current problems.
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>


Re: [IMPORTANT] Apache Ignite 2.7 and Java 11 support

2018-09-20 Thread Denis Magda
In general, I'm totally for the plan 2 - make sure Ignite works with Java
11 and release it in October. However, are we sure we'll be ready to adopt
JTA and Hibernate integrations? We can't release having them broken.

--
Denis

On Thu, Sep 20, 2018 at 4:00 PM Vladimir Ozerov 
wrote:

> Igniters, we have a problem.
>
> *TL;DR;*
> Ignite may be seriously broken in Java 11. This affects ignite.sh,
> Hibernate integration, JTA integration. And we cannot test it before code
> freeze due to Java 11 release schedule.
>
> We need to understand whether we shift release, or plan immediate AI 2.8
> afterwarвs, or ignore the problem until a number of user compliants appear.
>
> *Long story*
> As you may know we already put some efforts on Java 9 support in Ignite
> [1]. Specifically, during earlier releases we reworked all code affected by
> Java 9 changes and added several "--add-export" and "--add-module" flags to
> support some packages which are not accessible by default. We never
> implemented any modules in Ignite.
>
> As a result, currently Apache Ignite mostly works fine with Java 9. If node
> is started in standalone mode, we add mentioned flags to JVM arguments by
> default, and no actions are needed from user side. If node is started in
> embedded mode, user has to provide required flags manually [2]. This is
> acceptable state for us until module subsystem is integrated somehow with
> the product.
>
> But then we decided to perform extensive testing of current master on Java
> 9/10/11 versions. Thanks to Peter Ivanov, we setup required environment.
> During this activity we read more docs about Java 11. We revealed, that in
> this release a number of packages we depend on will be removed completely
> from JDK as a part of JEP 320 [3]. *JTA *and *Hibernate* integrations will
> stop work out of the box. Moreover, "--add-module" flag will stop working,
> what may affect ignite.sh.
>
> Things are even worse because Java 11 will be released exactly by our
> planned code freeze date, so we cannot even test it appropriately right
> now. So we need to revisit out Java 9+ support strategy for the nearest
> releases.
>
> *Possible solutions*
> 1) Relax and move Java 9+ support to AI 2.8 scope
> Pros: Java 8 will be supported till January 2019 [4] so we still have some
> time. We can plan AI 2.8 to Nov-Dec this year.
> Cons: more and more users will try Java 11 (not Java 9 or 10, they will be
> hidden from official page) during this time, and without Java 11 testing we
> may end up with not-working product.
>
> 2) Move AI 2.7 code freeze to the middle of October to have a time to test
> and fix big problems with Java 11.
> Pros: Java 11 will be released in the end of the next week [5]. We take
> some additional time to test us with Java 11, fix what can be fixed, find
> and document workaround for things which cannot be fixed.
> Cons: AI 2.7 will be released in the end of October.
>
> Another small "cons" for the second approach is that we will have more time
> for MVCC stabilization, and improve chances of service grid to be included
> into release (from what I heard from Nikolay and Vyacheslav, there is a
> good progress for now). But remember that our previous expirience with
> things like that is constantly shifting release dates.
>
> Please share your thoughts on what should we do with Java 11.
>
> Vladimir.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-6728
> [2] https://issues.apache.org/jira/browse/IGNITE-9288
> [3] http://openjdk.java.net/jeps/320
> [4] https://www.oracle.com/technetwork/java/javase/eol-135779.html
> [5] http://www.java-countdown.xyz/
>


Re: Apache Ignite 2.7 - Service Grid Redesign

2018-09-19 Thread Denis Magda
Val can be the final reviewer. He knows this component pretty well.

Val, could you help the guys with this?

--
Denis

On Wed, Sep 19, 2018 at 3:14 AM Anton Vinogradov  wrote:

> Nikolay,
>
> I'll perform prereview once PR will be ready.
> Then we'll need some time to fix all found issues.
> After that final review by another expert will be required.
>
> Let's define the deadline for final PR and reviewer (Alexey Goncharuk?).
> In other words, let's define the exact date until which final PR should be
> provided to have chances to become a part of 2.7.
>
>
> ср, 19 сент. 2018 г. в 12:46, Nikolay Izhikov :
>
> > Hello, Igniters.
> >
> > Currently, we are working on release scope for Apache Ignite 2.7
> > We had plans to include "Service Grid Redesign. Phase 1" to 2.7 release.
> >
> > If I understand correctly, the plan is following - We should have PR that
> > is ready for review at the end of September.
> >
> > This deadline is very close to the code freeze date.
> >
> > Let's discuss, how we should handle this risks?
> > Who will be able to review this PR when it's will be ready?
> > Vyacheslav, can you comment on this?
> >
>


Re: Cache scan efficiency

2018-09-18 Thread Denis Magda
Agree, it's just a matter of the documentation. If a user stores 100% in
RAM and on disk, and just wants to warm RAM up after a restart then he
knows everything will fit there. If during the preloading we detect that
the RAM is exhausted we can halt it and print out a warning.

--
Denis

On Tue, Sep 18, 2018 at 2:10 PM Dmitriy Pavlov 
wrote:

> Hi,
>
> I totally support the idea of cache preload.
>
> IMO it can be expanded. We can iterate over local partitions of the cache
> group and preload each.
>
> But it should be really clear documented methods so a user can be aware of
> the benefits of such method (e.g. if RAM region is big enough, etc).
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 18 сент. 2018 г. в 21:36, Denis Magda :
>
> > Folks,
> >
> > Since we're adding a method that would preload a certain partition, can
> we
> > add the one which will preload the whole cache? Ignite persistence users
> > I've been working with look puzzled once they realize there is no way to
> > warm up RAM after the restart. There are use cases that require this.
> >
> > Can the current optimizations be expanded to the cache preloading use
> case?
> >
> > --
> > Denis
> >
> > On Tue, Sep 18, 2018 at 3:58 AM Alexei Scherbakov <
> > alexey.scherbak...@gmail.com> wrote:
> >
> > > Summing up, I suggest adding new public
> > > method IgniteCache.preloadPartition(partId).
> > >
> > > I will start preparing PR for IGNITE-8873
> > > <https://issues.apache.org/jira/browse/IGNITE-8873> if no more
> > objections
> > > follow.
> > >
> > >
> > >
> > > вт, 18 сент. 2018 г. в 10:50, Alexey Goncharuk <
> > alexey.goncha...@gmail.com
> > > >:
> > >
> > > > Dmitriy,
> > > >
> > > > In my understanding, the proper fix for the scan query looks like a
> big
> > > > change and it is unlikely that we include it in Ignite 2.7. On the
> > other
> > > > hand, the method suggested by Alexei is quite simple  and it
> definitely
> > > > fits Ignite 2.7, which will provide a better user experience. Even
> > > having a
> > > > proper scan query implemented this method can be useful in some
> > specific
> > > > scenarios, so we will not have to deprecate it.
> > > >
> > > > --AG
> > > >
> > > > пн, 17 сент. 2018 г. в 19:15, Dmitriy Pavlov  >:
> > > >
> > > > > As I understood it is not a hack, it is an advanced feature for
> > warming
> > > > up
> > > > > the partition. We can build warm-up of the overall cache by calling
> > its
> > > > > partitions warm-up. Users often ask about this feature and are not
> > > > > confident with our lazy upload.
> > > > >
> > > > > Please correct me if I misunderstood the idea.
> > > > >
> > > > > пн, 17 сент. 2018 г. в 18:37, Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > >:
> > > > >
> > > > > > I would rather fix the scan than hack the scan. Is there any
> > > technical
> > > > > > reason for hacking it now instead of fixing it properly? Can some
> > of
> > > > the
> > > > > > experts in this thread provide an estimate of complexity and
> > > difference
> > > > > in
> > > > > > work that would be required for each approach?
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Mon, Sep 17, 2018 at 4:42 PM Alexey Goncharuk <
> > > > > > alexey.goncha...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I think it would be beneficial for some Ignite users if we
> added
> > > > such a
> > > > > > > partition warmup method to the public API. The method should be
> > > > > > > well-documented and state that it may invalidate existing page
> > > cache.
> > > > > It
> > > > > > > will be a very effective instrument until we add the proper
> scan
> > > > > ability
> > > > > > > that Vladimir was referring to.
> > > > > > >
> > > > > > > пн, 17 сент. 2018 г. в 13:05, Maxim Muzafarov <
> > maxmu...@gmail.com
> > > >:
> > > > > > >
> > > > > > > > Folk

Re: Cache scan efficiency

2018-09-18 Thread Denis Magda
Folks,

Since we're adding a method that would preload a certain partition, can we
add the one which will preload the whole cache? Ignite persistence users
I've been working with look puzzled once they realize there is no way to
warm up RAM after the restart. There are use cases that require this.

Can the current optimizations be expanded to the cache preloading use case?

--
Denis

On Tue, Sep 18, 2018 at 3:58 AM Alexei Scherbakov <
alexey.scherbak...@gmail.com> wrote:

> Summing up, I suggest adding new public
> method IgniteCache.preloadPartition(partId).
>
> I will start preparing PR for IGNITE-8873
>  if no more objections
> follow.
>
>
>
> вт, 18 сент. 2018 г. в 10:50, Alexey Goncharuk  >:
>
> > Dmitriy,
> >
> > In my understanding, the proper fix for the scan query looks like a big
> > change and it is unlikely that we include it in Ignite 2.7. On the other
> > hand, the method suggested by Alexei is quite simple  and it definitely
> > fits Ignite 2.7, which will provide a better user experience. Even
> having a
> > proper scan query implemented this method can be useful in some specific
> > scenarios, so we will not have to deprecate it.
> >
> > --AG
> >
> > пн, 17 сент. 2018 г. в 19:15, Dmitriy Pavlov :
> >
> > > As I understood it is not a hack, it is an advanced feature for warming
> > up
> > > the partition. We can build warm-up of the overall cache by calling its
> > > partitions warm-up. Users often ask about this feature and are not
> > > confident with our lazy upload.
> > >
> > > Please correct me if I misunderstood the idea.
> > >
> > > пн, 17 сент. 2018 г. в 18:37, Dmitriy Setrakyan  >:
> > >
> > > > I would rather fix the scan than hack the scan. Is there any
> technical
> > > > reason for hacking it now instead of fixing it properly? Can some of
> > the
> > > > experts in this thread provide an estimate of complexity and
> difference
> > > in
> > > > work that would be required for each approach?
> > > >
> > > > D.
> > > >
> > > > On Mon, Sep 17, 2018 at 4:42 PM Alexey Goncharuk <
> > > > alexey.goncha...@gmail.com>
> > > > wrote:
> > > >
> > > > > I think it would be beneficial for some Ignite users if we added
> > such a
> > > > > partition warmup method to the public API. The method should be
> > > > > well-documented and state that it may invalidate existing page
> cache.
> > > It
> > > > > will be a very effective instrument until we add the proper scan
> > > ability
> > > > > that Vladimir was referring to.
> > > > >
> > > > > пн, 17 сент. 2018 г. в 13:05, Maxim Muzafarov  >:
> > > > >
> > > > > > Folks,
> > > > > >
> > > > > > Such warming up can be an effective technique for performing
> > > > calculations
> > > > > > which required large cache
> > > > > > data reads, but I think it's the single narrow use case of all
> over
> > > > > Ignite
> > > > > > store usages. Like all other
> > > > > > powerfull techniques, we should use it wisely. In the general
> > case, I
> > > > > think
> > > > > > we should consider other
> > > > > > techniques mentioned by Vladimir and may create something like
> > > `global
> > > > > > statistics of cache data usage`
> > > > > > to choose the best technique in each case.
> > > > > >
> > > > > > For instance, it's not obvious what would take longer:
> multi-block
> > > > reads
> > > > > or
> > > > > > 50 single-block reads issues
> > > > > > sequentially. It strongly depends on used hardware under the hood
> > and
> > > > > might
> > > > > > depend on workload system
> > > > > > resources (CPU-intensive calculations and I\O access) as well.
> But
> > > > > > `statistics` will help us to choose
> > > > > > the right way.
> > > > > >
> > > > > >
> > > > > > On Sun, 16 Sep 2018 at 23:59 Dmitriy Pavlov <
> dpavlov@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi Alexei,
> > > > > > >
> > > > > > > I did not find any PRs associated with the ticket for check
> code
> > > > > changes
> > > > > > > behind this idea. Are there any PRs?
> > > > > > >
> > > > > > > If we create some forwards scan of pages, it should be a very
> > > > > > intellectual
> > > > > > > algorithm including a lot of parameters (how much RAM is free,
> > how
> > > > > > probably
> > > > > > > we will need next page, etc). We had the private talk about
> such
> > > idea
> > > > > > some
> > > > > > > time ago.
> > > > > > >
> > > > > > > By my experience, Linux systems already do such forward reading
> > of
> > > > file
> > > > > > > data (for corresponding sequential flagged file descriptors),
> but
> > > > some
> > > > > > > prefetching of data at the level of application may be useful
> for
> > > > > > O_DIRECT
> > > > > > > file descriptors.
> > > > > > >
> > > > > > > And one more concern from me is about selecting a right place
> in
> > > the
> > > > > > system
> > > > > > > to do such prefetch.
> > > > > > >
> > > > > > > Sincerely,
> > > > > > > Dmitriy Pavlov
> > > > > > >
> > > > > > > вс, 16 сент. 2018 

Re: Data streaming using Apache Ignite and Flink

2018-09-12 Thread Denis Magda
Prachi,

Did you forget to add Saikat's blog to the list?

--
Denis

On Wed, Sep 12, 2018 at 1:44 PM Dmitriy Pavlov 
wrote:

> Hi Denis,
>
> I didn't find the blog post there.
>
> Could you please add it or advice me how can I do it?
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 28 авг. 2018 г. в 4:19, Saikat Maitra :
>
> > Thank you so much Denis.
> >
> > Regards,
> > Saikat
> >
> > On Mon, Aug 27, 2018 at 5:00 PM, Denis Magda  wrote:
> >
> > > Hello Saikat,
> > >
> > > Thanks for the article and contribution. We'll get it added to:
> > > https://ignite.apache.org/blogs.html
> > >
> > > --
> > > Denis
> > >
> > > On Sun, Aug 26, 2018 at 2:59 PM Saikat Maitra  >
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > I recently published blog on how we can stream data using Apache
> Ignite
> > > and
> > > > Flink. This uses IgniteSink with recent changes merged (release due
> in
> > > > 2.7.0) which will allow us to run IgniteSink using Apache Flink in
> > > cluster
> > > > mode.
> > > >
> > > >
> > > >
> > > > https://samaitra.blogspot.com/2018/08/data-streaming-using-
> > > apache-flink-and.html
> > > >
> > > > Please review and let me know if you have feedback.
> > > >
> > > > Regards,
> > > > Saikat
> > > >
> > >
> >
>


[jira] [Created] (IGNITE-9564) Ignite K8 service and sticky session with AWS K8 deployments

2018-09-12 Thread Denis Magda (JIRA)
Denis Magda created IGNITE-9564:
---

 Summary: Ignite K8 service and sticky session with AWS K8 
deployments
 Key: IGNITE-9564
 URL: https://issues.apache.org/jira/browse/IGNITE-9564
 Project: Ignite
  Issue Type: Task
Reporter: Denis Magda
Assignee: Ilya Murchenko
 Fix For: 2.8


_sessionAffinity_ parameter is not supported by AWS Kubernetes which causes the 
failure of the standard Ignite service deployment:
http://apache-ignite-users.70518.x6.nabble.com/Error-installing-Ignite-on-K8s-td23999.html

See how Ignite service configuration have to be updated considering sticky 
session capabilities of AWS:
https://aws.amazon.com/elasticloadbalancing/details/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: TDE Implementation details.

2018-09-12 Thread Denis Magda
Hello Nikolay,

Excellent progress, look forward to seeing the TDE released in 2.7!

--
Denis

On Wed, Sep 12, 2018 at 2:47 AM Nikolay Izhikov  wrote:

> Hello, Denis.
>
> > Could you please list the limitations of Phase 1?
> > I'm curious what won't be supported in 2.7.
>
> 1. We will have ability to encrypt data stored on the disk for specific
> cache.
>
> - There is no limitation on API usage or something for an
> encrypted cache.
> - If some cache in a cache group is encrypted, all other caches in
> this group must be encrypted.
> - Setting up master key(standard jdk key storage) is prerequisite
> and should be done by an administrator.
> - `encryptionEnabled` flag setting up on cache creation and can't
> be changed in runtime.
>
> 2. We won't be able to change encryption keys for existing cache(key
> rotation procedure).
> This will be implemented in Phase 2.
>
> В Вт, 11/09/2018 в 23:41 -0400, Denis Magda пишет:
> > Nikolay,
> >
> > Could you please list the limitations of Phase 1? I'm curious what won't
> be
> > supported in 2.7.
> >
> > --
> > Denis
> >
> > On Tue, Sep 11, 2018 at 4:29 PM Nikolay Izhikov 
> wrote:
> >
> > > > As I understand the plan is to get TDE Phase 1 released in 2.7,
> right?
> > >
> > > Yes. We will release TDE in 2.7
> > >
> > > > Could you also list what needs to be done at Phase 2 and how much
> time
> > >
> > > it might take.
> > >
> > > Yes, I think Dmitry Ryabov will send Phase 2 design
> > >
> > >
> > > В Вт, 11/09/2018 в 23:27 +0300, Nikolay Izhikov пишет:
> > > > Hello, Denis.
> > > >
> > > > Yes, Vladimir made 2 rounds of review.
> > > > I planning to fix all issues with implementation in a few days.
> > > >
> > > >
> > > > В Вт, 11/09/2018 в 10:40 -0400, Denis Magda пишет:
> > > > > Hi Nikolay,
> > > > >
> > > > > Has anybody started looking into your request? As I understand the
> > >
> > > plan is
> > > > > to get TDE Phase 1 released in 2.7, right?
> > > > >
> > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89067473
> > > > >
> > > > > Could you also list what needs to be done at Phase 2 and how much
> time
> > >
> > > it
> > > > > might take.
> > > > >
> > > > > --
> > > > > Denis
> > > > >
> > > > > On Thu, Aug 9, 2018 at 8:48 AM Nikolay Izhikov <
> nizhi...@apache.org>
> > >
> > > wrote:
> > > > >
> > > > > > Hello, Igniters.
> > > > > >
> > > > > > I want to share with you TDE implementation details.
> > > > > > I think it can simplify review and acception of TDE
> implementation.
> > > > > > This mail is copy of wiki page [1].
> > > > > >
> > > > > > Please, share your thoughts.
> > > > > >
> > > > > > TDE is ready for review [2].
> > > > > > Please, let me know, who is able to make final review.
> > > > > >
> > > > > > This document describes some internal details of TDE.Phase 1
> > > > > > implementation [3].
> > > > > > I suggest that reader familiar with the general design described
> in
> > >
> > > IEP-18
> > > > > > [4].
> > > > > >
> > > > > >
> > > > > > Cache group key management and node join enhancements:
> > > > > >
> > > > > > 1. GridEncryptionManager encapsulates all logic related
> to
> > >
> > > key
> > > > > > management:
> > > > > > a. All group encryption keys are stored in
> MetaStore.
> > > > > >
> > > > > > b. Joining node sends to cluster:
> > > > > > * Master key digest.
> > > > > > * All group keys saved locally
> (Map > > > > > byte[]>). Keys send over a network in encrypted form.
> > > > > >
> > > > > > c. Coordinator on new node join:
> > > > > > * Compares new node master key digest
> with
> > >
> > > the

Re: TDE Implementation details.

2018-09-11 Thread Denis Magda
Nikolay,

Could you please list the limitations of Phase 1? I'm curious what won't be
supported in 2.7.

--
Denis

On Tue, Sep 11, 2018 at 4:29 PM Nikolay Izhikov  wrote:

> > As I understand the plan is to get TDE Phase 1 released in 2.7, right?
>
> Yes. We will release TDE in 2.7
>
> > Could you also list what needs to be done at Phase 2 and how much time
> it might take.
>
> Yes, I think Dmitry Ryabov will send Phase 2 design
>
>
> В Вт, 11/09/2018 в 23:27 +0300, Nikolay Izhikov пишет:
> > Hello, Denis.
> >
> > Yes, Vladimir made 2 rounds of review.
> > I planning to fix all issues with implementation in a few days.
> >
> >
> > В Вт, 11/09/2018 в 10:40 -0400, Denis Magda пишет:
> > > Hi Nikolay,
> > >
> > > Has anybody started looking into your request? As I understand the
> plan is
> > > to get TDE Phase 1 released in 2.7, right?
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89067473
> > >
> > > Could you also list what needs to be done at Phase 2 and how much time
> it
> > > might take.
> > >
> > > --
> > > Denis
> > >
> > > On Thu, Aug 9, 2018 at 8:48 AM Nikolay Izhikov 
> wrote:
> > >
> > > > Hello, Igniters.
> > > >
> > > > I want to share with you TDE implementation details.
> > > > I think it can simplify review and acception of TDE implementation.
> > > > This mail is copy of wiki page [1].
> > > >
> > > > Please, share your thoughts.
> > > >
> > > > TDE is ready for review [2].
> > > > Please, let me know, who is able to make final review.
> > > >
> > > > This document describes some internal details of TDE.Phase 1
> > > > implementation [3].
> > > > I suggest that reader familiar with the general design described in
> IEP-18
> > > > [4].
> > > >
> > > >
> > > > Cache group key management and node join enhancements:
> > > >
> > > > 1. GridEncryptionManager encapsulates all logic related to
> key
> > > > management:
> > > > a. All group encryption keys are stored in MetaStore.
> > > >
> > > > b. Joining node sends to cluster:
> > > > * Master key digest.
> > > > * All group keys saved locally (Map > > > byte[]>). Keys send over a network in encrypted form.
> > > >
> > > > c. Coordinator on new node join:
> > > > * Compares new node master key digest with
> the
> > > > local master key digest.
> > > > If differs then new node join is rejected.
> > > >
> > > > * Compares local and received group keys.
> > > > If some key differs then new node join is
> > > > rejected.
> > > >
> > > > d. All server nodes:
> > > > * If some of received keys are new then they
> save
> > > > locally.
> > > >
> > > > 2. Dynamic cache creation:
> > > > a. On server node - Encryption key is generated and
> added
> > > > to DynamicCacheChangeRequest.
> > > >
> > > > b. On client node:
> > > > * Prior to creation of
> DynamicCacheChangeRequest
> > > > we have to get new encryption key from some server node.
> > > > * New request added to solve this -
> > > > GenerateEncryptionKeyRequest.
> > > >
> > > >
> > > > WAL Record encryption:
> > > >
> > > >
> > > > 1. New type of WAL record created – EncryptedRecord.
> > > >
> > > > 2. EncryptedRecord is a container that stores some
> > > > WalRecordCacheGroupAware in encrypted form inside.
> > > >
> > > > 3. Write:
> > > > Each record for an encrypted group that implements
> > > > WalRecordCacheGroupAware written to WAL in encrypted form.
> > > > Instead of that record we write EncryptedRecord with plain
> record
> > > > inside(PageSnapshot, PageDeltaRecord, etc).
> > > >
> > > > 4. Read: There are 3 different cases on EncryptedRecord read:
> > > &g

Re: TDE Implementation details.

2018-09-11 Thread Denis Magda
Hi Nikolay,

Has anybody started looking into your request? As I understand the plan is
to get TDE Phase 1 released in 2.7, right?
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89067473

Could you also list what needs to be done at Phase 2 and how much time it
might take.

--
Denis

On Thu, Aug 9, 2018 at 8:48 AM Nikolay Izhikov  wrote:

> Hello, Igniters.
>
> I want to share with you TDE implementation details.
> I think it can simplify review and acception of TDE implementation.
> This mail is copy of wiki page [1].
>
> Please, share your thoughts.
>
> TDE is ready for review [2].
> Please, let me know, who is able to make final review.
>
> This document describes some internal details of TDE.Phase 1
> implementation [3].
> I suggest that reader familiar with the general design described in IEP-18
> [4].
>
>
> Cache group key management and node join enhancements:
>
> 1. GridEncryptionManager encapsulates all logic related to key
> management:
> a. All group encryption keys are stored in MetaStore.
>
> b. Joining node sends to cluster:
> * Master key digest.
> * All group keys saved locally (Map byte[]>). Keys send over a network in encrypted form.
>
> c. Coordinator on new node join:
> * Compares new node master key digest with the
> local master key digest.
> If differs then new node join is rejected.
>
> * Compares local and received group keys.
> If some key differs then new node join is
> rejected.
>
> d. All server nodes:
> * If some of received keys are new then they save
> locally.
>
> 2. Dynamic cache creation:
> a. On server node - Encryption key is generated and added
> to DynamicCacheChangeRequest.
>
> b. On client node:
> * Prior to creation of DynamicCacheChangeRequest
> we have to get new encryption key from some server node.
> * New request added to solve this -
> GenerateEncryptionKeyRequest.
>
>
> WAL Record encryption:
>
>
> 1. New type of WAL record created – EncryptedRecord.
>
> 2. EncryptedRecord is a container that stores some
> WalRecordCacheGroupAware in encrypted form inside.
>
> 3. Write:
> Each record for an encrypted group that implements
> WalRecordCacheGroupAware written to WAL in encrypted form.
> Instead of that record we write EncryptedRecord with plain record
> inside(PageSnapshot, PageDeltaRecord, etc).
>
> 4. Read: There are 3 different cases on EncryptedRecord read:
> a. WAL restore – we read EncryptedRecord, decrypt internal
> record and return it.
>
> b. Offline WAL iteration(StandaloneWalRecordsIterator) -
> EncryptionSpi is null so wecan’t decrypt EncryptedRecord and just return it
> from an iterator.
>
> c. Meta storage restore process – On node start we restore
> MetaStore.
> When we do it no encryption keys are available, because
> they are stored in MetaStore.
> So we can’t decrypt EncryptedRecord and just return it
> from an iterator.
> We don't need decrypted record on this step to operate
> properly.
>
>
> Page encryption:
>
>
> 1. We have to write on disk pages aligned on 2^n (2kb, 4kb, etc)
> for gain maximum perfrormance.
>
> 2. There is a 16 byte overhead for and AES CBC mode.
>
> 3. So we have to preserve 16 bytes in page in memory to get
> encrypted page size equal to 2^n when written it to disk.
>
> 4. PageIO has many methods with pageSize parameter.
> So for encrypted cache groups page size is calculated as
> cfg.getPageSize() - 16 byte.
>
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89067473
> [2] https://github.com/apache/ignite/pull/4167
> [3] https://issues.apache.org/jira/browse/IGNITE-8485
> [4]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-18%3A+Transparent+Data+Encryption
>


[jira] [Created] (IGNITE-9503) Visor CMD shows wrong REPLICATED cache max size

2018-09-09 Thread Denis Magda (JIRA)
Denis Magda created IGNITE-9503:
---

 Summary: Visor CMD shows wrong REPLICATED cache max size
 Key: IGNITE-9503
 URL: https://issues.apache.org/jira/browse/IGNITE-9503
 Project: Ignite
  Issue Type: Task
  Components: visor
Affects Versions: 2.5
Reporter: Denis Magda
Assignee: Alexey Kuznetsov
 Fix For: 2.6


Started 2 nodes with a single replicated cache. Preloaded _50_ entries 
there. 

Visor CMD *cache* command shows a wrong total
{code}
++
|Name(@)|Mode| Nodes | Entries (Heap / 
Off-heap) |   Hits|  Misses   |   Reads   |  Writes   |
++
| CacheDataStreamerExample(@c0) | REPLICATED | 2 | min: 246106 (0 / 246106) 
 | min: 0| min: 0| min: 0| min: 0|
|   ||   | avg: 25.00 (0.00 / 
25.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
|   ||   | max: 253894 (0 / 253894) 
 | max: 0| max: 0| max: 0| max: 0|
++
{code}

You find a correct total number only if *cache -a* is used and you calculate 
the sum of entries on each node manually:
{code}
+===+
|   Node ID8(@), IP   | CPUs | Heap Used | CPU Load |   Up Time|
 Size | Hi/Mi/Rd/Wr |
+===+
| 44A2CF9C(@n1), 192.168.1.64 | 4| 19.63 %   | 0.43 %   | 00:01:46.229 | 
Total: 253894| Hi: 0   |
| |  |   |  |  |   
Heap: 0| Mi: 0   |
| |  |   |  |  |   
Off-Heap: 253894   | Rd: 0   |
| |  |   |  |  |   
Off-Heap Memory: 0 | Wr: 0   |
+-+--+---+--+--+--+-+
| 72DEC7B5(@n0), 192.168.1.64 | 4| 9.69 %| 0.50 %   | 00:02:00.456 | 
Total: 246106| Hi: 0   |
| |  |   |  |  |   
Heap: 0| Mi: 0   |
| |  |   |  |  |   
Off-Heap: 246106   | Rd: 0   |
| |  |   |  |  |   
Off-Heap Memory: 0 | Wr: 0   |
+---+
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


Re: AI 2.7 Documentation Scope

2018-09-07 Thread Denis Magda
A full list of doc tickets can be retrieved with the following filter in
JIRA:

project = Ignite AND component = documentation AND fixVersion = 2.7 AND
status not in (Resolved, Closed)

--
Denis

On Thu, Sep 6, 2018 at 11:27 PM Denis Magda  wrote:

> Igniters,
>
> I've put together a page with the most prominent capabilities that have to
> be on our radar and must be documented *prior* to the release. Expand it
> if anything is missing:
> https://cwiki.apache.org/confluence/display/IGNITE/Required+Docs
>
> A full list of doc tickets can be retrieved with the following filter:
>
> Prachi and Artem Budnik will be able to assist as technical writers and
> editors.
>
> Presently "Doc Ticket" field is empty for almost every feature. Please
> locate a ticket assigned to you (Doc writer field) on the page and create a
> respective doc ticket:
>
>- Nikolay Izhikov
>- Vycheslav Daradur
>- Vladimir Ozerov
>- Igor Sapego
>- Artem Budnik
>- Prachi Garg
>- Yuri Babak
>- Amir Akhmedov
>
> --
> Denis
>


AI 2.7 Documentation Scope

2018-09-07 Thread Denis Magda
Igniters,

I've put together a page with the most prominent capabilities that have to
be on our radar and must be documented *prior* to the release. Expand it if
anything is missing:
https://cwiki.apache.org/confluence/display/IGNITE/Required+Docs

A full list of doc tickets can be retrieved with the following filter:

Prachi and Artem Budnik will be able to assist as technical writers and
editors.

Presently "Doc Ticket" field is empty for almost every feature. Please
locate a ticket assigned to you (Doc writer field) on the page and create a
respective doc ticket:

   - Nikolay Izhikov
   - Vycheslav Daradur
   - Vladimir Ozerov
   - Igor Sapego
   - Artem Budnik
   - Prachi Garg
   - Yuri Babak
   - Amir Akhmedov

--
Denis


Re: ScanQuery fails with OutOfMemory when iterating

2018-09-04 Thread Denis Magda
Hi Slava,

Thanks for looking into it. Looks like exactly what happened on the user
side.

--
Denis

On Tue, Sep 4, 2018 at 5:43 AM Вячеслав Коптилин 
wrote:

> Hi Denis,
>
> It looks like a known issue
> https://issues.apache.org/jira/browse/IGNITE-8892
> It is already fixed and will be available in Apache Ignite 2.7
>
> Thanks,
> S.
>
> вс, 2 сент. 2018 г. в 17:40, Denis Magda :
>
> > Igniters,
> >
> > A user reported the issue on SO:
> >
> >
> https://stackoverflow.com/questions/52117160/ignite-consumes-all-memory-and-fails-with-outofmemory-when-iterating-over-cache
> >
> > Does it sound familiar? Were we fixing something like that before?
> >
> > Please scan query and Ignite memory experts join the conversation there.
> >
> > --
> > Denis
> >
>


Re: IGNITE-640: multimap initial implementation

2018-09-04 Thread Denis Magda
Amir, Anton,

How is dev/review process going? Is there any chance we get this capability
into 2.7?

--
Denis

On Mon, Jul 9, 2018 at 10:27 PM Amir Akhmedov 
wrote:

> Hi Anton,
>
> I checked your last comments in the ticket and left some responses. Please
> check them and let me know
>
> Thanks,
> Amir
>
> P.S. do you mind to have a chat/call through gitter/Skype to discuss the
> details? Sometimes 5 minutes of chat can be more productive than long
> running email chains. Please, do not hesitate to directly email me if you
> mind to have a chat/call.
>
> On Wed, Jun 27, 2018 at 11:26 AM Anton Vinogradov  wrote:
>
> > Sure,
> > Hope it will be tomorrow
> >
> > ср, 27 июн. 2018 г. в 18:11, Amir Akhmedov :
> >
> > > Anton V,
> > > I put some comments into jira ticket. Can you please take a look once
> you
> > > have a chance?
> > >
> > > Thanks,
> > > Amir
> > >
> > > On Mon, Jun 18, 2018, 7:54 AM Anton Vinogradov  wrote:
> > >
> > > > Amir,
> > > >
> > > > Everything is fine, I'll check changes this week.
> > > >
> > > > вс, 17 июн. 2018 г. в 6:09, Amir Akhmedov :
> > > >
> > > > > Anton,
> > > > > I created a news PR [1]. Since it includes the same changes I did
> not
> > > run
> > > > > TC tests on it. Please let me know if you think otherwise.
> > > > >
> > > > > [1]  https://github.com/apache/ignite/pull/4207
> > > > >
> > > > > Thanks,
> > > > > Amir
> > > > >
> > > > >
> > > > > On Wed, Jun 13, 2018 at 8:38 AM Anton Vinogradov 
> > > wrote:
> > > > >
> > > > > > Amir,
> > > > > >
> > > > > > Thanks for attempt.
> > > > > > As far as I can see you have all changes at this commit:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/pull/3926/commits/cd0e50e05d3860788378ebf1a29dc0525460872a
> > > > > >
> > > > > > You can simply apply it to local branch based on master by patch
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/commit/cd0e50e05d3860788378ebf1a29dc0525460872a.patch
> > > > > >
> > > > > > In case you use IDEA, just apply patch from clipboard, and that's
> > > will
> > > > be
> > > > > > you PR.
> > > > > >
> > > > > > BTW, next time you can use easiest way to squash your changes -
> > just
> > > to
> > > > > > pull all changes from existing PR with squash
> > > > > > > git pull https://github.com/apache/ignite.git pull/XXX/head
> > > --squash
> > > > > >
> > > > > >
> > > > > >
> > > > > > вт, 5 июн. 2018 г. в 19:34, Amir Akhmedov <
> amir.akhme...@gmail.com
> > >:
> > > > > >
> > > > > > > Dmitry P., Anton V.,
> > > > > > > I made some changes and updated the ticket. Also as was asked I
> > > tried
> > > > > to
> > > > > > > squash the commits into one but looks like I screwed up
> > everything
> > > > and
> > > > > > the
> > > > > > > PR now looks completely terrible. Since I'm not an advanced git
> > > user,
> > > > > > could
> > > > > > > you please check the PR and let me know if anything could be
> done
> > > > > there?
> > > > > > If
> > > > > > > not I will try to create a new PR.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Amir
> > > > > > >
> > > > > > > On Tue, May 29, 2018 at 10:37 AM, Dmitry Pavlov <
> > > > dpavlov@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Amir,
> > > > > > > >
> > > > > > > > As far as I know, several Igniters provided some feedback in
> > > > ticket.
> > > > > > Are
> > > > > > > > you agree?
> > > > > > > >
> > > > > > > > Sincerely,
> > > > > > > > Dmitriy Pavlov
> > > > > > > >
> > > > > > > > чт, 10 мая 2018 г. в 20:01, Dmitry Pavlov <
> > dpavlov@gmail.com
> > > >:
> > > > > > > >
> > > > > > > > > Hi Amir,
> > > > > > > > >
> > > > > > > > > This is a very necessary contribution, the patch defenetely
> > > will
> > > > > not
> > > > > > be
> > > > > > > > > ignored.
> > > > > > > > >
> > > > > > > > > Igniters,
> > > > > > > > >
> > > > > > > > > Who can make a review from the committers?
> > > > > > > > >
> > > > > > > > > Sincerely,
> > > > > > > > > Dmitriy Pavlov
> > > > > > > > >
> > > > > > > > > вт, 8 мая 2018 г. в 5:52, Amir Akhmedov <
> > > amir.akhme...@gmail.com
> > > > >:
> > > > > > > > >
> > > > > > > > >> Hi Igniters,
> > > > > > > > >>
> > > > > > > > >> Can someone take a look at this PR please?
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> Amir
> > > > > > > > >>
> > > > > > > > >> On Mon, Apr 30, 2018 at 5:28 AM, Pavel Tupitsyn <
> > > > > > ptupit...@apache.org
> > > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi Amir,
> > > > > > > > >> >
> > > > > > > > >> > I have filed [1] for multimap in .NET, it will be done
> > > later.
> > > > > > > > >> > In order to fix IgniteParityTest failures, please add
> the
> > > > > > following
> > > > > > > to
> > > > > > > > >> > MissingMembers array there:
> > > > > > > > >> >
> > > > > > > > >> > "multimap" // IGNITE-8425
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> > Pavel
> > > > > > > > >> >
> > > > 

ScanQuery fails with OutOfMemory when iterating

2018-09-02 Thread Denis Magda
Igniters,

A user reported the issue on SO:
https://stackoverflow.com/questions/52117160/ignite-consumes-all-memory-and-fails-with-outofmemory-when-iterating-over-cache

Does it sound familiar? Were we fixing something like that before?

Please scan query and Ignite memory experts join the conversation there.

--
Denis


Re: Request for review : IGNITE-3303 Apache Flink Integration - Flink source

2018-08-30 Thread Denis Magda
Hello Saikat,

Hopefully, someone from the community will review the changes in the
nearest time.

--
Denis

On Thu, Aug 30, 2018 at 4:37 PM Saikat Maitra 
wrote:

> Hello,
>
> The changes for IGNITE-3303 for IgniteSource is complete. This will help is
> streaming data from Ignite cluster and process, filter, transform and
> publish it back to Ignite using IgniteSink or in any other data sink.
>
> I was hoping if the changes can be approved I can go ahead merge the
> changes.
>
>
> Regards,
> Saikat
>
>
>
> On Tue, Aug 28, 2018 at 12:56 AM, Saikat Maitra 
> wrote:
>
> > Hi Andrew,
> >
> > As discussed I have incorporated the changes. Please review and let me
> > know if any changes required.
> >
> > Regards,
> > Saikat
> >
> > On Mon, Aug 27, 2018 at 1:45 AM, Saikat Maitra 
> > wrote:
> >
> >> Hi,
> >>
> >> I have updated the PR with additional tests.
> >>
> >> Please review and share feedback.
> >>
> >> This PR is related to IgniteSink but allows to stream data from Ignite.
> >>
> >> PR https://github.com/apache/ignite/pull/870/files
> >>
> >> Review https://reviews.ignite.apache.org/ignite/review/IGNT-CR-135
> >>
> >> Regards,
> >> Saikat
> >>
> >
> >
>


Re: Hello Ignite Team, IGNITE-9438 contribution

2018-08-30 Thread Denis Magda
Welcome Sergey! Added you to JIRA.

--
Denis

On Thu, Aug 30, 2018 at 10:16 AM antonovsergey93 
wrote:

> Hello Team,
>
> I'd like to join to Apache Ignite development.
> Currently, I work on IGNITE-9438
> https://issues.apache.org/jira/browse/IGNITE-9438
>
> My Jira login is antonovsergey93
>
> BR, Sergey Antonov
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>


New committer: Dmitriy Govorukhin

2018-08-29 Thread Denis Magda
The Project Management Committee (PMC) for Apache Ignite
has invited Dmitriy Govorukhin to become a committer and we are pleased
to announce that he has accepted.

Being a committer enables easier contribution to the
project since there is no need to go via the patch
submission process. This should enable better productivity.

Congrats, Dmitriy! Look forward to your contributions.

--
Denis


New PMC member: Dmitriy Pavlov

2018-08-29 Thread Denis Magda
The Project Management Committee (PMC) for Apache Ignite
has invited Dmitriy Pavlov to become a PMC member and we are pleased
to announce that he has accepted.

Being a PMC member enables assistance with the management
and to guide the direction of the project.

Congratulations Dmitriy! Keep contributing to Ignite success the way you do
;)

Denis


Re: Retries in write-behind store

2018-08-28 Thread Denis Magda
Val,

Sounds like a handy configuration option. I would allow setting a number of
retries. If the number is set to 0 then a failed update is discarded right
away.

--
Denis

On Tue, Aug 28, 2018 at 9:14 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Folks,
>
> Is there a way to limit or disable retries of failed updates in the
> write-behind store? I can't find one, it looks like if an update fails, it
> is moved to the end of the queue and then eventually retried. If it fails
> again, process is repeated.
>
> Such behavior *might* be OK if failures are caused by database being
> temporarily unavailable. But what if update fails deterministically, for
> example due to a constraint violation? There is absolutely no reason to
> retry it, and at the same time it can cause stability and performance
> issues when buffer is full with such "broken" updates.
>
> Does it makes sense to add an option that would allow to limit number of
> retries (or even disable them)?
>
> -Val
>


Re: Apache Ignite 2.7 release

2018-08-28 Thread Denis Magda
Nikolay, Igniters, let me help you with the list.

That what I was tracking on my side (something we can announce). Don't have
a JIRA ticket for every ticket but CC-ed everyone who claimed to be in
charge. Nikolay, please work with the community members to add these
capabilities to the release wiki page. If something doesn't get delivered
then let's exclude it.

1. Partition map exchange optimizations. Are we releasing any of them? *(Sergey
Chugunov, Andrey Mashenkov)*
https://cwiki.apache.org/confluence/display/IGNITE/IEP-25%3A+Partition+Map+Exchange+hangs+resolving

2. Java 9/10/11 Support. Are we on track to support it better? *(Peter,
Vladimir)*.

3. SQL *(Vladimir)*:

   - Transactional SQL beta?
   - Basic monitoring facilities (inline index alerts, page reads/writes
   per type)?
   - SQL index update optimizations? (
   
https://cwiki.apache.org/confluence/display/IGNITE/IEP-19%3A+SQL+index+update+optimizations
   )
   - ODBC/JDBC session management
   - Result set offload to disk. Looks it doesn't get to the release?
   https://issues.apache.org/jira/browse/IGNITE-7526

4. JCache 1.1 support. Completed!

5. Transparent data encryption? What exactly goes in 2.7? *(Nikolay)*.

6. Ignite + Informatica integration

7. Ignite and Spring Session integration (heard it was done but the ticket
is still Open):
https://issues.apache.org/jira/browse/IGNITE-2741

8. Service Grid 2.0. What exactly goes in 2.7? (*Vyacheslav)*
https://cwiki.apache.org/confluence/display/IGNITE/IEP-17%3A+Oil+Change+in+Service+Grid

9. Ignite Multi Map *(Amir, Anton)*
https://issues.apache.org/jira/browse/IGNITE-640

10. Thin Clients:

   - Node.JS (https://issues.apache.org/jira/browse/IGNITE-) - *Pavel
   Petroshenko*
   - Python (https://issues.apache.org/jira/browse/IGNITE-7782) - *Pavel
   Petroshenko, **Dmitry Melnichuk*
   - PHP (https://issues.apache.org/jira/browse/IGNITE-7783) - *Pavel P.,
   Ekaterina*
   - C++: *Igor S.*
   - Affinity awareness for thin clients (*Igor S.*):
   
https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients

11. Machine and Deep Learning (preprocessing APIs, TensorFlow integration,
extra algorithms) - *Yuri and our ML experts*



On Tue, Aug 28, 2018 at 10:17 AM Dmitriy Setrakyan 
wrote:

> Hi Nikolai,
>
> Generally looks OK, however, It is hard to comment on your schedule without
> seeing a full list of all must-have features we plan to add to this
> release. I am hoping that the community will see this list at some point.
>
> D.
>
> On Tue, Aug 28, 2018 at 8:23 AM, Nikolay Izhikov 
> wrote:
>
> > Hello, Igniters.
> >
> > I think we should discuss the release schedule.
> >
> > Current dates are following:
> >
> > * Code Freeze: September 30, 2018
> > * Voting Date: October 1, 2018
> > * Release Date: October 15, 2018
> >
> > We discussed it privately with Vladimir Ozerov.
> >
> > Is seems better to reschedule a bit:
> >
> > * Scope freeze - September 17 - We should have a full list of
> > tickets for 2.7 here.
> > * Code freeze - October 01 - We should merge all 2.7 tickets to
> > master here.
> > * Vote - October 08.
> >
> > What do you think?
> >
> >
> > В Сб, 25/08/2018 в 00:57 +0300, Dmitriy Pavlov пишет:
> > > I hope Vyacheslav can comment better than me. I suppose it is, more or
> > > less, rectifications and clarifications of design aspects. Not overall
> > > redesign.
> > >
> > > I also hope Igniters, especially Services experts, will join the
> > discussion
> > > in the separate topic. Now after a couple of days there is no reaction.
> > >
> > > сб, 25 авг. 2018 г. в 0:53, Dmitriy Setrakyan :
> > >
> > > > On Fri, Aug 24, 2018 at 2:50 PM, Dmitriy Pavlov <
> dpavlov@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Dmitriy, I suppose it highly depends on how fast community will
> > come
> > > >
> > > > to
> > > > > a consensus about design. So it is up to us to make this happen
> > > > >
> > > >
> > > > I am confused then. If we are still discussing design, then we are
> > miles
> > > > away. Do you know if there anything in service grid that has already
> > been
> > > > implemented and can be released?
> > > >
> >
>


Re: Ignite new contributor

2018-08-28 Thread Denis Magda
Hello Ravil and welcome to the community!

I've added you to the contributor's list in JIRA. Please feel free to take
over the tasks.

--
Denis

On Tue, Aug 28, 2018 at 7:04 AM Ravil Galeyev  wrote:

> Hi Team,
>
> I'd like to join to Apache Ignite development.
> Especially to the ML part.
> Currently, I work on IGNITE-9285
> 
> and I'm going to continue work with subtasks from IGNITE-9281
>  (ML starter tasks).
>
> My Jira login is rgaleyev
>
> Best regards,
> Ravil
>


Re: Hello

2018-08-27 Thread Denis Magda
Hi Mikhail!

Done, welcome to the community!

--
Denis

On Mon, Aug 27, 2018 at 6:52 AM Mikhail Petrov 
wrote:

> I'm new to Ignite and I would like to join Apache Ignite development.
> My JIRA's login: PetrovMikhail
>


Re: Data streaming using Apache Ignite and Flink

2018-08-27 Thread Denis Magda
Hello Saikat,

Thanks for the article and contribution. We'll get it added to:
https://ignite.apache.org/blogs.html

--
Denis

On Sun, Aug 26, 2018 at 2:59 PM Saikat Maitra 
wrote:

> Hello,
>
> I recently published blog on how we can stream data using Apache Ignite and
> Flink. This uses IgniteSink with recent changes merged (release due in
> 2.7.0) which will allow us to run IgniteSink using Apache Flink in cluster
> mode.
>
>
>
> https://samaitra.blogspot.com/2018/08/data-streaming-using-apache-flink-and.html
>
> Please review and let me know if you have feedback.
>
> Regards,
> Saikat
>


  1   2   3   4   5   6   7   8   9   10   >