Re: Traditional RDBMS PARTITION concept on Ignite

[Denis Magda]

Hey folks, let me chime in to clarify the matters. What Wen Bo and Mengliao
are really asking for is a combination of two features "partitioning +
sharding".

In the relational world (Postgres, MySQL, Oracle) partitioning is used to
split a large table into smaller tables (called partitions). For example,
imagine you have table PizzaOrders and you want the database to keep
DELIVERED orders in one partition and all the others in a different one.
You can easily achieve this by partitioning the primary PizzaOrders table
by the order status into PizzaOrdersDelivered and PizzaOrdersOther. With
those partitions in place, the SQL engine can apply the partition pruning
optimization [1] and, as Megliao highlighted, it comes with data management
benefits [2].

This has nothing to do with the partitioning in Ignite. What Ignite does is
sharding - distributing table data across a cluster of nodes. Distributed
databases that support both partitioning and sharding do exist and usually,
they are built on Postgres or MySQL (and have nothing to do with in-memory
computing). For instance, YugabyteDB can partition your primary PizzaOrders
table first into PizzaOrdersDelivered and PizzaOrdersOther, and then have
those partitioned tables sharded automatically across the cluster.

As long as Ignite doesn't have the partitioning feature of relational
databases, you have these options:

   1. Use affinity keys in Ignite as you would use partition keys in
   Postgres/MySQL. But remember that all the data that matches an affinity key
   will be stored together on a single Ignite node. It might be a capacity
   problem if there are way too many records that belong to the affinity key.
   2. Implement the Postgres/MySQL-like partitioning at the application
   layer. Create Ignite tables for each logical partition, intercept user
   queries and, depending on the value of a partitioning column, place a
   record in one of the Ignite tables. Then Ignite will take care of the next
   step - sharding.


[1]
https://dmagda.hashnode.dev/optimizing-application-queries-with-partition-pruning
[2]
https://dmagda.hashnode.dev/managing-data-placement-with-table-partitioning

--
Denis

On Wed, Jul 20, 2022 at 6:35 AM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> Ignite’s SQL is ANSI 99 compliant. Windowing functions such as PARTITION
> BY came in SQL 2003 (and later). It’s possible that the new Calcite engine (
> sql-calcite )
> supports the keywords, but I have not checked.
>
>
>- While querying we can only scan a small portion of the data to
>improve performance
>
> As you suggested: indexes.
>
>
>- Quickly and safely manage data in one partition in particular. For
>example, in some RDBMS you can build index or compress data for only one
>partition, or delete one partition without locking other partitions being
>updated
>
> In general, traditional databases increase performance by grouping related
> stuff together. Ignite increases performance by distributing the data
> across multiple machines, which allows tasks to be parallelised. A
> different architecture results in different solutions.
>
>
>- Partitioning on multiple columns
>
> That’s an affinity key. But as I noted previously, you don’t want to do
> that if you only have three distinct values.
>
> Regards,
> Stephen
>
> On 18 Jul 2022, at 16:53, Mengliao(Mike) Wang 
> wrote:
>
> Hi Stephen,
>
> What we are looking for is the table partition with SQL in particular,
> instead of the data partition people mostly refer to in Ignite which is
> more from the infrastructure perspective. A.k.a the "PARTITION BY" keyword
> in traditional RDBMS. In the Ignite official document (
> https://ignite.apache.org/docs/latest/SQL/schemas) we didn't see anything
> like that, so not sure if there is anything in Ignite that could achieve
> these:
>
>- While querying we can only scan a small portion of the data to
>improve performance
>- Quickly and safely manage data in one partition in particular. For
>example, in some RDBMS you can build index or compress data for only one
>partition, or delete one partition without locking other partitions being
>updated
>- Partitioning on multiple columns
>
>
> Thanks
> Mike
>
> On Thu, Jul 14, 2022 at 9:05 AM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
>
>> As you say, partitions in Ignite are about the distribution of data. You
>> can group together related data using affinity keys, but if you only have
>> three distinct values that would be a really bad idea. You can’t change the
>> number of partitions after a table has been created.
>>
>> Either of your other solutions would work but, to be honest, I’m not
>> completely sure what problem you’re trying to solve.
>>
>> On 13 Jul 2022, at 19:24, Wen Bo (Bill) Li  wrote:
>>
>> Hi,
>>
>> The traditional RDBMS has the concept of partitioning a table into
>> different chunks, 

Re: [COMMUNITY] Apache Ignite Community Recap and Celebration of Project Evangelists

[Denis Magda]

Excellent 2021 recap! Thanks Kseniya for sharing, driving and helping with
so many activities. Can't wait to see the non-code contributions service to
track responses on the user lists and other forums.

--
Denis

On Wed, Dec 22, 2021 at 8:50 AM Kseniya Romanova 
wrote:

> Hi, Igniters!
>
> As 2021 comes to a close, let's review and celebrate the amazing
> community-driven contributions that made Apache Ignite better throughout
> the year:
>
> —  The community produced 2 releases 2.10 and 2.11 (2.12 is on its way)
> which introduced:
> ‒   Transparent Data Encryption (TDE) technology - the Cache Key Rotation
> machinery. From now on the Apache Ignite fully satisfies the PCI DSS[1],
> which requires the protection of stored data at rest;
> ‒   Significant improvements to Ignite thin clients: Added async API,
> Kubernetes discovery, extended transaction, cluster and SQL API for various
> platforms, added Continuous Query support, partition awareness, and more;
> ‒   Change Data Capture (CDC) support, which allows users to build their
> deployments on a new type of architecture - asynchronous data replication
> between clusters;
> ‒   IndexQuery API, which enables building fast index queries over
> build-in Ignite indexes. Unlike SQL, it works without the overhead for
> queries parsing and planning
>
> —  Ignite 3.0 Alphas were released [2][3]
> —  We welcomed 10 new committers and 1 PMC member
>
> Check and celebrate the most active code contributors here:
> ‒  Core and Documentation[4]
> ‒  Extensions[5]
> ‒  Ignite website[6]
>
> — Downloads for 2021 exceeded downloads for 2020 by 65%, with hundreds of
> thousands of downloads per month.
> — The Apache Software Foundation (ASF)  ranked Ignite second on the Top 5
> Big Data user lists, third on the Top 5 Big Data dev lists, second on the
> Top 5 user lists, and third on the Top 5 repos by size [7][8]
> —  In DB-Engines Ranking[9], Apache Ignite gained nine points (December
> 2021 versus December 2020).
> —  The Apache Ignite website was updated and improved [10].
> —  The community organized 13 Ignite meetups[11].
> —  Ignite Summit[12] was hosted first in May and then in November—with 33
> speakers, 8 pre-conference training, and 700 attendees.
> —  Fifty-eight developers contributed to Project Awareness by creating
> blogs, tutorials, and demos and by presenting at online and offline events
> worldwide.
> —  The Recognition service to track non-code Project Awareness
> contributions was launched[13]. I’ve recorded a video about the service for
> you[14]
>
> That’s what we as a community did in 2021. Impressive, isn’t it?
>
> Now, I’ll list 2021’s 10 most active non-code contributors, all of whom
> contributed to spreading the word about Ignite:
>
> Val Kulichenko
> Denis Magda
> Alexander Kozhenkov
> Stephen Darlington
> Pavel Tupitsyn
> Anton Vinogradov
> Courtney Robinson
> Maxim Muzafarov
> Nikolay Izhikov
> Semyon Danilov
>
> We are all proud of you! You did an outstanding job. My team and I want
> others to recognize your contributions! So, we created badges[15] that you
> can add to your LinkedIn profile (or to any other profile) to emphasize
> your role! Please check your emails.
>
> Have a Merry Christmas, Happy New Year, and stay awesome!
> Next year, we will rock again together!
>
> Cheers,
> Kseniya
> Apache Ignite PMC,
> Director of Community Relations at GridGain
>
> [1]
> https://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard
> [2]https://github.com/apache/ignite-3/
> [3]Ignite 3.0 Alpha 3 Community Gathering recording
> https://youtu.be/XanOleiBLlU
> [4]Ignite Core & Docs top-contributors 2021
> https://github.com/apache/ignite/graphs/contributors?from=2021-01-03=2021-12-22=c
> [5]Ignite Extensions top-contributors 2021
> https://github.com/apache/ignite-extensions/graphs/contributors?from=2021-01-01=2021-12-22=c
> [6]Ignite website top-contributors 2021
> https://github.com/apache/ignite-website/graphs/contributors?from=2021-01-01=2021-12-22=c
> [7] https://www.apache.org/foundation/docs/FY2021AnnualReport.pdf
> [8]
> https://blogs.apache.org/ignite/entry/apache-ignite-momentum-highlights-from
> [9] https://db-engines.com/en/ranking
> [10] https://ignite.apache.org/
> [11] https://ignite.apache.org/events.html#meetups
> [12] https://ignite-summit.org/
> [13] https://recognition.gridgain.com/main?date=this-year
> [14] https://youtu.be/AmO4dNdZHb8
> [15]
> https://www.credly.com/org/gridgain/badge/ignite-project-awareness-top-contributor-2021
>
>

Re: Real-world use cases: Ignite and High-Performance Computing

[Denis Magda]

Piper, have a look at the Yardstick framework that we use for the
general-purpose benchmarking tasks:
https://ignite.apache.org/docs/latest/perf-and-troubleshooting/yardstick-benchmarking

If you have anything specific in mind, then start a separate discussion on
the user list.

-
Denis


On Thu, Dec 2, 2021 at 10:07 PM Piper H  wrote:

> To compare the speed, is there a perftest tool for benchmarking the
> performance?
> Like this one for Rabbitmq:
>
> https://rabbitmq.github.io/rabbitmq-perf-test/stable/htmlsingle/#installation
>
> Thank you.
> Piper
>
> On Thu, Dec 2, 2021 at 11:52 PM Denis Magda  wrote:
>
>> Folks,
>>
>> I've just published an article that lists some real-world use cases of
>> Ignite for high-performance computing. Some of you might be interested in
>> how the compute APIs are leveraged in practice:
>>
>> https://www.gridgain.com/resources/blog/how-apache-ignite-empowers-high-performance-computing-real-use-cases
>>
>> That's a short summary of recordings from past conferences and meetup
>> talks. Thanks to everyone who shared their stories!
>>
>> -
>> Denis
>>
>

Real-world use cases: Ignite and High-Performance Computing

[Denis Magda]

Folks,

I've just published an article that lists some real-world use cases of
Ignite for high-performance computing. Some of you might be interested in
how the compute APIs are leveraged in practice:
https://www.gridgain.com/resources/blog/how-apache-ignite-empowers-high-performance-computing-real-use-cases

That's a short summary of recordings from past conferences and meetup
talks. Thanks to everyone who shared their stories!

-
Denis

Re: Re client node connects to server nodes behind NAT

[Denis Magda]

Hi Ma,

Streaming is supported for the thin clients (see a related discussion on
the user list). As for the peer-class-loading support, not sure that it's
coming. As an alternative, you can implement the UriDeploymentSpi interface
that will load/reload code from your custom location/repository:
https://ignite.apache.org/docs/latest/code-deployment/deploying-user-code

If you don't want to bother with the UriDeploymentSpi implementation, then
check this Deployment feature of GridGain Nebula (it will be available for
non-Nebula clusters through the same Control Center interface soon). That
feature implements the UriDeploymentSpi:
https://www.youtube.com/watch?v=8hS1ux1d3Ck=PLMc7NR20hA-J5tnwx-ltnBB9X-gF8cP9S=2

Finally, as Max suggested, configure the thick client and server nodes
following this documentation:
https://ignite.apache.org/docs/latest/clustering/running-client-nodes-behind-nat
(it should solve your networking problems).

-
Denis


On Tue, Nov 16, 2021 at 2:53 AM Maksim Timonin 
wrote:

> Hi!
>
> Unfortunately, support of peer class loading and data streams is limited
> for thin clients. AFAIK, .Net thin client supports data streamer.
>
> I found that Ignite has a special mode of running client nodes
> `forceClientToServerConnections` [1]. But this feature is marked as
> experimental since 2.9 and I don't see any updates there. Actually, I'm not
> aware about this feature much. I found the feature discussion on dev list
> [2]. Maybe somebody from the community can say more about this feature. But
> looks like it can be a solution in your case.
>
> [1]
> https://ignite.apache.org/docs/latest/clustering/running-client-nodes-behind-nat
> [2] https://www.mail-archive.com/dev@ignite.apache.org/msg44990.html
>
> On Mon, Nov 15, 2021 at 12:15 PM MJ <6733...@qq.com> wrote:
>
>> That would be peer class loading && data stream .
>>
>> Thanks
>>
>> -- 原始邮件 --
>> *发件人:* "user" ;
>> *发送时间:* 2021年11月15日(星期一) 下午3:59
>> *收件人:* "user";
>> *主题:* Re: client node connects to server nodes behind NAT
>>
>> Hi, I'll touch on a similar topic a little tomorrow on IgniteSummit [1],
>> you're welcome :)
>>
>> But, a thick client is part of topology, and it has to be able to connect
>> every node in the cluster directly, then thick clients have to be run
>> within the same kubernetes cluster. Thin clients were designed to eliminate
>> this problem.
>>
>> What features of thick clients do you want to use?
>>
>> [1] https://ignite-summit.org/sessions/293596
>>
>> On Mon, Nov 15, 2021 at 3:50 AM MJ <6733...@qq.com> wrote:
>>
>>> Hi,
>>>
>>> Is that possible for a non-Kubernetes client node connects to server
>>> nodes within Kubernetes ?
>>>
>>> have read below docs seems impossible
>>>
>>> https://ignite.apache.org/docs/latest/installation/kubernetes/azure-deployment#connecting-client-nodes
>>>
>>> Have tried with thin client outside of Kubernetes - that works fine
>>> client node(thick client) - always throw exceptions, most likely the
>>> internal ips bebind NAT cannot be detected from external , is there any
>>> workaround to implement that non-Kubernetes client node connects to server
>>> nodes within Kubernetes ? I'd like to utilise the power features of thick
>>> client. and They can be depoloyed everywhere if there is the way of making
>>> it.
>>>
>>>
>>>
>>> Thanks,
>>> Ma Jun
>>>
>>

Re: protocol of streaming

[Denis Magda]

Nikita, Igor, looping you in...

Do we folks have tickets for this documentation update?

-
Denis


On Tue, Nov 16, 2021 at 6:11 AM Pavel Tupitsyn  wrote:

> Hello Timothy,
>
> Thin client protocol supports streaming [1],
> the documentation just was not updated yet.
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-68%3A+Thin+Client+Data+Streamer
>
> On Tue, Nov 16, 2021 at 2:06 PM Timothy Peng  wrote:
>
>> Hello
>>
>> I looked through this documentation,
>>
>> https://ignite.apache.org/docs/latest/binary-client-protocol/binary-client-protocol
>>
>> It seems that the binary client protocol doesn't have streaming
>> supported. Am I missing something?
>>
>> Thanks
>>
>

Re: test

[Denis Magda]

Test Passed

-
Denis


On Wed, Nov 10, 2021 at 7:12 AM Veena Mithare 
wrote:

>
> --
>
>
> Spread bets and CFDs are complex instruments and come with a high risk of
> losing money rapidly due to leverage. *66% of retail investor accounts
> lose money when spread betting and/or trading CFDs with this provider*.
> You should consider whether you understand how spread bets and CFDs work
> and whether you can afford to take the high risk of losing your money.
>
> Professional clients: Losses can exceed deposits when spread betting and
> trading CFDs. Countdowns carry a level of risk to your capital as you could
> lose all of your investment. These products may not be suitable for all
> clients therefore ensure you understand the risks and seek independent
> advice. Invest only what you can afford to lose.
>
> CMC Markets UK plc (173730) and CMC Spreadbet plc (170627) are authorised
> and regulated by the Financial Conduct Authority in the United Kingdom. CMC
> Markets UK plc and CMC Spreadbet plc are registered in England and Wales
> with Company Numbers 02448409 and 02589529 and with their registered
> offices at 133 Houndsditch, London, EC3A 7BX.
>
> The content of this e-mail (including any attachments) is strictly
> confidential and is for the sole use of the intended addressee(s). If you
> are not the intended recipient of this e-mail please notify the sender
> immediately and delete this e-mail from your system. Any disclosure,
> copying, dissemination or use of its content (including any attachments) is
> strictly prohibited. CMC Markets UK plc and CMC Spreadbet plc reserve the
> right to intercept and monitor the content of the e-mail messages to and
> from its systems.
>
> E-mails may be interfered with or may contain viruses or other defects for
> which CMC Markets UK plc and CMC Spreadbet plc accept no responsibility. It
> is the responsibility of the recipient to carry out a virus check on the
> e-mail and any attachment(s).
>
> This communication is not intended as an offer or solicitation for the
> purchase or sale of a financial instrument or as an official confirmation
> of any transaction unless specifically presented as such.
>

Re: [Ignite Summit: Cloud Edition] The full schedule is published

[Denis Magda]

Btw, check out this 3 minute-long video produced by Kevin that does a great
overview of the upcoming event: https://www.youtube.com/watch?v=5Oas92pBSPM

-
Denis


On Wed, Oct 20, 2021 at 2:11 PM Kseniya Romanova 
wrote:

> Hi Igniters!
>
> The Ignite Summit: Cloud Edition schedule is live on ignite-summit.org!
> It will be a two-day event for architects and developers who use or are
> interested in using Apache Ignite in the cloud.
>
> On November 15, 2021, free training sessions for EMEA and the Americas
> will be offered:
>
> ‒ Apache Ignite and Kubernetes: Deployment and Orchestration Strategies
> ‒ Apache Ignite Essentials: Key Design Principles for Building
> Data-Intensive Applications
> ‒ Apache Ignite for Spring Boot and Spring Data Development
> ‒ How to Monitor and Manage Apache Ignite with GridGain Control Center
>
> More details and registration: https://ignite-summit.org/training
>
> On November 16, 2021, Ignite experts from IBM, Hewlett Packard Enterprise
> and GridGain will share architectural insights and practical tips for
> solving challenges in cloud environments:
>
> Keynotes
>
> ‒ In-Memory Computing on the Wave of Cloud Transformation – Nikita Ivanov,
> Apache Ignite PMC, Co-founder and CTO, GridGain
>
>
> ‒ Why and How to Put Apache Ignite in the Cloud and the 5 Lessons We
> Learned Doing It! – Shahir Daya, IBM Distinguished Engineer and Chief
> Architect
>
> Sessions
>
> ‒ Building Cloud-Native 5G Networks Functions with Apache Ignite – Alain
> Guigui, 5G Shared Data Environment Lead Architect, Communication & Media
> Solutions, Hewlett Packard Enterprise
>
> ‒ Deploying Apache Ignite in the Cloud: A Checklist – Alexander Shapkin,
> Apache Ignite Committer, GridGain Engineer
>
> ‒ The Pragmatic Deployment Strategy for Multi-Cloud Architecture – Stan
> Lukyanov, GridGain Technical Director
>
> ‒ Deploying Apache Ignite to the Cloud in Minutes: Lessons Learned from
> GridGain Nebula – Greg Stachnick, GridGain Director of Cloud Product
> Management
>
> ‒ Stateful Microservices with Spring Cloud and Apache Ignite – Alexander
> Kozhenkov, GridGain Team Lead
>
>
> Schedule and more about presentations and speakers:
> https://ignite-summit.org/schedule
>
> Join us and invite your fellow developers to attend!
>
> The Summit team will be grateful for any help with the promotion. Please
> join  https://www.linkedin.com/events/6851552481613250560/, make a repost
> in Twitter https://twitter.com/ApacheIgnite/status/1450794060697018374 or
> https://www.linkedin.com/feed/update/urn:li:activity:6856559750939856896
>
>
> Cheers,
> Kseniya
>
>

Ignite Momentum: Recap of 2020-2021

[Denis Magda]

Igniters,

I've contributed an article listing what we, as a community, achieved
together in 2020-2021:
https://blogs.apache.org/ignite/entry/apache-ignite-momentum-highlights-from

Apologize if I forgot to mention anything else. My goal was to share the
project highlights after ASF mentioned us as a top 5 project in the recent
FY2021 report.

Please share through social media.

-
Denis

Fwd: Celebrating ASF's success: FY2021 Annual Report

[Denis Magda]

Igniters,

I've looked into a full report that is available under this address:
https://www.apache.org/foundation/docs/FY2021AnnualReport.pdf

And compiled Ignite-specific achievements:
https://docs.google.com/document/d/1xZ2cfH_YiuNrf8-YSIsupGU3eU9qovhrfdF0etPhp3c/edit

Ignite keeps growing and remains as a top-5 project of the ASF in various
categories! Congrats to all of us!

--
Denis

-- Forwarded message -
From: Sally Khudairi 
Date: Tue, Aug 31, 2021 at 10:37 AM
Subject: Celebrating ASF's success: FY2021 Annual Report
To: 
Cc: ASF Marketing & Publicity 


Hello ASF Committers --I hope you are all well.

We have just published the ASF's Annual Report for FY2021 (1 May 2020 - 30
April 2021).

The press release is at GlobeNewswire at
https://www.globenewswire.com/news-release/2021/08/31/2289277/17401/en/The-Apache-Software-Foundation-Announces-Annual-Report-for-2021-Fiscal-Year.html
as well as the ASF "Foundation" blog
https://s.apache.org/FY2021AnnualReport-pressrelease

The Annual Report is at https://s.apache.org/FY2021AnnualReport

Should you wish to spread the word on social media, feel free to like and
share our tweet at https://twitter.com/TheASF/status/1432690473274052612
and post on LinkedIn
https://www.linkedin.com/feed/update/urn:li:activity:6838456339170172928/

Our achievements are the result of our collective efforts. All of us. Thank
you for your ongoing dedication and continued support that keeps us moving
forward.

As always, if you have any questions or need anything, feel free to reach
out to us (ASF Marketing & Publicity) at pr...@apache.org

Here's to continued success!

Best,
Sally

- - -
Vice President Sponsor Relations
The Apache Software Foundation

Tel +1 617 921 8656 | s...@apache.org

-
Denis

Re: Ignite High memory usage though very less disk size

[Denis Magda]

To easily collect what Zhenya is asking for and expedite the
troubleshooting, you can do a simple monitoring set up:
https://www.gridgain.com/docs/tutorials/management-monitoring/ignite-storage-monitoring


-
Denis

On Fri, Jul 30, 2021 at 2:32 PM Zhenya Stanilovsky 
wrote:

> hi, Devakumar J
> There is not enough information for analysis.
> Do you have any monitoring ? If no — plz enable it and try to understand
> how huge cpu consumption and possibly gc pauses correlates with you tasks.
> Do you have enough heap (-Xmx param) ? What kind of processes are consume
> most heap ?
> Without all these info we can`t move forward in analysis.
>
> thanks !
>
>
> Hi,
>
> We have 3server+2client cluster setup. Also we have 2 completely different
> clusters for different regions.
>
> Both has similar set of integrations in terms of SQL queries/ CQ
> listeners/ Client connections.
>
> Also the VM hardware/OS settings also same.
>
> In cluster 1 through we have disk of 20GB but the cluster performance is
> really good and heap usage/CPU usage is optimal.
>
> In cluster 2 we do have less data only in disk but there is heavy
> fluctuations in heap usage and lot FULL GC happening pausing JVM for 7 to 8
> secs every minute. Only restart helps in this case.
>
>
> Only difference noticed between machines is memory page cache utilization.
> We have done page cache cleanup and restarted the cluster and page cache
> utilization become 105 GB out of 126GB RAM with in a day.
>
> Please find the metrics below and suggest any debugging steps to carry
> out/document to refer.
>
>
> Cluster 1:
>
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=27529ecd, name=server-node-3, uptime=2 days, 17:48:09.803]
> ^-- H/N/C [hosts=3, nodes=5, CPUs=24]
> ^-- CPU [cur=1.33%, avg=3.05%, GC=0%]
> ^-- PageMemory [pages=3375870]
> ^-- Heap [used=4372MB, free=73.31%, comm=5600MB]
> ^-- Off-heap [used=13341MB, free=20.03%, comm=16584MB]
> ^--   sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
> ^--   metastoreMemPlc region [used=0MB, free=99.85%, comm=0MB]
> ^--   TxLog region [used=0MB, free=100%, comm=100MB]
> ^--   DefaultRegion region [used=13341MB, free=18.57%, comm=16384MB]
> ^-- Ignite persistence [used=20052MB]
> ^--   sysMemPlc region [used=0MB]
> ^--   metastoreMemPlc region [used=0MB]
> ^--   TxLog region [used=0MB]
> ^--   DefaultRegion region [used=20052MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=0, idle=7, qSize=0]
>
> Cluster 2:
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=5905afb7, name=server-node-1, uptime=2 days, 05:49:04.925]
> ^-- H/N/C [hosts=3, nodes=5, CPUs=24]
> ^-- CPU [cur=1.23%, avg=6.4%, GC=0%]
> ^-- PageMemory [pages=1173731]
> ^-- Heap [used=13043MB, free=20.39%, comm=16384MB]
> ^-- Off-heap [used=4638MB, free=72.2%, comm=16584MB]
> ^--   sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
> ^--   metastoreMemPlc region [used=0MB, free=99.91%, comm=0MB]
> ^--   TxLog region [used=0MB, free=100%, comm=100MB]
> ^--   DefaultRegion region [used=4638MB, free=71.69%, comm=16384MB]
> ^-- Ignite persistence [used=5423MB]
> ^--   sysMemPlc region [used=0MB]
> ^--   metastoreMemPlc region [used=0MB]
> ^--   TxLog region [used=0MB]
> ^--   DefaultRegion region [used=5422MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=0, idle=5, qSize=0]
>
>
> Thanks & Regards,
> Devakumar J
>
>
> 
>  Virus-free.
> www.avast.com
> 
>
>
>
>
>
>

Re: Defrag?

[Denis Magda]

Ignite fellows,

I remember some of us worked on the persistence defragmentation features.
Has it been merged?

@Valentin Kulichenko  probably you know the
latest state.

-
Denis

On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> You can probably drop the entire cache and then re-populate it via
> loadCache(), etc.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 23 июн. 2021 г. в 21:47, Ryan Trollip :
>
>> Thanks, Ilya, we may have to consider moving back to non-native storage
>> and caching more selectively as the performance degrades when there is a
>> lot of write/delete activity or tables with large amounts of rows. This is
>> with SQL with indexes and the use of query plans etc.
>>
>> Is there any easy way to rebuild the entire native database after hours?
>> e.g. with a batch run on the weeknds?
>>
>> On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> I don't think there's anything ready to use, but "killing performance"
>>> from fragmentation is also not something reported too often.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> ср, 16 июн. 2021 г. в 04:39, Ryan Trollip :
>>>
 We see continual very large growth to data with ignite native. We have
 a very chatty use case that's creating and deleting stuff often. The data
 on disk just keeps growing at an explosive rate. So much so we ported this
 to a DB to see the difference and the DB is much smaller. I was searching
 to see if someone has the same issue. This is also killing performance.

 Founds this:

 https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation

 Apparently, there is no auto-rebalancing of pages? or cleanup of pages?

 Has anyone implemented a workaround to rebuild the cache and indexes
 say on a weekly basis to get it to behave reasonably?

 Thanks

>>>

Re: [ANNOUNCE] Apache Ignite 2.10.0 Released

[Denis Magda]

Excellent, thanks Maxim for driving the release!

As for the product definition (in-memory platform vs. distributed
database), just copy the positioning we use on the landing page
(Distributed Database For High-Performance Computing With In-Memory Speed)

-
Denis


On Thu, Mar 18, 2021 at 6:31 AM Maxim Muzafarov  wrote:

> The Apache Ignite Community is pleased to announce the release of
> Apache Ignite 2.10.0.
>
> Apache Ignite® is an in-memory computing platform for transactional,
> analytical, and streaming workloads delivering in-memory speeds at
> a petabyte scale.
> https://ignite.apache.org
>
> The Apache Ignite community has made a lot of changes in the 2.10.0
> release, hopefully, this blog post will help you to know about some
> valuable improvements:
> https://blogs.apache.org/ignite/entry/apache-ignite-2-10-thin
>
> For the full list of changes, you can refer to the RELEASE_NOTES list
> which is trying to catalogue the most significant improvements for
> this version of the platform.
> https://ignite.apache.org/releases/2.10.0/release_notes.html
>
> Download the latest Ignite version from here:
> https://ignite.apache.org/download.cgi
>
> Please let us know if you encounter any problems:
> https://ignite.apache.org/community/resources.html#ask
>
>
> Regards,
> Maxim Muzafarov on behalf of the Apache Ignite community.
>

Re: very fast loading of very big table

[Denis Magda]

Hello Vladimir,

Good to hear from you! How much is that in gigabytes?

-
Denis


On Thu, Feb 18, 2021 at 10:06 PM  wrote:

> Sep 2020 I've published the paper about Loading Large Datasets into Apache
> Ignite by Using a Key-Value API (English [1] and Russian [2] version). The
> approach described works in production, but shows inacceptable perfomance
> for very large tables.
>
> The story continues, and yesterday I've finished the proof of concept for
> very fast loading of very big table. The partitioned MSSQL table about 295
> million rows was loaded by the 4-node Ignite cluster in 3 min 35 sec. Each
> node had executed its own SQL queries in parallel and then distributed the
> loaded values across the other cluster nodes.
>
> Probably that result will be of interest for the community.
>
> Regards,
> Vladimir Chernyi
>
> [1]
> https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api
> [2] https://m.habr.com/ru/post/526708/
>
>

Instructor-Led Ignite Essentials Training

[Denis Magda]

Folks,

We're starting regular training sessions covering Ignite Essentials. See if
it's valuable for you. The first session is on Feb 10th and the more to
follow:
https://www.gridgain.com/resources/webinars/developer-training-apache-ignite-essentials-key-design-principles-building-data

This free, two-hour training is for Java developers and architects who
build high-performance and data-intensive applications that are powered by
Apache Ignite. During the course, you are introduced to three of Ignite's
essential capabilities (data partitioning, affinity co-location, and
co-located processing) and learn how to apply your newly acquired knowledge
to increase the speed and scale of your applications.

-
Denis

Re: Control Center core limit info

[Denis Magda]

Hello Carsten,

To my knowledge, you can have up to 2 nodes monitored by Control Center for
free. Then you need to request a license file.

As a temporary workaround, you can use the hosted version of the tool that
doesn't put any limitations:
https://control.gridgain.com

-
Denis


On Fri, Jan 8, 2021 at 7:09 AM Carsten 
wrote:

> Hello all,
>
> after an install marathon yesterday night, I was ready to test Ignite
> 2.9.1 + GridGainControl Center 2020.12.
>
> But when starting everything (4 nodes + control center) I got the
> message "Core limit has been exceeded for the current license".
>
> I was just wondering what the core limit is and were to find information
> on it (i did search but came up empty)
>
> Any advice is greatly appreciated
>
>
> All the best,
>
> Carsten
>
>

Re: Ignite in-memory + other SQL store without fully loading all data into Ignite

[Denis Magda]

Hi Courtney,

Glad to hear from you! It's been a while since we met last time. It's truly
disappointing seeing you struggle with Ignite that much. Thanks for being
open and kicking the discussion off.

How about three of us (you, Val, and I) meet the second week of January and
talk out the issues? Then you can share a talk summary here with a broader
community, if you wish. Between us, I've been personally championing the
column-type-change feature for a while, and with this IEP of Ignite 3.0

it should be doable.

In the meantime, some items for our agenda and pointers:

   - How does your upgrade procedure look like? Ignite doesn't have a
   rolling-upgrade feature; thus, to upgrade from version A to B in a
   *consistent way* the one should stop the cluster.
   - re: the frustration about spending too much time on the administration
   and infrastructure-related activities. Are you considering any managed
   service options? As a developer, I understand this frustration. I was just
   lucky to stay away from a need to administer Postgres, Oracle, MySQL, or
   Ignite just because we either had an IT administrator or made use of a
   managed-service option. Not selling anything here, just know that all those
   administration routines are unavoidable. The thing with Ignite is that
   distributed nature makes things more complicated.
   - Ignite ML and SQL: in fact, you can already call Ignite ML models from
   a SQL query. I need to search for pointers. @Alexey Zinoviev
   , you can probably share some examples quicker.
   - Calcite and Ignite: in Q2 we're planning to release the
   Calcite-powered SQL engine in Ignite. That would be a dramatic improvement
   in our SQL capabilities. It should be possible to enable push-downs much
   more easily.


-
Denis


On Mon, Dec 28, 2020 at 11:50 PM Courtney Robinson <
courtney.robin...@hypi.io> wrote:

> Hi Val,
> Thanks. You're not missing anything and we have been using Ignite
> persistence for years.
> Among the reasons we want to switch to in-memory only is how easily we
> seem to get corrupt nodes. I mentioned in the first email. We haven't had a
> situation where upgrading corrupts more than 1 of our 3 replicas but we
> genuinely fear upgrading sometimes as a result and is why we put so much
> effort into our backup/restore solution.
>
> One of the arguments for disabling persistence is that other DBs seems to
> have had longer to solve the issue. We've operated Postgres for longer than
> Ignite persistence and have never had it corrupt its data on disk
> (admittedly it's not distributed so that could play a role) but we've also
> been running Cassandra for longer as well without ever having any data
> corruption. All 3 in Kubernetes. With Ignite we've tried many things and a
> few settings between Ignite rebalanceDelay (I forget which other ones) and
> k8s readiness/liveness probes seems to have landed in a sweet spot that's
> reduced how often it happens but if we happen to have any issues with the
> k8s control plane, scheduling delays or network issues then the chances of
> it skyrockets. It then requires manual intervention I believe the most
> frequent issue is that a node will start and the topology diverged so the
> cluster doesn't think it's a part of it, the most effective thing we've
> found is deleting the data on that node and having it join as a new node
> for re-balancing to kicking and push data back to it. I think we have a few
> tickets somewhere with details.
>
> This is the primary motivator for wanting to replace Ignite persistence.
>
>  Here's our use case:
> We provide a low-code app development platform. As a developer you define
> tables in a visual interface or via GraphQL. We map these on to Ignite
> tables. It is a multi-tenant system complete with
> authentication/authorisation, serverless functions, API Gateway and
> more...all the bells and whistles for building modern applications in
> record time. We make it easy and convenient to get the scale and
> performance you expect from a large company with as little as the effort of
> an intern.
>
> The thing about convenience is that there are expectations set by others
> (the industry in this case) that the convenient things you provide include
> a baseline that matches industry or whoever has set the expectations.
> It is a sizeable task to implement/enforce referential integrity and
> delete cascade application side and I don't know of any distributed
> database which has this built in but the lack of transactions in SQL means
> we can't really implement it without pretty much implementing what the DB
> should be doing to keep track and rollback on failure.
>
> In an earlier version of the platform we implemented this using the key
> value APIs but we were not using SQL at all. We moved to SQL 1.5yrs ago or
> so because we needed greater and greater query capability and we were
> implementing a 

Re: Implementing HA among Ignite Clusters

[Denis Magda]

This DR feature is what you are looking for, complete solution:
https://www.gridgain.com/docs/latest/administrators-guide/data-center-replication/introduction

On Sunday, December 6, 2020, Alexandr Shapkin  wrote:

> Hi,
>
>
>
> I don’t think that ignite support this out of the box, cause data
> replication is something beyond the default distribution. The usage depends
> on how your data is replicated, whether you want active-active
> synchronization, and so on.
>
>
>
> But if it’s about switching from one cluster to another, and always using
> a single cluster at a time, something like dynamic DNS
>
> or a custom load balancer should work here. I.e. you can configure your
> consumers to access domain *my.cluster.com  *that
> resolves to the primary cluster, IP=192.168.1.1. When you want to switch
> the load, you can adjust your DNS to resolve the name
>
> to, say, IP=192.168.1.13 which means the secondary one. This approach has
> some drawbacks but is quite simple to use and straightforward.
>
>
>
> *From: *Naveen 
> *Sent: *Sunday, December 6, 2020 2:47 PM
> *To: *user@ignite.apache.org
> *Subject: *Implementing HA among Ignite Clusters
>
>
>
> Hi All
>
>
>
> We have 2 clusters (Primary and secondary)  running and both are always in
>
> sync from data perspective.
>
> Always consumers are using Primary only, but when primary is not reachable,
>
> it should get redirected to secondary cluster How can this be achieved ?
>
>
>
> 1. Can we do anything on the cluster side which will take care of this,
> like
>
> with config changes on the primary cluster like changing the connecton
>
> string etc.
>
> Like some of the messaging products uses connection URLs like
>
> tcp://server1||tco://server2
>
> something like this, if server1 is not reachable, it will get routed to
>
> server2.
>
> Somehing like this can be done here ?
>
> 2. DOing this from consumer side only, as and when they get exceptions like
>
> disconnect etc, try connecting it to secondary and vice versa
>
>
>
> What are the options we have for this scenario
>
>
>
> Thanks
>
> Naveen
>
>
>
>
>
>
>
> --
>
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>


-- 
-
Denis

Re: IgniteDataStreamer.keepBinary proposal

[Denis Magda]

Hi Vladimir,

Most of the code snippets are already arranged in complete and
ready-for-usage samples:
https://github.com/apache/ignite/tree/master/docs/_docs/code-snippets/java/src/main/java/org/apache/ignite/snippets

Anyway, those are code snippets that are injected into quite generic
documentation pages. Your case represents a situation when someone needs to
work with binary objects and streaming APIs. What if we add a data streamer
example for BinaryObjects into Ignite's examples and put a reference to
that example from the documentation page? Are you interested in
contributing the example?
https://github.com/apache/ignite/tree/master/examples

-
Denis


On Fri, Dec 4, 2020 at 2:58 AM Vladimir Tchernyi 
wrote:

> Hi, community
>
>
>
> I've just finished drilling a small page [1] about Ignite data streaming
> and I want to share my impressions. The situation is common for many Ignite
> documentation pages, impressions are the same.
>
>
>
> My problem was to adapt IgniteDataStreamer to data loading using the
> binary format as described in my article [2]. I try to use the same
> approach:
>
> 1) load data on the client node;
>
> 2) convert it to the binary form;
>
> 3) use IgniteDataStreamer/StreamReceiver pair (instead of
> ComputeTaskAdapter/ComputeJobAdapter) to ingest data in the cache.
>
>
>
> I modified my production code using IgniteDataStreamer BinaryObject> and  StreamReceiver, tried to
> start on the dev cluster made of 2 server nodes and 1 client node. That is
> it: ClassNotFoundException for the class that exists on the client node
> only.
>
>
>
> The solution to the problem seems to be in setting
> streamer.keepBinary(true), but page [1] never says about it. I found that
> setter in the IgniteDataStreamer source code after a single day of
> troubleshooting. Definitely, "In Ignite We Trust" - what else reason would
> drive me to spend so much time?
>
>
>
> The code snippets on the page [1] are hard to implement in real-world
> applications because of using only primitive types String, Integer, etc.
> These are more like unit tests.
>
>
>
> My proposal - it would be great to create a small GitHub repo containing a
> complete compilable code example, one repo for every page. I think such
> repos will keep the newbie Ignite users inside the project and prevent them
> from leaving.
>
>
>
> Regards,
>
> Vladimir Tchernyi
>
> --
>
> [1] https://ignite.apache.org/docs/latest/data-streaming
>
> [2]
> https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api
>
>
>

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[Denis Magda]

Yes, once the data is loaded into Ignite you can query it with SQL.

-
Denis


On Wed, Dec 2, 2020 at 9:41 AM adumalagan  wrote:

> Hi Vlad, I've started to read through your link and I just wanted to know
> that, if I were to implement your tutorial, could I perform SQL queries on
> the cache after loading?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Native persistence - No space left on device error

[Denis Magda]

Keep an eye on the storage usage metrics to avoid such incidents
proactively in production:
https://www.gridgain.com/docs/tutorials/management-monitoring/ignite-storage-monitoring#track-disk-storage-size

-
Denis


On Wed, Nov 18, 2020 at 7:41 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> When you run out of disk space, the Ignite node will stop functioning
> normally, and your underlying PDS storage may become corrupted as well.
>
> Thus it is recommended to avoid running out of disk space. This is not in
> any means a normal mode of operation of Apache Ignite.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 18 нояб. 2020 г. в 18:29, facundo.maldonado <
> maldonadofacu...@gmail.com>:
>
>> What is the expected behavior of native persistence when there is no more
>> space on disk?
>>
>> I'm getting this error when I reach the max size of the mounted volume
>> (storage, not wal):
>> *class org.apache.ignite.IgniteCheckedException: No space left on device*
>>
>> I assumed, that when there is no more space for allocating pages, it
>> removes
>> the older ones. I may be wrong, confusing this with expiry.
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: Is there any change to integrate ignite into Android environment?

[Denis Magda]

We are not running Ignite test suites on Android, thus, don't have a full
picture of what works or fails. Please file a ticket with the stack trace
and details about your Android environment.

But, you have the REST protocol at your service that will work on any edge
device. Use Android networking APIs to make calls to your Ignite cluster
via REST: https://ignite.apache.org/docs/latest/restapi


-
Denis


On Wed, Nov 11, 2020 at 6:58 PM xingjl6280  wrote:

> Hi IIya,
>
> yes, I tried, with the java thin client.
>
> There's no issue with build, but get fatal error during client initiation.
> Ignite try to call some virtual method of Object by reflection, but the
> method doesn't exist.
> I guess it's due to the difference of Android SDK and JDK.
>
> Actually, in many manufacturing scenarios, Apps are installed on Pad with
> good performance and LAN network,
> Ignite will provide those Apps with more convenient communication with
> backend, or even IOT device.
>
> So the initial design of Ignite didn't take Android into consideration,
> right? And there's no another version for Android?
> I may give up this idea.
>
> Thank you
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Is there any change to integrate ignite into Android environment?

[Denis Magda]

Hi,

What's the device? Are you planning to start a cluster on those devices? Or
connect from those devices to a cluster deployed on server machines?

-
Denis


On Mon, Nov 9, 2020 at 7:46 PM xingjl6280  wrote:

> Hi team,
>
> Is it possible to integrate ignite into Android?
> My Android device will only work in LAN environment, I hope it could be
> facilitated with the memory sharing and service call function of ignite.
>
> thank you
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: WAL and WAL Archive volume size recommendation

[Denis Magda]

Hello Facundo,

Just go ahead and disable the WAL archives. You need the archives for the
point-in-time-recovery feature that is supported by GridGain. I'll check
with the community why we have the archives enabled by default in a
separate discussion.
https://ignite.apache.org/docs/latest/persistence/native-persistence#disabling-wal-archive

-
Denis


On Thu, Nov 5, 2020 at 11:37 AM facundo.maldonado <
maldonadofacu...@gmail.com> wrote:

> Well, I found some useful numbers between two pages in the documentation.
>
> "By default, there are 10 active segments."  wal ref
> <
> https://ignite.apache.org/docs/latest/persistence/native-persistence#write-ahead-log>
>
>
> "The number of segments kept in the archive is such that the total size of
> all segments does not exceed the specified size of the WAL archive.
> By default, the maximum size of the WAL archive (total space it occupies on
> disk) is defined as 4 times the size of the checkpointing buffer."
> wal-archive ref
> <
> https://ignite.apache.org/docs/latest/persistence/native-persistence#wal-archive>
>
>
> "The default buffer size is calculated as a function of the data region
> size:
>
> Data Region Size   Default Checkpointing Buffer Size
> < 1 GB MIN (256 MB, Data_Region_Size)
> between 1 GB and 8 GB Data_Region_Size / 4
> > 8 GB 2 GB"   checkpoint buffer size
> > <
> https://ignite.apache.org/docs/latest/persistence/persistence-tuning#adjusting-checkpointing-buffer-size>
>
>
> So, if i have:
> data region max size: 5Gb
> storage vol size: 10Gi
> I can set:
> WAL vol size: 1Gb  # WAL size is 10 * wal segment 64Mb
> WAL archive vol size: 5Gi
> # 4 times checkpoint size
> # region < 8Gb, checkpoint size is region/4 --> wal archive size is equals
> to region size
> # region > 8Gb, checkpoint is 2 Gb --> wal archive is at least 4*2Gb == 8GB
>
> With those settings, I can keep the test running some more time but the pod
> keeps crashing.
> At least, it seems that I'm not getting the same error as before.
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: New Node - Rebalancing

[Denis Magda]

Hi Mahesh,

Use these metrics to monitor the progress:

   - JMX:
   
https://ignite.apache.org/docs/latest/monitoring-metrics/metrics#monitoring-rebalancing
   - Rebalancing widget of Control Center:
   
https://www.gridgain.com/docs/control-center/latest/monitoring/configuring-widgets#rebalance-widget


-
Denis


On Tue, Nov 3, 2020 at 11:14 AM Mahesh Renduchintala <
mahesh.renduchint...@aline-consulting.com> wrote:

> Hi,
>
> As soon as we add a new server node into the cluster, rebalancing starts
>  this is clear.
> is there a way to know when the rebalancing successfully ends on the new
> server node?
> Caches in the cluster are both replicated and partitioned.
>
> regards
> Mahesh
>
>
>

Re: Execution of local SqlFieldsQuery on client node disallowed

[Denis Magda]

Hi Narges,

Then just send a task to a required node. If the cluster topology changes
while the task was running you can re-submit it to ensure the result is
accurate.

-
Denis


On Fri, Oct 30, 2020 at 2:16 PM narges saleh  wrote:

> Hi Denis,
>
> My problem with using affinity call/run is that I have to have the key in
> order to run it. I just want to run a function on the data on the current
> node, without knowing the key. Is there anyway to do this and also
> guard against partition rebalancing?
>
> thanks
>
> On Tue, Oct 27, 2020 at 10:31 AM narges saleh 
> wrote:
>
>> Thanks Ilya, Denis for the feedback.
>>
>> On Mon, Oct 26, 2020 at 1:44 PM Denis Magda  wrote:
>>
>>> Narges,
>>>
>>> Also, keep in mind that if a local query is executed over a partitioned
>>> table and it happens that partitions rebalancing starts, the local query
>>> might return a wrong result (if partitions the query was executed over were
>>> rebalanced to another node during the query execution time). To address
>>> this:
>>>
>>>1. Execute the local query inside of an affinityCall/Run function (
>>>
>>> https://ignite.apache.org/docs/latest/distributed-computing/collocated-computations#colocating-by-key).
>>>Those functions don't let partitions be evicted until the function
>>>execution completes.
>>>2. Don't use the local queries, let the Ignite SQL engine to run
>>>standard queries, and to take care of possible optimizations.
>>>
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Mon, Oct 26, 2020 at 8:50 AM Ilya Kasnacheev <
>>> ilya.kasnach...@gmail.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> You are using an Ignite Thick Client driver. As its name implies, it
>>>> will start a local client node and then connect to it, without the option
>>>> of doing local queries.
>>>>
>>>> You need to use Ignite Thin JDBC driver: jdbc:ignite:thin://
>>>> Then you can do local queries.
>>>>
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>>
>>>>
>>>> сб, 24 окт. 2020 г. в 16:04, narges saleh :
>>>>
>>>>> Hello Ilya
>>>>> Yes, it happens all the time. It seems ignite forces the "client"
>>>>> establishing the jdbc connection into a client mode, even if I set
>>>>> client=false.  The sample code and config are attached. The question is 
>>>>> how
>>>>> do I force JDBC connections from a server node.
>>>>> thanks.
>>>>>
>>>>> On Fri, Oct 23, 2020 at 10:31 AM Ilya Kasnacheev <
>>>>> ilya.kasnach...@gmail.com> wrote:
>>>>>
>>>>>> Hello!
>>>>>>
>>>>>> Does this happen every time? If so, do you have a reproducer for the
>>>>>> issue?
>>>>>>
>>>>>> Regards,
>>>>>> --
>>>>>> Ilya Kasnacheev
>>>>>>
>>>>>>
>>>>>> пт, 23 окт. 2020 г. в 13:06, narges saleh :
>>>>>>
>>>>>>> Denis -- Just checked. I do specify my services to be deployed on
>>>>>>> server nodes only. Why would ignite think that I am running my code on a
>>>>>>> client node?
>>>>>>>
>>>>>>> On Fri, Oct 23, 2020 at 3:50 AM narges saleh 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Denis
>>>>>>>> What would make an ignite node a client node? The code is invoked
>>>>>>>> via an ignite service deployed on each node and I am not setting the 
>>>>>>>> client
>>>>>>>> mode anywhere. The code sets the jdbc connection to local and tries to
>>>>>>>> execute a sql code on the node in some interval. By the way, I didn't 
>>>>>>>> know
>>>>>>>> one could deploy a service on client nodes. Do I need to explicitly 
>>>>>>>> mark a
>>>>>>>> node as a server node when deploying a service?
>>>>>>>> thanks
>>>>>>>>
>>>>>>>> On Thu, Oct 22, 2020 at 9:42 PM Denis Magda 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> The error message says you're attempting to run the query on a
>>>>>>>>> client node. If that's the case (if the service is deployed on the 
>>>>>>>>> client
>>>>>>>>> node), then the local flag has no effect because client nodes don't 
>>>>>>>>> keep
>>>>>>>>> your data locally but rather consume it from servers.
>>>>>>>>>
>>>>>>>>> -
>>>>>>>>> Denis
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Oct 22, 2020 at 6:26 PM narges saleh 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi All,
>>>>>>>>>> I am trying to execute a sql query via a JDBC  connection on the
>>>>>>>>>> service node (the query is run via a service), but I am getting 
>>>>>>>>>> *Execution
>>>>>>>>>> of local SqlFieldsQuery on client node disallowed.*
>>>>>>>>>> *The JDBC connection has the option local=true as I want to run
>>>>>>>>>> the query on the data on the local node only.*
>>>>>>>>>> *Any idea why I am getting this error?*
>>>>>>>>>>
>>>>>>>>>> *thanks.*
>>>>>>>>>>
>>>>>>>>>

Re: Inserting date into ignite with spark jdbc

[Denis Magda]

Andrey,

Do we need to update our docs? It feels like the docs miss these details or
have an outdated example.

-
Denis


On Fri, Oct 30, 2020 at 7:03 AM Andrei Aleksandrov 
wrote:

> Hi,
>
> Here's an example with correct syntax that should work fine:
>
>  DataFrameWriter < Row > df = resultDF
>   .write()
>   .format(IgniteDataFrameSettings.FORMAT_IGNITE())
>   .option(IgniteDataFrameSettings.OPTION_CONFIG_FILE(), configPath)
>   .option(IgniteDataFrameSettings.OPTION_TABLE(), "Person")
>   .option(IgniteDataFrameSettings.OPTION_CREATE_TABLE_PRIMARY_KEY_FIELDS(), 
> "id, city_id")
>   .option(IgniteDataFrameSettings.OPTION_CREATE_TABLE_PARAMETERS(), 
> "template=partitioned,backups=1")
>   .mode(Append);
>
> Please let me know if something is wrong here.
>
> BR,
> Andrei
> 10/30/2020 2:20 AM, Humphrey пишет:
>
> Hello guys this question has been asked on  Stack 
> Overflow
>  
> 
> but yet no answer is a provided.
>
> I'm facing the same issue (trying to insert data in ignite using
> spark.jdbc):
> Exception in thread "main" java.sql.SQLException: No PRIMARY KEY defined for
> CREATE TABLE
>   at
> org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:1004)
>
> Code:
> println("-- writing using jdbc --")
> val prop = Properties()
> prop["driver"] = "org.apache.ignite.IgniteJdbcThinDriver"
>
> df.write().apply {
> mode(SaveMode.Overwrite)
> format("jdbc")
> option("url", "jdbc:ignite:thin://127.0.0.1")
> option("dbtable", "comments")
>
> option(IgniteDataFrameSettings.OPTION_CREATE_TABLE_PRIMARY_KEY_FIELDS(),
> "last_name")
> }.save()
>
> The last option doesn't seem to work/help.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>

Re: Execution of local SqlFieldsQuery on client node disallowed

[Denis Magda]

Narges,

Also, keep in mind that if a local query is executed over a partitioned
table and it happens that partitions rebalancing starts, the local query
might return a wrong result (if partitions the query was executed over were
rebalanced to another node during the query execution time). To address
this:

   1. Execute the local query inside of an affinityCall/Run function (
   
https://ignite.apache.org/docs/latest/distributed-computing/collocated-computations#colocating-by-key).
   Those functions don't let partitions be evicted until the function
   execution completes.
   2. Don't use the local queries, let the Ignite SQL engine to run
   standard queries, and to take care of possible optimizations.


-
Denis


On Mon, Oct 26, 2020 at 8:50 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> You are using an Ignite Thick Client driver. As its name implies, it will
> start a local client node and then connect to it, without the option of
> doing local queries.
>
> You need to use Ignite Thin JDBC driver: jdbc:ignite:thin://
> Then you can do local queries.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> сб, 24 окт. 2020 г. в 16:04, narges saleh :
>
>> Hello Ilya
>> Yes, it happens all the time. It seems ignite forces the "client"
>> establishing the jdbc connection into a client mode, even if I set
>> client=false.  The sample code and config are attached. The question is how
>> do I force JDBC connections from a server node.
>> thanks.
>>
>> On Fri, Oct 23, 2020 at 10:31 AM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> Does this happen every time? If so, do you have a reproducer for the
>>> issue?
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пт, 23 окт. 2020 г. в 13:06, narges saleh :
>>>
>>>> Denis -- Just checked. I do specify my services to be deployed on
>>>> server nodes only. Why would ignite think that I am running my code on a
>>>> client node?
>>>>
>>>> On Fri, Oct 23, 2020 at 3:50 AM narges saleh 
>>>> wrote:
>>>>
>>>>> Hi Denis
>>>>> What would make an ignite node a client node? The code is invoked via
>>>>> an ignite service deployed on each node and I am not setting the client
>>>>> mode anywhere. The code sets the jdbc connection to local and tries to
>>>>> execute a sql code on the node in some interval. By the way, I didn't know
>>>>> one could deploy a service on client nodes. Do I need to explicitly mark a
>>>>> node as a server node when deploying a service?
>>>>> thanks
>>>>>
>>>>> On Thu, Oct 22, 2020 at 9:42 PM Denis Magda  wrote:
>>>>>
>>>>>> The error message says you're attempting to run the query on a client
>>>>>> node. If that's the case (if the service is deployed on the client node),
>>>>>> then the local flag has no effect because client nodes don't keep your 
>>>>>> data
>>>>>> locally but rather consume it from servers.
>>>>>>
>>>>>> -
>>>>>> Denis
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 22, 2020 at 6:26 PM narges saleh 
>>>>>> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>> I am trying to execute a sql query via a JDBC  connection on the
>>>>>>> service node (the query is run via a service), but I am getting 
>>>>>>> *Execution
>>>>>>> of local SqlFieldsQuery on client node disallowed.*
>>>>>>> *The JDBC connection has the option local=true as I want to run the
>>>>>>> query on the data on the local node only.*
>>>>>>> *Any idea why I am getting this error?*
>>>>>>>
>>>>>>> *thanks.*
>>>>>>>
>>>>>>

Re: [ANNOUNCE] Apache Ignite 2.9.0 Released

[Denis Magda]

We made it! Congrats, community! Thanks, Alex for pushing the release till
the end, that was a long road ;)

I decided to record a quick video featuring top-3 changes in the release.
Those are my top-3 additions while you can have yours. Anyway, check it
out:
https://youtu.be/ewwjq8SIk-s

-
Denis


On Fri, Oct 23, 2020 at 7:38 AM Aleksey Plekhanov  wrote:

> The Apache Ignite Community is pleased to announce the release of
> Apache Ignite 2.9.0.
>
> Apache Ignite® is an in-memory computing platform for transactional,
> analytical, and streaming workloads delivering in-memory speeds at
> petabyte scale.
> https://ignite.apache.org
>
> For the full list of changes, you can refer to the RELEASE_NOTES list
> which is trying to catalogue the most significant improvements for
> this version of the platform.
> https://ignite.apache.org/releases/2.9.0/release_notes.html
>
> Download the latest Ignite version from here:
> https://ignite.apache.org/download.cgi
>
> Please let us know if you encounter any problems:
> https://ignite.apache.org/community/resources.html#ask
>
>
> Regards,
> Aleksey Plekhanov on behalf of Apache Ignite community.
>

Re: Execution of local SqlFieldsQuery on client node disallowed

[Denis Magda]

The error message says you're attempting to run the query on a client node.
If that's the case (if the service is deployed on the client node), then
the local flag has no effect because client nodes don't keep your data
locally but rather consume it from servers.

-
Denis


On Thu, Oct 22, 2020 at 6:26 PM narges saleh  wrote:

> Hi All,
> I am trying to execute a sql query via a JDBC  connection on the service
> node (the query is run via a service), but I am getting *Execution of
> local SqlFieldsQuery on client node disallowed.*
> *The JDBC connection has the option local=true as I want to run the query
> on the data on the local node only.*
> *Any idea why I am getting this error?*
>
> *thanks.*
>

Re: Isolating server nodes to a fixed virtual IP or interface

[Denis Magda]

Hi Gilles,

Thanks for considering Ignite for your project! Let's help to get you to
production avoiding bumpy roads )

Try to set the 'localAddress' parameter (and 'localPortRanges' if needed)
for both the discovery and communication settings:

   - Discovery:
   
https://ignite.apache.org/docs/latest/clustering/network-configuration#discovery
   - Communication:
   
https://ignite.apache.org/docs/latest/clustering/network-configuration#communication


-
Denis


On Thu, Oct 22, 2020 at 10:54 AM Gilles 
wrote:

> Hello,
>
> I'm currently moving a project from development stage to production. The
> aim is that my cluster server nodes are running on multiple virtual private
> servers, inside a VPN (10.0.0.0/24).
>
> But how do I make sure that I lock any communication of a node to either a
> specific network interface, or a static virtual IP (eg 10.0.0.3)?
>
> Some googling got me to this answer from old documentation.
>
> 
>  class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
>   
> 
>  
>
> However the nodes are still accessible on their public IP addresses. So
> the question is, what is the correct way to isolate them from the public?
>
> I will be using a software firewall on these servers too, but I like to
> have the peace of mind from the extra layer of security.
>
>
> Thanks in advance,
> Gilles
>
> And to the creators, maintainers and contributors, thank you so much for
> this great piece of software! Never had so much fun doing "cumbersome"
> database work.
>
>
>
>

Video overview of eight Ignite-specific talks scheduled for the IMCS Summit 2020

[Denis Magda]

Igniters,

We've got eight (!!!) Ignite-specific sessions scheduled for the In-Memory
Computing Summit 2020. I've gone ahead and produced a quick overview of
those sessions. Check and weigh up which one to join:
https://youtu.be/SCqd4qfBY6Q

The sessions are delivered by our community folks: @Saikat Maitra
, @Alexey Zinoviev ,
@Stanislav
Lukyanov , @Andrey Alexandrov
, @Glenn Wiebe , @Greg
Stachnick , me

Support the speakers and share the video within your network.

-
Denis

Tech Talk, October 14th, Pacific Time: ways to prevent out-of-memory issues in Ignite

[Denis Magda]

Igniters,

I've been with the community for more than 5 years and would say that
out-of-memory issues are among the most notorious ones, as well as
unexpected ones, that are reported by fellow Ignite developers.  Even if we
do a great job estimating a required cluster capacity, the data volume can
burst in 3,5,10, etc. years, and the capacity of today won't be enough to
meet the needs of tomorrow. The “throw more resources into the cluster”
approach is an often-heard piece of advice. However, in practice, most of
us cannot scale out a cluster in a moment.

So, I collected some material that summarizes out-of-memory preventive
measures. Join tomorrow  online or register and watch later on-demand:
https://www.gridgain.com/resources/webinars/apache-ignite-workshop-cluster-memory-management-techniques

-
Denis

Re: Continuous Query

[Denis Magda]

I recalled a complete solution. That's what you would need to do if decide
to process records in real-time with continuous queries in the
*fault-tolerant fashion* (using pseudo-code rather than actual Ignite APIs).

First. You need to add a flag field to the record's class that keeps the
current processing status. Like that:

MyRecord {
int id;
Date created;
byte status; //0 - not processed, 1 - being processed withing a
continuous query filter, 2 - processed by the filter, all the logic
successfully completed
}

Second. The continuous query filter (that will be executed on nodes that
store a copy of a record) needs to have the following structure.

@IgniteAsyncCallback
filterMethod(MyRecords updatedRecord) {

if (isThisNodePrimaryNodeForTheRecord(updatedRecord)) { // execute on a
primary node only
updatedRecord.status = 1 // setting the flag to signify the processing
is started.

//execute your business logic

updatedRecord.status = 2 // finished processing
}
  return false; //you don't want a notification to be sent to the client
application or another node that deployed the continuous query
}

Third. If any node leaves the cluster or the whole cluster is restarted,
then you need to execute your custom for all the records with status=0 or
status=1. To do that you can broadcast a compute task:

// Application side

int[] unprocessedRecords = "select id from MyRecord where status < 2;"

IgniteCompute.affinityRun(idsOfUnprocessedRecords, taskWithMyCustomLogic);
//the task will be executed only on the nodes that store the records

// Server side

taskWithMyCustomLogic() {
updatedRecord.status = 1 // setting the flag to signify the processing
is started.

//execute your business logic

updatedRecord.status = 2 // finished processing
}


That's it. So, the third step already requires you to have a compute task
that would run the calculation in case of failures. Thus, if the real-time
aspect of the processing is not crucial right now, then you can start with
the batch-based approach by running a compute task once at a time and then
introduce the continuous queries-based improvement whenever is needed. You
decide. Hope it helps.


-
Denis


On Wed, Oct 7, 2020 at 9:19 AM Denis Magda  wrote:

> So, a lesson for the future, would the continuous query approach still be
>> preferable if the calculation involves the cache with continuous query and
>> say a look up table? For example, if I want to see the country in the cache
>> employee exists in the list of the countries that I am interested in.
>
>
> You can access other caches from within the filter but the logic has to be
> executed asynchronously to avoid deadlocks:
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/lang/IgniteAsyncCallback.html
>
> Also, what do I need to do if I want the filter for the continuous query
>> to execute on the cache on the local node only? Say, I have the continuous
>> query deployed as singleton service on each node, to capture certain
>> changes to a cache on the local node.
>
>
> <https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/lang/IgniteAsyncCallback.html>
>
> The filter will be deployed and executed on every server node. The filter
> is executed only on a server node that owns a record that is being modified
> and passed into a filter. Hmm, it's also said that the filter can be
> executed on a backup node. Check if it's true, and then you need to add a
> special check into the filter that would allow executing the logic only if
> it's the primary node:
>
> https://ignite.apache.org/docs/latest/key-value-api/continuous-queries#remote-filter
>
>
> -
> Denis
>
>
> On Wed, Oct 7, 2020 at 4:39 AM narges saleh  wrote:
>
>> Also, what do I need to do if I want the filter for the continuous query
>> to execute on the cache on the local node only? Say, I have the continuous
>> query deployed as singleton service on each node, to capture certain
>> changes to a cache on the local node.
>>
>> On Wed, Oct 7, 2020 at 5:54 AM narges saleh  wrote:
>>
>>> Thank you, Denis.
>>> So, a lesson for the future, would the continuous query approach still
>>> be preferable if the calculation involves the cache with continuous query
>>> and say a look up table? For example, if I want to see the country in the
>>> cache employee exists in the list of the countries that I am interested in.
>>>
>>> On Tue, Oct 6, 2020 at 4:11 PM Denis Magda  wrote:
>>>
>>>> Thanks
>>>>
>>>> Then, I would consider the continuous queries based solution as long as
>>>> the records can be updated in real-time:
>>>>
>>>>- You can process the records on the fly and don't need to come up

Re: Continuous Query

[Denis Magda]

>
> So, a lesson for the future, would the continuous query approach still be
> preferable if the calculation involves the cache with continuous query and
> say a look up table? For example, if I want to see the country in the cache
> employee exists in the list of the countries that I am interested in.


You can access other caches from within the filter but the logic has to be
executed asynchronously to avoid deadlocks:
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/lang/IgniteAsyncCallback.html

Also, what do I need to do if I want the filter for the continuous query to
> execute on the cache on the local node only? Say, I have the continuous
> query deployed as singleton service on each node, to capture certain
> changes to a cache on the local node.

<https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/lang/IgniteAsyncCallback.html>

The filter will be deployed and executed on every server node. The filter
is executed only on a server node that owns a record that is being modified
and passed into a filter. Hmm, it's also said that the filter can be
executed on a backup node. Check if it's true, and then you need to add a
special check into the filter that would allow executing the logic only if
it's the primary node:
https://ignite.apache.org/docs/latest/key-value-api/continuous-queries#remote-filter


-
Denis


On Wed, Oct 7, 2020 at 4:39 AM narges saleh  wrote:

> Also, what do I need to do if I want the filter for the continuous query
> to execute on the cache on the local node only? Say, I have the continuous
> query deployed as singleton service on each node, to capture certain
> changes to a cache on the local node.
>
> On Wed, Oct 7, 2020 at 5:54 AM narges saleh  wrote:
>
>> Thank you, Denis.
>> So, a lesson for the future, would the continuous query approach still be
>> preferable if the calculation involves the cache with continuous query and
>> say a look up table? For example, if I want to see the country in the cache
>> employee exists in the list of the countries that I am interested in.
>>
>> On Tue, Oct 6, 2020 at 4:11 PM Denis Magda  wrote:
>>
>>> Thanks
>>>
>>> Then, I would consider the continuous queries based solution as long as
>>> the records can be updated in real-time:
>>>
>>>- You can process the records on the fly and don't need to come up
>>>with any batch task.
>>>- The continuous query filter will be executed once on a node that
>>>stores the record's primary copy. If the primary node fails in the middle
>>>of the filter's calculation execution, then the filter will be executed 
>>> on
>>>a backup node. So, you will not lose any updates but might need to
>>>introduce some logic/flag that confirms the calculation is not executed
>>>twice for a single record (this can happen if the primary node failed in
>>>the middle of the calculation execution and then the backup node picked 
>>> up
>>>and started executing the calculation from scratch).
>>>- Updates of other tables or records from within the continuous
>>>query filter must go through an async thread pool. You need to use
>>>IgniteAsyncCallback annotation for that:
>>>
>>> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/lang/IgniteAsyncCallback.html
>>>
>>> Alternatively, you can always run the calculation in the batch-fashion:
>>>
>>>- Run a compute task once in a while
>>>- Read all the latest records that satisfy the requests with SQL or
>>>any other APIs
>>>- Complete the calculation, mark already processed records just in
>>>case if everything is failed in the middle and you need to run the
>>>calculation from scratch
>>>
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Mon, Oct 5, 2020 at 8:33 PM narges saleh 
>>> wrote:
>>>
>>>> Denis
>>>>  The calculation itself doesn't involve an update or read of another
>>>> record, but based on the outcome of the calculation, the process might make
>>>> changes in some other tables.
>>>>
>>>> thanks.
>>>>
>>>> On Mon, Oct 5, 2020 at 7:04 PM Denis Magda  wrote:
>>>>
>>>>> Good. Another clarification:
>>>>>
>>>>>- Does that calculation change the state of the record (updates
>>>>>any fields)?
>>>>>- Does the calculation read or update any other records?
>>>>>
>>>>> -
>>>>&g

Re: Continuous Query

[Denis Magda]

Thanks

Then, I would consider the continuous queries based solution as long as the
records can be updated in real-time:

   - You can process the records on the fly and don't need to come up with
   any batch task.
   - The continuous query filter will be executed once on a node that
   stores the record's primary copy. If the primary node fails in the middle
   of the filter's calculation execution, then the filter will be executed on
   a backup node. So, you will not lose any updates but might need to
   introduce some logic/flag that confirms the calculation is not executed
   twice for a single record (this can happen if the primary node failed in
   the middle of the calculation execution and then the backup node picked up
   and started executing the calculation from scratch).
   - Updates of other tables or records from within the continuous query
   filter must go through an async thread pool. You need to use
   IgniteAsyncCallback annotation for that:
   
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/lang/IgniteAsyncCallback.html

Alternatively, you can always run the calculation in the batch-fashion:

   - Run a compute task once in a while
   - Read all the latest records that satisfy the requests with SQL or any
   other APIs
   - Complete the calculation, mark already processed records just in case
   if everything is failed in the middle and you need to run the calculation
   from scratch


-
Denis


On Mon, Oct 5, 2020 at 8:33 PM narges saleh  wrote:

> Denis
>  The calculation itself doesn't involve an update or read of another
> record, but based on the outcome of the calculation, the process might make
> changes in some other tables.
>
> thanks.
>
> On Mon, Oct 5, 2020 at 7:04 PM Denis Magda  wrote:
>
>> Good. Another clarification:
>>
>>- Does that calculation change the state of the record (updates any
>>fields)?
>>- Does the calculation read or update any other records?
>>
>> -
>> Denis
>>
>>
>> On Sat, Oct 3, 2020 at 1:34 PM narges saleh  wrote:
>>
>>> The latter; the server needs to perform some calculations on the data
>>> without sending any notification to the app.
>>>
>>> On Fri, Oct 2, 2020 at 4:25 PM Denis Magda  wrote:
>>>
>>>> And after you detect a record that satisfies the condition, do you need
>>>> to send any notification to the application? Or is it more like a server
>>>> detects and does some calculation logically without updating the app.
>>>>
>>>> -
>>>> Denis
>>>>
>>>>
>>>> On Fri, Oct 2, 2020 at 11:22 AM narges saleh 
>>>> wrote:
>>>>
>>>>> The detection should happen at most a couple of minutes after a record
>>>>> is inserted in the cache but all the detections are local to the node. But
>>>>> some records with the current timestamp might show up in the system with
>>>>> big delays.
>>>>>
>>>>> On Fri, Oct 2, 2020 at 12:23 PM Denis Magda  wrote:
>>>>>
>>>>>> What are your requirements? Do you need to process the records as
>>>>>> soon as they are put into the cluster?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Friday, October 2, 2020, narges saleh 
>>>>>> wrote:
>>>>>>
>>>>>>> Thank you Dennis for the reply.
>>>>>>> From the perspective of performance/resource overhead and
>>>>>>> reliability, which approach is preferable? Does a continuous query based
>>>>>>> approach impose a lot more overhead?
>>>>>>>
>>>>>>> On Fri, Oct 2, 2020 at 9:52 AM Denis Magda 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Narges,
>>>>>>>>
>>>>>>>> Use continuous queries if you need to be notified in real-time,
>>>>>>>> i.e. 1) a record is inserted, 2) the continuous filter confirms the
>>>>>>>> record's time satisfies your condition, 3) the continuous queries 
>>>>>>>> notifies
>>>>>>>> your application that does require processing.
>>>>>>>>
>>>>>>>> The jobs are better for a batching use case when it's ok to process
>>>>>>>> records together with some delay.
>>>>>>>>
>>>>>>>>
>>>>>>>> -
>>>>>>>> Denis
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Oct 2, 2020 at 3:50 AM narges saleh 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>  If I want to watch for a rolling timestamp pattern in all the
>>>>>>>>> records that get inserted to all my caches, is it more efficient to 
>>>>>>>>> use
>>>>>>>>> timer based jobs (that checks all the records in some interval) or
>>>>>>>>> continuous queries that locally filter on the pattern? These records 
>>>>>>>>> can
>>>>>>>>> get inserted in any order  and some can arrive with delays.
>>>>>>>>> An example is to watch for all the records whose timestamp ends in
>>>>>>>>> 50, if the timestamp is in the format -mm-dd hh:mi.
>>>>>>>>>
>>>>>>>>> thanks
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> -
>>>>>> Denis
>>>>>>
>>>>>>

Re: Continuous Query

[Denis Magda]

Good. Another clarification:

   - Does that calculation change the state of the record (updates any
   fields)?
   - Does the calculation read or update any other records?

-
Denis


On Sat, Oct 3, 2020 at 1:34 PM narges saleh  wrote:

> The latter; the server needs to perform some calculations on the data
> without sending any notification to the app.
>
> On Fri, Oct 2, 2020 at 4:25 PM Denis Magda  wrote:
>
>> And after you detect a record that satisfies the condition, do you need
>> to send any notification to the application? Or is it more like a server
>> detects and does some calculation logically without updating the app.
>>
>> -
>> Denis
>>
>>
>> On Fri, Oct 2, 2020 at 11:22 AM narges saleh 
>> wrote:
>>
>>> The detection should happen at most a couple of minutes after a record
>>> is inserted in the cache but all the detections are local to the node. But
>>> some records with the current timestamp might show up in the system with
>>> big delays.
>>>
>>> On Fri, Oct 2, 2020 at 12:23 PM Denis Magda  wrote:
>>>
>>>> What are your requirements? Do you need to process the records as soon
>>>> as they are put into the cluster?
>>>>
>>>>
>>>>
>>>> On Friday, October 2, 2020, narges saleh  wrote:
>>>>
>>>>> Thank you Dennis for the reply.
>>>>> From the perspective of performance/resource overhead and reliability,
>>>>> which approach is preferable? Does a continuous query based approach 
>>>>> impose
>>>>> a lot more overhead?
>>>>>
>>>>> On Fri, Oct 2, 2020 at 9:52 AM Denis Magda  wrote:
>>>>>
>>>>>> Hi Narges,
>>>>>>
>>>>>> Use continuous queries if you need to be notified in real-time, i.e.
>>>>>> 1) a record is inserted, 2) the continuous filter confirms the record's
>>>>>> time satisfies your condition, 3) the continuous queries notifies your
>>>>>> application that does require processing.
>>>>>>
>>>>>> The jobs are better for a batching use case when it's ok to process
>>>>>> records together with some delay.
>>>>>>
>>>>>>
>>>>>> -
>>>>>> Denis
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 2, 2020 at 3:50 AM narges saleh 
>>>>>> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>  If I want to watch for a rolling timestamp pattern in all the
>>>>>>> records that get inserted to all my caches, is it more efficient to use
>>>>>>> timer based jobs (that checks all the records in some interval) or
>>>>>>> continuous queries that locally filter on the pattern? These records can
>>>>>>> get inserted in any order  and some can arrive with delays.
>>>>>>> An example is to watch for all the records whose timestamp ends in
>>>>>>> 50, if the timestamp is in the format -mm-dd hh:mi.
>>>>>>>
>>>>>>> thanks
>>>>>>>
>>>>>>>
>>>>
>>>> --
>>>> -
>>>> Denis
>>>>
>>>>

Re: Ignite thread pool configuration

[Denis Magda]

Give it a try but do some load testing close to your production workload.
And then ramp the numbers up if needed.

-
Denis


On Mon, Oct 5, 2020 at 12:56 AM VeenaMithare 
wrote:

> Thanks Denis,
>
> I am thinking of setting the below thread pools as this on both client and
> server since we dont use data streamer, IGFS or Peer Class loading:
>
> 
> 
> 
>
> Also our thick clients dont connect using REST . So thinking of adding the
> below configuration on our thick client configuration.
>
> 
>  class="org.apache.ignite.configuration.ConnectorConfiguration">
> 
> 
> 
>
> Hope that is okay ?
>
> regards,
> Veena.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: [Announcement] New Ignite docs merged to the master

[Denis Magda]

Just in case, we're still fighting some odd servers-synchronization issue
with the INFRA team: https://issues.apache.org/jira/browse/INFRA-20925

Thus, empty browsers cache to see the latest version of the docs. Sometimes
you might need to do this several times depending on where your request is
served :)

-
Denis


On Mon, Oct 5, 2020 at 3:00 PM Denis Magda  wrote:

> Igniters,
>
> Just a shout-out that the new docs made it to the master (sources in
> AsciiDoc format) with the latest version published to the website (HTML
> format): https://ignite.apache.org/docs/latest/
>
> Thanks to everyone involved. Special thanks go to @Artem Budnikov
>  and @Mauricio Stekl  who
> were the main contributing force behind the effort.
>
> Effectively now, send all pull-request and do changes in the master
> branch. I'll prepare the how-to-contribute instruction in the following
> week or two.
>
> Also, I'll prepare a detailed status and share it in the other discussion
> below. Use that discussion to report any issues:
> http://apache-ignite-developers.2346864.n4.nabble.com/New-Ignite-Docs-Status-and-Incomplete-Items-td49268.html
>
> -
> Denis
>

[Announcement] New Ignite docs merged to the master

[Denis Magda]

Igniters,

Just a shout-out that the new docs made it to the master (sources in
AsciiDoc format) with the latest version published to the website (HTML
format): https://ignite.apache.org/docs/latest/

Thanks to everyone involved. Special thanks go to @Artem Budnikov
 and @Mauricio Stekl  who
were the main contributing force behind the effort.

Effectively now, send all pull-request and do changes in the master branch.
I'll prepare the how-to-contribute instruction in the following week or two.

Also, I'll prepare a detailed status and share it in the other discussion
below. Use that discussion to report any issues:
http://apache-ignite-developers.2346864.n4.nabble.com/New-Ignite-Docs-Status-and-Incomplete-Items-td49268.html

-
Denis

Re: Ignite thread pool configuration

[Denis Magda]

You can certainly reduce its size (
https://ignite.apache.org/docs/latest/perf-and-troubleshooting/thread-pools-tuning#data-streamer-pool).
But at the same time created but unused threads are just some objects that
consume a bit of memory but don't use any CPU resources (until you start
using those).

-
Denis


On Sat, Oct 3, 2020 at 6:12 AM VeenaMithare 
wrote:

> I noticed that the DataStreamer thread pool size seems to be around 32 in
> my
> local machine. In our cluster setup we dont stream data from any external
> db
> . We use native persistence. Do you think it makes sense to reduce this
> pool
> size for both server and client configuraiton. I am not sure if this pool
> will ever be used in our project.
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Continuous Query

[Denis Magda]

And after you detect a record that satisfies the condition, do you need to
send any notification to the application? Or is it more like a server
detects and does some calculation logically without updating the app.

-
Denis


On Fri, Oct 2, 2020 at 11:22 AM narges saleh  wrote:

> The detection should happen at most a couple of minutes after a record is
> inserted in the cache but all the detections are local to the node. But
> some records with the current timestamp might show up in the system with
> big delays.
>
> On Fri, Oct 2, 2020 at 12:23 PM Denis Magda  wrote:
>
>> What are your requirements? Do you need to process the records as soon as
>> they are put into the cluster?
>>
>>
>>
>> On Friday, October 2, 2020, narges saleh  wrote:
>>
>>> Thank you Dennis for the reply.
>>> From the perspective of performance/resource overhead and reliability,
>>> which approach is preferable? Does a continuous query based approach impose
>>> a lot more overhead?
>>>
>>> On Fri, Oct 2, 2020 at 9:52 AM Denis Magda  wrote:
>>>
>>>> Hi Narges,
>>>>
>>>> Use continuous queries if you need to be notified in real-time, i.e. 1)
>>>> a record is inserted, 2) the continuous filter confirms the record's time
>>>> satisfies your condition, 3) the continuous queries notifies your
>>>> application that does require processing.
>>>>
>>>> The jobs are better for a batching use case when it's ok to process
>>>> records together with some delay.
>>>>
>>>>
>>>> -
>>>> Denis
>>>>
>>>>
>>>> On Fri, Oct 2, 2020 at 3:50 AM narges saleh 
>>>> wrote:
>>>>
>>>>> Hi All,
>>>>>  If I want to watch for a rolling timestamp pattern in all the records
>>>>> that get inserted to all my caches, is it more efficient to use timer 
>>>>> based
>>>>> jobs (that checks all the records in some interval) or  continuous queries
>>>>> that locally filter on the pattern? These records can get inserted in any
>>>>> order  and some can arrive with delays.
>>>>> An example is to watch for all the records whose timestamp ends in 50,
>>>>> if the timestamp is in the format -mm-dd hh:mi.
>>>>>
>>>>> thanks
>>>>>
>>>>>
>>
>> --
>> -
>> Denis
>>
>>

Re: Continuous Query

[Denis Magda]

What are your requirements? Do you need to process the records as soon as
they are put into the cluster?



On Friday, October 2, 2020, narges saleh  wrote:

> Thank you Dennis for the reply.
> From the perspective of performance/resource overhead and reliability,
> which approach is preferable? Does a continuous query based approach impose
> a lot more overhead?
>
> On Fri, Oct 2, 2020 at 9:52 AM Denis Magda  wrote:
>
>> Hi Narges,
>>
>> Use continuous queries if you need to be notified in real-time, i.e. 1) a
>> record is inserted, 2) the continuous filter confirms the record's time
>> satisfies your condition, 3) the continuous queries notifies your
>> application that does require processing.
>>
>> The jobs are better for a batching use case when it's ok to process
>> records together with some delay.
>>
>>
>> -
>> Denis
>>
>>
>> On Fri, Oct 2, 2020 at 3:50 AM narges saleh  wrote:
>>
>>> Hi All,
>>>  If I want to watch for a rolling timestamp pattern in all the records
>>> that get inserted to all my caches, is it more efficient to use timer based
>>> jobs (that checks all the records in some interval) or  continuous queries
>>> that locally filter on the pattern? These records can get inserted in any
>>> order  and some can arrive with delays.
>>> An example is to watch for all the records whose timestamp ends in 50,
>>> if the timestamp is in the format -mm-dd hh:mi.
>>>
>>> thanks
>>>
>>>

-- 
-
Denis

Re: Continuous Query

[Denis Magda]

Hi Narges,

Use continuous queries if you need to be notified in real-time, i.e. 1) a
record is inserted, 2) the continuous filter confirms the record's time
satisfies your condition, 3) the continuous queries notifies your
application that does require processing.

The jobs are better for a batching use case when it's ok to process records
together with some delay.


-
Denis


On Fri, Oct 2, 2020 at 3:50 AM narges saleh  wrote:

> Hi All,
>  If I want to watch for a rolling timestamp pattern in all the records
> that get inserted to all my caches, is it more efficient to use timer based
> jobs (that checks all the records in some interval) or  continuous queries
> that locally filter on the pattern? These records can get inserted in any
> order  and some can arrive with delays.
> An example is to watch for all the records whose timestamp ends in 50, if
> the timestamp is in the format -mm-dd hh:mi.
>
> thanks
>
>

Re: Question about Ignite persistence disk space used after clearing cache

[Denis Magda]

Hi Scott,

The disk space is not compacted even after you clear the entire cache. The
compaction feature will be introduced to Ignite soon. So, the metric shows
the allocated size. This doc section suggests an approach for the actual
size calculation:
https://www.gridgain.com/docs/latest/administrators-guide/monitoring-metrics/metrics#allocated-space-vs-actual-size-of-data



-
Denis


On Fri, Sep 25, 2020 at 1:52 PM Scott Prater  wrote:

> I have a question about how the off-heap usage is reported when Ignite
> persistence is configured.  I have a single node set up.  I stored about
> 1GB of items in the cache, then cleared the cache (remotely, using the Java
> thin client:  ClientCache.clear()).
>
> I then verified that the items were no longer in the cache.
>
> However, when I look at the Ignite log, I do not see that the disk space
> was freed:
>
> [2020-09-25T11:17:36,299][INFO ][grid-timeout-worker-#23][IgniteKernal]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=db4ed295, uptime=00:34:00.176]
> ^-- H/N/C [hosts=1, nodes=1, CPUs=8]
> ^-- CPU [cur=0.2%, avg=0.3%, GC=0%]
> ^-- PageMemory [pages=250315]
> ^-- Heap [used=180MB, free=94.85%, comm=438MB]
> ^-- Off-heap [used=989MB, free=88.35%, comm=8392MB]
> ^--   sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
> ^--   default_region region [used=989MB, free=87.92%, comm=8192MB]
> ^--   metastoreMemPlc region [used=0MB, free=99.94%, comm=0MB]
> ^--   TxLog region [used=0MB, free=100%, comm=100MB]
> ^-- Ignite persistence [used=998MB]
> ^--   sysMemPlc region [used=0MB]
> ^--   default_region region [used=998MB]
> ^--   metastoreMemPlc region [used=0MB]
> ^--   TxLog region [used=0MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=0, idle=6, qSize=0]
>
> "Ignite persistence [used=998MB]" seems to indicate that 1GB of data is
> still in the cache.  Is this simply a report of the disk space *allocated*,
> or is actual disk space in use?  Is there a way to get both measurements?
>
> thanks,
>
> -- Scott
>

Re: Fast Load Large Datasets

[Denis Magda]

Vladimir,

It was a pleasure working with you on this article. Igniters, you'll find
useful tips & tricks on how to load large tables. Read and bookmark.

-
Denis


On Thu, Sep 24, 2020 at 9:43 PM  wrote:

>
> Igniters,
>
> My tutorial post about loading big tables into Apache Ignite has finally
> arrived [1]. Many thanks to @Denis Magda and @Ksenia Romanova for their
> valuable help.
>
> [1]
> https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api
>
> Vladimir
>

Re: [DISCUSSION] Renaming Ignite's product category

[Denis Magda]

Adam,

Like your way of thinking. It makes sense to me.

For me, a computing platform comprises two essential components - a storage
engine and compute APIs (not only compute grid, but also streaming, SQL,
etc.). Under this definition, Ignite fits the "compute platform" category
for sure, but the problem (and the reason why I started this discussion
thread) is that the "in-memory computing platform" is not a ubiquitous
product category. At the same time, we all know that Ignite is used to
speed things up by storing and executing in memory. And if you need to make
software faster, you usually search for something scalable or in-memory
(in-memory caches, in-memory, NoSQL, distributed databases). Basically,
most of the folks search for "caches" and "in-memory database" which is
related to Ignite and only the tip of the iceberg searches for "in-memory
computing platforms" and "data grids".

Btw, how has your story started with Ignite? How did you find out about it?

-
Denis


On Wed, Sep 23, 2020 at 1:24 PM Carbone, Adam 
wrote:

> Good Evening Denis,
>
> I’m sure there are more out there as well,  I would consider platforms
> that the entire applications but that all the execution of code happens
> exclusively on the platform, and most of the applications are written to
> run in, not connected to the platform.
>
> Hmmm by this criteria k8s could possible fit the bill…
>
> Spark might even be considered a compute grid as well but it is not a
> generic compute framework and people don’t usually run there whole
> applications inside.
>
> What is the vision for the platform? That might help in this discussion,
> set your category with the direction you are heading, and what you are
> trying to achieve.
>
> Regards
>
>
>
> Adam Carbone | Director of Innovation – Intelligent Platform Team |
> Bottomline Technologies
> Office: 603-501-6446 | Mobile: 603-570-8418
> www.bottomline.com
>
>
>
>
>
>
>
> *From: *Denis Magda 
> *Date: *Wednesday, September 23, 2020 at 4:14 PM
> *To: *dev , "Carbone, Adam" <
> adam.carb...@bottomline.com>
> *Cc: *"user@ignite.apache.org" 
> *Subject: *Re: [DISCUSSION] Renaming Ignite's product category
>
>
>
> Adam,
>
>
>
> You defined GigaSpaces as a true in-memory computing platform. What is the
> true platform for you?
>
>
>
>
> -
>
> Denis
>
>
>
>
>
> On Fri, Sep 18, 2020 at 7:02 AM Carbone, Adam 
> wrote:
>
> So when I came across Ignite It was described as an In Memory Data Grid
>
> So one way to look at this is who do you fashion as Ignite competing
> against?
>
> Are competing against Redis, Aerospike - In Memory Databases
>
> Or are you more competing with
>
> Gigaspaces - True In memory Compute platform
>
> And then you have like of
>
> Hazelcast that started as a Distributed Hash and have gained some
> features...
>
> On thing that I think is a differentiator that isn't being highlighted but
> Is  unique feature to Ignited, and the primary reason we ended up here; The
> integration with spark and it's distributed/shared Datasets/Dataframes.
>
> I don't know for me the In Memory Data Grid I think fits what Ignite is...
>
> Regards
>
> ~Adam
>
> Adam Carbone | Director of Innovation – Intelligent Platform Team |
> Bottomline Technologies
> Office: 603-501-6446 | Mobile: 603-570-8418
> www.bottomline.com
>
>
>
> On 9/17/20, 11:45 AM, "Glenn Wiebe"  wrote:
>
> I agree with Stephen about "database" devaluing what Ignite can do
> (though
> it probably hits the majority of existing use cases). I tend to go with
> "massively distributed storage and compute platform"
>
> I know, I didn't take sides, I just have both.
>
> Cheers,
>   Glenn
>
> On Thu., Sep. 17, 2020, 7:04 a.m. Stephen Darlington, <
> stephen.darling...@gridgain.com> wrote:
>
> > I think this is a great question. Explaining what Ignite does is
> always a
> > challenge, so having a useful “tag line” would be very valuable.
> >
> > I’m not sure what the answer is but I think calling it a “database”
> > devalues all the compute facilities. "Computing platform” may be too
> vague
> > but it at least says that we do more than “just” store data.
> >
> > On 17 Sep 2020, at 06:29, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > My vote is for the "distributed memory-first database". It clearly
> states
> > that Ignite is a database (which is true at this point), while still
>  

Re: [DISCUSSION] Renaming Ignite's product category

[Denis Magda]

Adam,

You defined GigaSpaces as a true in-memory computing platform. What is the
true platform for you?


-
Denis


On Fri, Sep 18, 2020 at 7:02 AM Carbone, Adam 
wrote:

> So when I came across Ignite It was described as an In Memory Data Grid
>
> So one way to look at this is who do you fashion as Ignite competing
> against?
>
> Are competing against Redis, Aerospike - In Memory Databases
>
> Or are you more competing with
>
> Gigaspaces - True In memory Compute platform
>
> And then you have like of
>
> Hazelcast that started as a Distributed Hash and have gained some
> features...
>
> On thing that I think is a differentiator that isn't being highlighted but
> Is  unique feature to Ignited, and the primary reason we ended up here; The
> integration with spark and it's distributed/shared Datasets/Dataframes.
>
> I don't know for me the In Memory Data Grid I think fits what Ignite is...
>
> Regards
>
> ~Adam
>
> Adam Carbone | Director of Innovation – Intelligent Platform Team |
> Bottomline Technologies
> Office: 603-501-6446 | Mobile: 603-570-8418
> www.bottomline.com
>
>
>
> On 9/17/20, 11:45 AM, "Glenn Wiebe"  wrote:
>
> I agree with Stephen about "database" devaluing what Ignite can do
> (though
> it probably hits the majority of existing use cases). I tend to go with
> "massively distributed storage and compute platform"
>
> I know, I didn't take sides, I just have both.
>
> Cheers,
>   Glenn
>
> On Thu., Sep. 17, 2020, 7:04 a.m. Stephen Darlington, <
> stephen.darling...@gridgain.com> wrote:
>
> > I think this is a great question. Explaining what Ignite does is
> always a
> > challenge, so having a useful “tag line” would be very valuable.
> >
> > I’m not sure what the answer is but I think calling it a “database”
> > devalues all the compute facilities. "Computing platform” may be too
> vague
> > but it at least says that we do more than “just” store data.
> >
> > On 17 Sep 2020, at 06:29, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > My vote is for the "distributed memory-first database". It clearly
> states
> > that Ignite is a database (which is true at this point), while still
> > emphasizing the in-memory computing power endorsed by the platform.
> >
> > The "in-memory computing platform" is an ambiguous term and doesn't
> really
> > reflect what Ignite is, especially in its current state.
> >
> > -Val
> >
> > On Wed, Sep 16, 2020 at 3:53 PM Denis Magda 
> wrote:
> >
> >> Igniters,
> >>
> >> Throughout the history of our project, we could see how the
> addition of
> >> certain features required us to reassess the project's name and
> category.
> >>
> >> Before Ignite joined the ASF, it supported only compute APIs
> resembling
> >> the
> >> MapReduce engine of Hadoop. Those days, it was fair to define
> Ignite as "a
> >> distributed in-memory computing engine". Next, at the time of the
> project
> >> donation, it already included key-value/SQL/transactional APIs, was
> used
> >> as
> >> a distributed cache, and significantly outgrew the "in-memory
> computing
> >> engine" use case. That's how the project transitioned to the product
> >> category of in-memory caches and we started to name it as an
> "in-memory
> >> data grid" or "in-memory computing platform" to differentiate from
> >> classical caching products such as Memcached and Redis.
> >>
> >> Nowadays, the project outgrew its caching use case, and the
> classification
> >> of Ignite as an "in-memory data grid" or "in-memory computing
> platform"
> >> doesn't sound accurate. We rebuilt our storage engine by replacing a
> >> typical key-value engine with a B-tree engine that spans across
> memory and
> >> disk tiers. And it's not surprising to see more deployments of
> Ignite as a
> >> database on its own. So, it feels like we need to reconsider Ignite
> >> positioning again so that a) application developers can discover it
> easily
> >> via search engines and b) the project can stand out from in-memory
> >> projects
> >> with intersecting capabilities.
> >>
> >> To the poin

[DISCUSSION] Renaming Ignite's product category

[Denis Magda]

Igniters,

Throughout the history of our project, we could see how the addition of
certain features required us to reassess the project's name and category.

Before Ignite joined the ASF, it supported only compute APIs resembling the
MapReduce engine of Hadoop. Those days, it was fair to define Ignite as "a
distributed in-memory computing engine". Next, at the time of the project
donation, it already included key-value/SQL/transactional APIs, was used as
a distributed cache, and significantly outgrew the "in-memory computing
engine" use case. That's how the project transitioned to the product
category of in-memory caches and we started to name it as an "in-memory
data grid" or "in-memory computing platform" to differentiate from
classical caching products such as Memcached and Redis.

Nowadays, the project outgrew its caching use case, and the classification
of Ignite as an "in-memory data grid" or "in-memory computing platform"
doesn't sound accurate. We rebuilt our storage engine by replacing a
typical key-value engine with a B-tree engine that spans across memory and
disk tiers. And it's not surprising to see more deployments of Ignite as a
database on its own. So, it feels like we need to reconsider Ignite
positioning again so that a) application developers can discover it easily
via search engines and b) the project can stand out from in-memory projects
with intersecting capabilities.

To the point, I'm suggesting to reposition Ignite in one of the following
ways:

   1. Ignite is a "distributed X database". We are indeed a distributed
   partitioned database where X can be "multi-tiered" or "memory-first" to
   emphasize that we are more than an in-memory database.
   2. Keep defining Ignite as "an in-memory computing platform" but name
   our storage engine uniquely as "IgniteDB" to highlight that the platform is
   powered by a "distributed multi-tiered/memory-first database".

What is your thinking?


(Also, regardless of a selected name, Ignite still will be used as a cache
and grid, and we're not going to stop appealing to those use cases. But
those are just use cases while Ignite has to figure out its new identity
... again).


-
Denis

Re: Tech talk: thoughts and suggestions on how to use Ignite with serverless functions

[Denis Magda]

Pavel, thanks for catching that. These are the correct coordinates of the
tutorial
<https://www.gridgain.com/docs/tutorials/serverless/azure-functions-tutorial>
.

-
Denis


On Tue, Sep 15, 2020 at 10:21 AM Pavel Tupitsyn 
wrote:

> Hi Denis,
>
> The tutorial link leads to Martin's website
>
> On Tue, Sep 15, 2020 at 7:43 PM Denis Magda  wrote:
>
>> Igniters,
>>
>> The serverless architecture
>> <https://martinfowler.com/articles/serverless.html#WhatIsServerless> is
>> getting more traction these days since it helps us to reduce the costs of
>> our cloud-based applications.
>>
>> Several months ago I decided to research if Ignite is 100% ready for
>> serverless deployments. This tutorial
>> <https://martinfowler.com/articles/serverless.html#WhatIsServerless> is
>> the first result of the research, plus we planned to perform several
>> optimizations in the Ignite codebase. Overall, there are some specificities
>> and limitations, but, nothing bars us from using Ignite with serverless
>> functions.
>>
>> Tomorrow, September 16th, I'd like to elucidate more on my current
>> findings by giving a talk on the topic. Please join the talk
>> <https://www.gridgain.com/resources/webinars/going-cloud-native-serverless-applications-apache-ignite>
>> or respond here if you'd like to get a reference to the recording.
>>
>> -
>> Denis
>>
>

Tech talk: thoughts and suggestions on how to use Ignite with serverless functions

[Denis Magda]

Igniters,

The serverless architecture
 is
getting more traction these days since it helps us to reduce the costs of
our cloud-based applications.

Several months ago I decided to research if Ignite is 100% ready for
serverless deployments. This tutorial
 is the
first result of the research, plus we planned to perform several
optimizations in the Ignite codebase. Overall, there are some specificities
and limitations, but, nothing bars us from using Ignite with serverless
functions.

Tomorrow, September 16th, I'd like to elucidate more on my current findings
by giving a talk on the topic. Please join the talk

or respond here if you'd like to get a reference to the recording.

-
Denis

Re: Map 10 caches to 10 different ignite nodes

[Denis Magda]

Take a look at this documentation

page that share a bit more detail on how to use the node filters.

-
Denis


On Fri, Sep 11, 2020 at 5:19 AM abhi2703  wrote:

> Thanks a lot Vladimir, that's exactly what I was searching for.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite Segmentation Behaviour

[Denis Magda]

Hi Samuel,

With the current behavior, the segments will not rejoin automatically. Once
the network is recovered from a network partitioning event, you need to
restart all the nodes of one of the segments. Those nodes will join the
other nodes and the cluster will become fully operational.

Let me know if you have any other questions or guidance with this.

-
Denis


On Fri, Sep 11, 2020 at 7:38 AM Samuel Ueltschi <
samuel.uelts...@bsi-software.com> wrote:

> Hi
>
>
>
> I've been testing Ignite (2.8.1) and it's behaviour under network
> segmentation.
>
> According to the docs, Ignite nodes should be able to detect network
> segmentation and apply the configured SegmentationPolicy.
>
>
>
> However the segmentation handling didn't trigger as I would have expected
> it to do.
>
> For my tests, I setup three cluster nodes c1, c2 and c3 running in docker
> containers, all competing for a shared IgniteLock instance in a loop.
>
> Then I used iptables in container c2 to drop all incoming and outgoing
> packages on that node.
>
> After a few seconds I got the following events:
>
>
>
> c1:
>
> - EVT_NODE_FAILED for c2
>
>
>
> c2:
>
> - EVT_NODE_FAILED for c1
>
> - EVT_NODE_FAILED for c3
>
>
>
> c3:
>
> - EVT_NODE_FAILED for c2
>
>
>
> Then I reset the iptables rules expecting that c2 would rejoin the cluster
> and detect segmentation.
>
> However this didn't happen, c2 just keeps running as a second standalone
> cluster instance.
>
> Only after restarting c2 it rejoined the cluster.
>
>
>
> Eventually I was able to trigger the EVT_NODE_SEGMENTED event by pausing
> the c2 container for 1minute. After resuming, c2 detects the segmentation
> and runs the segmentation policy as excepcted.
>
>
>
> Is this behaviour correct? Shouldn't the Ignite cluster be able to recover
> from the first scenario?
>
> During a network segmentation no packages would be able to move between
> nodes, so the iptables approach should be realistic in my oppinion.
>
>
>
> Maybe I have some wrong assumptions about network segmentation so any
> feedback would be greatly appreciated.
>
>
>
> Cheers Sam
>
>
>
> --
> Software Engineer
> BSI Business Systems Integration AG
> Erlachstrasse 16B, CH-3012 Bern
> Telefon +41 31 850 12 06
>
> www.bsi-software.com
>
>
>

Re: InvalidClassException local class incompatible for custom cache store factory

[Denis Magda]

Hi Xinmin,

Are you getting this error for your custom objects? If yes, did you change
a type of a field in those objects?

Probably, you're hitting this problem:
https://www.gridgain.com/docs/latest/perf-troubleshooting-guide/troubleshooting#cluster-doesnt-start-after-field-type-changes


-
Denis


On Tue, Sep 8, 2020 at 11:42 AM xmw45688  wrote:

> Hi Denis,  I have the same issue when persistenceEnabled = true and when
> Ignite is upgrade  from 2.8.0 to 2.8.1.  The data has been stored in the
> disk (single node) running on Ignite 2.8.0. This exception happens for
> persistence classes that do not implement java.io.serializable when
> upgrading to 2.8.1.  Adding java.io.serializable does not resolve this
> exception.
>
> In general when persistenceEnabled = true and the data is stored in the
> disk
> 1) how to resolve the discrepancies issues of Ignite internal
> implementation
> between Ignite releases, i.e. the issue I'm experiencing.
> 2) how to resolve the key and value class changes (add/remove variables)
>
> Thanks,
> Xinmin
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Regarding TRANSACTIONAL_SNAPSHOT

[Denis Magda]

Hi Dana,

Here you can find a list of known high-level limitations:
https://apacheignite.readme.io/docs/multiversion-concurrency-control#limitations

Also, the performance might not be satisfactory and not all the cluster
failure scenarios might be covered.

Generally, as of know, I would advise you to use the key-value transactions
of Ignite that are battle-tested and used in productions since the
inception of Ignite.

-
Denis


On Thu, Sep 3, 2020 at 6:41 AM danami  wrote:

> Roman Kondakov wrote
> > 1. TRANSACTIONAL_SNAPSHOT is still in beta version. Perhaps the current
> > version of MVCC will be completely redone in the future. So, it is not a
> > good idea to use it in production systems right now.
> >
> > 2. Events are not implemented for MVCC yet. As well as several other
> > features that presented in conventional TRANSACTIONAL cache:
> > interceptors, recovery, etc.
>
>
> Hi all,
>
> Can someone elaborate on the missing features of TRANSACTIONAL_SNAPSHOT
> cache mode or point me to some source of information that elaborates them?
>
> Are there any known bugs related to it in current Ignite version?
>
> Besides the possibility of it completely getting redone in the future, what
> would other major throwbacks be if it were used in production?
>
> Thank you very much,
> Dana
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: [External]Re: unable to start node

[Denis Magda]

@Taras Ledkov , @Igor Seliverstov
, @Yuriy Gerzhedovich
, did you guys see this issue before? It looks
like a bug in the SQL engine.

Kamlesh, in the meantime, try to follow the procedure described in this
JIRA ticket (you need to clean the index that we'll be rebuild on the
startup):
https://issues.apache.org/jira/browse/IGNITE-11252

Let us know if it worked out for you. We'll document this procedure.

-
Denis


On Wed, Sep 2, 2020 at 10:07 AM Kamlesh Joshi  wrote:

>
> Hi Tom,
>
> We are using java 1.8.151 and Ignite 2.7.6.
>
> Regards,
> Kamlesh
>
>
> --
> *From:* Tom Black 
> *Sent:* Wednesday, 2 September, 2020, 1:31 pm
> *To:* user@ignite.apache.org
> *Subject:* [External]Re: unable to start node
>
> The e-mail below is from an external source. Please do not open
> attachments or click links from an unknown or suspicious origin.
>
> Kamlesh Joshi wrote:
> > */[2020-08-28T19:05:03,296][ERROR][sys-#940187%EDIFReplica%][] JVM will
> > be halted immediately due to the failure: [failureCtx=FailureContext
> > [type=CRITICAL_ERROR, err=org.h2.message.DbException: General error:
> > "class o.a.i.IgniteCheckedException: Unexpected binary object class
> > [type=class java.lang.String]" [5-197]]]/*
>
> what ignite and java version are you using?
>
> regards.
>
>
>
> "*Confidentiality Warning*: This message and any attachments are intended
> only for the use of the intended recipient(s), are confidential and may be
> privileged. If you are not the intended recipient, you are hereby notified
> that any review, re-transmission, conversion to hard copy, copying,
> circulation or other use of this message and any attachments is strictly
> prohibited. If you are not the intended recipient, please notify the sender
> immediately by return email and delete this message and any attachments
> from your system.
>
> *Virus Warning:* Although the company has taken reasonable precautions to
> ensure no viruses are present in this email. The company cannot accept
> responsibility for any loss or damage arising from the use of this email or
> attachment."
>

Re: ignite partition mode

[Denis Magda]

Explain the steps in detail
-
Denis


On Wed, Aug 26, 2020 at 5:13 AM itsmeravikiran.c 
wrote:

> By using ignite web console
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Executing select query in Partition mode (cluster with two nodes) taking more than 10 minutes.

[Denis Magda]

Hi Bhavesh,

Follow these basic performance tuning recommendations for SQL
 to
understand how to optimize the query. Take the execution plan first. Most
likely, you need to set secondary indexes. If that's not the case, then
confirm the query doesn't cause long stop-the-world pauses.

-
Denis


On Tue, Aug 25, 2020 at 6:09 AM Bhavesh Modi 
wrote:

>
>
> Hi Team,
>
>
>
>   I have created cluster with two nodes. After creating new
> table ,  we are executing select query for first time , It is taking too
> much time to fetch data. (almost 10+ min)
>
>   Please find attachment my server xml configuration and below
> is  client configuration class . any suggestion to solve this problem,
> thanks in advance
>
>
>
>   public static synchronized void init() {
>
>  try {
>
> Ignition.setClientMode(true);
>
>
>
> DataStorageConfiguration storageCfg = new
> DataStorageConfiguration();
>
> storageCfg.setWalMode(WALMode.BACKGROUND);
>
>
>
> IgniteConfiguration cfg = new IgniteConfiguration();
>
> cfg.setDataStorageConfiguration(storageCfg);
>
> // Ignite Cluster changes
>
> cfg.setPeerClassLoadingEnabled(true);
>
> cfg.setRebalanceThreadPoolSize(4);
>
>
>
> TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
>
> TcpDiscoveryVmIpFinder ipFinder = new
> TcpDiscoveryVmIpFinder();
>
> ipFinder.setAddresses( Arrays.asList("127.0.0.1:47500
> ","12.16.0.19:47500"));
>
>
>
>
>
> discoverySpi.setLocalPort(47500);
>
>
>
>
>
>
> discoverySpi.setIpFinder(ipFinder).setJoinTimeout(3);
>
>
>
> TcpCommunicationSpi commSpi = new
> TcpCommunicationSpi();
>
>
>
>
> commSpi.setConnectTimeout(3).setLocalPort(48110);
>
>
>
> // this timeout is used to reconnect client to server
> if server has
>
> // failed/restarted
>
>
>
>  cfg.setClientFailureDetectionTimeout(4320);
>
>
>
> cfg.setDiscoverySpi(discoverySpi);
>
> cfg.setCommunicationSpi(commSpi);
>
>
>
> ignite = Ignition.start(cfg);
>
> ignite.cluster().active(true);
>
>
>
>ignite.cluster().baselineAutoAdjustEnabled(true);
>
>ignite.cluster().baselineAutoAdjustTimeout(0);
>
>
>
>  } catch (Exception e) {
>
> logger.error("Error in starting ignite cluster", e);
>
>  }
>
>}
>
>
>
>
>
>
> “Confidentiality Notice: The contents of this email message and any
> attachments are intended solely for the addressee(s) and may contain
> confidential and/or privileged information and may be legally protected
> from disclosure. If you are not the intended recipient of this message or
> their agent, or if this message has been addressed to you in error, please
> immediately alert the sender by reply email and then delete this message
> and any attachments. If you are not the intended recipient, you are hereby
> notified that any use, dissemination, copying, or storage of this message
> or its attachments is strictly prohibited.”
>

Re: How to stop a node from dying when memory is full?

[Denis Magda]

Folks, this article should be relevant to you. It covers all techniques to
avoid the OOM except for swapping:
https://www.gridgain.com/resources/blog/out-of-memory-apache-ignite-cluster-handling-techniques

You can use swapping as a part of your toolbox to survive the time when a
node is running out of the memory space:
https://apacheignite.readme.io/docs/swap-space

-
Denis


On Thu, Aug 27, 2020 at 2:13 AM danami  wrote:

> I'd like to extend Colin's question.
>
> What if I'm using TRANSACTIONAL_SNAPSHOT mode, therefore I can't use Page
> Eviction?
> How then, other than persistence, can I avoid OOM errors?
> Can I write a custom failure handler, to clear the cache/data region for
> example? Is it technically possible? (If so, how?) Will it work or is it
> bad
> and inconsistent as Colin suggested?
> Is using persistence my only option to avoid OOM errors or do I have other
> choices?
>
> Thank you for your help,
> Dana
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Select clause is quite more slow than same Select on MySQL

[Denis Magda]

This resource

tries to explain your issue in a bit more detail. In short, there are
scenarios when an RDBMS can easily show much better performance numbers.

-
Denis


On Thu, Aug 27, 2020 at 9:41 AM manueltg89 <
manuel.trinidad.gar...@hotmail.com> wrote:

> Hi Mike, thanks for information. I will try that.
>
> Thanks.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Failed to map SQL query

[Denis Magda]

If it's expected that the SQL query will produce a large data set then
consider using the memory quotas feature with the offloading to disk option
.
It's not available in Ignite though. GridGain releases it in its free
version - Community Edition.

-
Denis


On Tue, Aug 25, 2020 at 3:03 AM Andrey Mashenkov 
wrote:

> Hi,
>
> Most likely, the query intermediate result doesn't fit to JVM heap memory.
> The query may require all table data fetched before applying sorting.
>
> You can try to create a composite index over "act_id,mer_id,score" columns.
>
>
>
> On Tue, Aug 25, 2020 at 8:42 AM 1115098...@qq.com 
> wrote:
>
>> Hi,an error happened when I run a sql in ignite cluster. Thanks.
>>
>> Some info as follow:
>> -- sql
>> -- act_rank  has 5,000,000 rows
>> select * from act_rank
>> order by act_id,mer_id,score
>> limit 100 ;
>>
>> -- sql error info:
>> Error: javax.cache.CacheException: Failed to map SQL query to topology on
>> data node [dataNodeId=ca448962-9ce9-4321-82a7-2d12e147f34c, msg=Data node
>> has left the grid during query execution
>> [nodeId=ca448962-9ce9-4321-82a7-2d12e147f34c]] (state=5,code=1)
>> java.sql.SQLException: javax.cache.CacheException: Failed to map SQL
>> query to topology on data node
>> [dataNodeId=ca448962-9ce9-4321-82a7-2d12e147f34c, msg=Data node has left
>> the grid during query execution
>> [nodeId=ca448962-9ce9-4321-82a7-2d12e147f34c]]
>> at
>> org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:750)
>> at
>> org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute0(JdbcThinStatement.java:212)
>> at
>> org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute(JdbcThinStatement.java:475)
>> at sqlline.Commands.execute(Commands.java:823)
>> at sqlline.Commands.sql(Commands.java:733)
>> at sqlline.SqlLine.dispatch(SqlLine.java:795)
>> at sqlline.SqlLine.begin(SqlLine.java:668)
>> at sqlline.SqlLine.start(SqlLine.java:373)
>> at sqlline.SqlLine.main(SqlLine.java:265)
>>
>> -- ignite server error log
>> SELECT
>> __Z0.ID __C0_0,
>> __Z0.ACT_ID __C0_1,
>> __Z0.MEM_ID __C0_2,
>> __Z0.MER_ID __C0_3,
>> __Z0.SHOP_ID __C0_4,
>> __Z0.AREA_ID __C0_5,
>> __Z0.PHONE_NO __C0_6,
>> __Z0.SCORE __C0_7
>> FROM PUBLIC.ACT_RANK __Z0
>> ORDER BY 2, 4, 8 LIMIT 100 [90108-197]
>> at
>> org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
>> at org.h2.message.DbException.get(DbException.java:168)
>> at org.h2.message.DbException.convert(DbException.java:301)
>> at org.h2.command.Command.executeQuery(Command.java:214)
>> at
>> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:114)
>> at
>> org.apache.ignite.internal.processors.query.h2.PreparedStatementExImpl.executeQuery(PreparedStatementExImpl.java:67)
>> at
>> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:1421)
>> ... 13 more
>> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
>> at
>> org.h2.command.dml.Select$LazyResultQueryFlat.fetchNextRow(Select.java:1457)
>> at org.h2.result.LazyResult.hasNext(LazyResult.java:79)
>> at org.h2.result.LazyResult.next(LazyResult.java:59)
>> at org.h2.command.dml.Select.queryFlat(Select.java:527)
>> at org.h2.command.dml.Select.queryWithoutCache(Select.java:633)
>> at
>> org.h2.command.dml.Query.queryWithoutCacheLazyCheck(Query.java:114)
>> at org.h2.command.dml.Query.query(Query.java:352)
>> at org.h2.command.dml.Query.query(Query.java:333)
>> at
>> org.h2.command.CommandContainer.query(CommandContainer.java:114)
>> at org.h2.command.Command.executeQuery(Command.java:202)
>> ... 16 more
>>
>>
>> -- table struct   (total rows:5,000,000)
>> CREATE TABLE act_rank(
>>   id varchar(50) primary key,
>>   act_id VARCHAR(50),
>>   mem_id VARCHAR(50),
>>   mer_id VARCHAR(50),
>>   shop_id VARCHAR(50),
>>   area_id VARCHAR(50),
>>   phone_no VARCHAR(16),
>>   score INT
>> );
>>
>> -- visor info
>> visor> cache -c=@c4 -a
>> Time of the snapshot: 2020-08-24 11:20:50
>>
>> ++
>> | Name(@)  |Mode | Nodes | Total entries (Heap /
>> Off-heap) |  Primary entries (Heap / Off-heap)  |   Hits|  Misses   |
>>  Reads   |  Writes   |
>>
>> ++
>> | SQL_PUBLIC_ACT_RANK(@c4) | PARTITIONED | 3 | 500 (0 / 500)
>>  | min: 1635268 (0 / 1635268)  | min: 0| min: 0|
>> min: 0| min: 0|
>> |  | |   

Re: ignite partition mode

[Denis Magda]

How do you check that the second node doesn't keep any records? What your
verification method?

-
Denis


On Tue, Aug 18, 2020 at 3:17 AM itsmeravikiran.c 
wrote:

> Our application running with 2 ignite node,.
> But data is available in one node, and not available in the second node.
> Our cache configuration:
> Cache mode:partitioned
> Automatic mode:atomic
> Write Synchronization mode:full_sync
> Backups:1
> Rebalance delay:1
>
> We have above cache configuration.
>
> Could you please help, data is not available in secound node.
> Second node only cache ids are created and it's having no data in the cache
> ids.
>
> Could you please help me on this, shall I need add any of the configuration
> for this.
>
> My requiremnt is if node down, other node automatically re balance and get
> the data.
>
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Issue with using Ignite with Spring Data

[Denis Magda]

>
> However as I read them again, I realised that it was anyway necessary to
> load the cache before executing the SELECT sql queries on top of a cache,
> now, would this hold true in the case of Spring Data as well ? (Very likely
> yes, but want to get the confirmation) If so, then are we expected to
> preload the cache on start up and only after that will the read-through
> property kick in to add the entries into the cache for the ones which are
> missing ?


You got it right. If your records are stored in an external database, then
you have to pre-load them in Ignite first before using Ignite SQL. The
latter reads records from disk only if those are stored in Ignite
persistence. Ignite SQL can't query external databases.

Ignite can pre-load a missing record from an external database only if the
record is requested via key-value APIs. With those APIs, it's all trivial -
if Ignite doesn't find the record in its memory storage then it will create
an external-database-specific query to load the record from the external
db. SQL is much more complicated.

-
Denis


On Fri, Aug 21, 2020 at 1:51 PM Srikanta Patanjali 
wrote:

> Hi Denis,
>
> Thanks for taking time to reply and sharing those links. I can confirm to
> you that I've read through them before and have been following them as
> well.
>
> However as I read them again, I realised that it was anyway necessary to
> load the cache before executing the SELECT sql queries on top of a cache,
> now, would this hold true in the case of Spring Data as well ? (Very likely
> yes, but want to get the confirmation) If so, then are we expected to
> preload the cache on start up and only after that will the read-through
> property kick in to add the entries into the cache for the ones which are
> missing ?
>
> If my above understanding is correct then that explains why I was getting
> null results from the queries executed once the Spring boot is instantiated
> as the cache load on startup was not complete yet.
>
>
> Regards,
> Srikanta
>
> On Fri, Aug 21, 2020 at 6:47 PM Denis Magda  wrote:
>
>> Hi Srikanta,
>>
>> You forgot to share the configuration. Anyway, I think it's clear what
>> you are looking for.
>>
>> Check this example showing how to configure CacheJdbcPojoStoreFactory
>> programmatically (click on the "Java" tab, by default the example shows the
>> XML version):
>>
>> https://www.gridgain.com/docs/latest/developers-guide/persistence/external-storage#cachejdbcpojostore
>>
>> Also, if you need to import an existing schema of a relational database
>> and turn it into the CacheJdbcPojoStore config, then this feature of Web
>> Console can be helpful:
>>
>> https://www.gridgain.com/docs/web-console/latest/automatic-rdbms-integration
>>
>> Finally, keep an eye on this Spring Data + Ignite tutorial that covers
>> other areas of the integration. You might have other questions and issue
>> going forward and the tutorial can help to address them quickly:
>> https://www.gridgain.com/docs/tutorials/spring/spring-ignite-tutorial
>>
>> -
>> Denis
>>
>>
>> On Fri, Aug 21, 2020 at 9:09 AM Srikanta Patanjali 
>> wrote:
>>
>>> I'm trying to integrate a Spring data project (without JPA) with Ignite
>>> and struggling to understand some basic traits. Would be very helpful if
>>> you can share some insights on the issue I'm facing.
>>>
>>> Currently the cache has been defined as below with the client node, this
>>> config is not present in the server node, gets created when the client node
>>> joins the cluster. The repositories are detected during the instantiation
>>> of the Spring Boot application.
>>>
>>> All the documentation including the official example repo of the Apache
>>> Ignite does not pass in the data source but instead cachConfig is set with
>>> the IndexedTypes.
>>>
>>> Question: Where should I pass on the DataSource object ? Should I create
>>> a CacheJdbcPojoStoreFactory and pass on the dataSource ?
>>>
>>>
>>> Thanks,
>>> Srikanta
>>>
>>

Re: Issue with using Ignite with Spring Data

[Denis Magda]

Hi Srikanta,

You forgot to share the configuration. Anyway, I think it's clear what you
are looking for.

Check this example showing how to configure CacheJdbcPojoStoreFactory
programmatically (click on the "Java" tab, by default the example shows the
XML version):
https://www.gridgain.com/docs/latest/developers-guide/persistence/external-storage#cachejdbcpojostore

Also, if you need to import an existing schema of a relational database and
turn it into the CacheJdbcPojoStore config, then this feature of Web
Console can be helpful:
https://www.gridgain.com/docs/web-console/latest/automatic-rdbms-integration

Finally, keep an eye on this Spring Data + Ignite tutorial that covers
other areas of the integration. You might have other questions and issue
going forward and the tutorial can help to address them quickly:
https://www.gridgain.com/docs/tutorials/spring/spring-ignite-tutorial

-
Denis


On Fri, Aug 21, 2020 at 9:09 AM Srikanta Patanjali 
wrote:

> I'm trying to integrate a Spring data project (without JPA) with Ignite
> and struggling to understand some basic traits. Would be very helpful if
> you can share some insights on the issue I'm facing.
>
> Currently the cache has been defined as below with the client node, this
> config is not present in the server node, gets created when the client node
> joins the cluster. The repositories are detected during the instantiation
> of the Spring Boot application.
>
> All the documentation including the official example repo of the Apache
> Ignite does not pass in the data source but instead cachConfig is set with
> the IndexedTypes.
>
> Question: Where should I pass on the DataSource object ? Should I create a
> CacheJdbcPojoStoreFactory and pass on the dataSource ?
>
>
> Thanks,
> Srikanta
>

Re: Lost node again.

[Denis Magda]

>
> Dennis, wouldn't 15 seconds faillureDetectionTimeout cause even more
> shutdowns?


What's your current value? For sure, It doesn't make sense to decrease the
value until all mysterious pauses are figured out. The downside of a high
failureDetectionTimeout is that the cluster won't remove a node that failed
for a reason until the timeout expires. So, if there is a failed node that
has to process some operations then the rest of the cluster will be trying
to reach it out until the failureDetectionTimeout is reached. That affects
performance of some operations where the failed node has to be involved.

Btw, what's the tool you are using for the monitoring? Looks nice.

-
Denis


On Thu, Aug 20, 2020 at 6:44 AM John Smith  wrote:

> Hi here is an example of our cluster during our normal "high" usage. The
> node shutting down seems to happen on "off" hours.
>
> Dennis, wouldn't 15 seconds faillureDetectionTimeout cause even more
> shutdowns?
> We also considered more tuning stuff in the docs, we' ll see I guess...
> As for now we don't have separate disks for now.
>
>
>
> On Wed, 19 Aug 2020 at 23:35, Denis Magda  wrote:
>
>> John,
>>
>> I would try to get to the bottom of the issue, especially, if the case is
>> reproducible.
>>
>> If that's not GC then check if that's the I/O (your logs show that the
>> checkpointing rate is high):
>>
>>- You can monitor checkpointing duration with a JMX tool
>>
>> <https://www.gridgain.com/docs/latest/administrators-guide/monitoring-metrics/metrics#monitoring-checkpointing-operations>
>>  or
>>Control Center
>>
>> <https://www.gridgain.com/docs/control-center/latest/monitoring/metrics#checkpoint-duration>
>>.
>>- Configure write-throttling
>>
>> <https://www.gridgain.com/docs/latest/perf-troubleshooting-guide/persistence-tuning#pages-writes-throttling>
>>if the checkpointing buffer fills in quickly.
>>- Ideally, storage files and WALs should be stored on different SSD
>>media
>>
>> <https://www.gridgain.com/docs/latest/perf-troubleshooting-guide/persistence-tuning#keep-wals-separately>.
>>SSDs also do garbage collection and you might hit it frequently.
>>
>> As for the failureDetectionTimeout, I would set it to 15 secs until your
>> cluster is battle-tested and well-tuned for your use case.
>>
>> -
>> Denis
>>
>>
>> On Tue, Aug 18, 2020 at 10:37 AM John Smith 
>> wrote:
>>
>>> I don't see why we would get such a huge pause, in fact I have provided
>>> GC logs before and we found nothing...
>>>
>>> All operations are in the "big" partitioned 3 million cache are put or
>>> get and a query on another cache which has 450 entries. There no other
>>> caches.
>>>
>>> The nodes all have 6G off heap and 26G off heap.
>>>
>>> I think it can be IO related but I can't seem to be able to correlate it
>>> to IO. I saw some heavy IO usage but the node failed way after.
>>>
>>> Now my question is should I put the failure detection to 60s just for
>>> the sake of trying it? Isn't that too high? If i put the servers to 60s how
>>> how high should I put the clients?
>>>
>>> On Tue., Aug. 18, 2020, 7:32 a.m. Ilya Kasnacheev, <
>>> ilya.kasnach...@gmail.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> [13:39:53,242][WARNING][jvm-pause-detector-worker][IgniteKernal%company]
>>>> Possible too long JVM pause: 41779 milliseconds.
>>>>
>>>> It seems that you have too-long full GC. Either make sure it does not
>>>> happen, or increase failureDetectionTimeout to be longer than any expected
>>>> GC.
>>>>
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>>
>>>>
>>>> пн, 17 авг. 2020 г. в 17:51, John Smith :
>>>>
>>>>> Hi guys it seems every couple of weeks we lose a node... Here are the
>>>>> logs:
>>>>> https://www.dropbox.com/sh/8cv2v8q5lcsju53/AAAU6ZSFkfiZPaMwHgIh5GAfa?dl=0
>>>>>
>>>>> And some extra details. Maybe I need to do more tuning then what is
>>>>> already mentioned below, maybe set a higher timeout?
>>>>>
>>>>> 3 server nodes and 9 clients (client = true)
>>>>>
>>>>> Performance wise the cluster is not doing any kind of high volume on
>>>>> average it does about 15-20 puts/gets/queries (any combination of) per
&g

Re: Lost node again.

[Denis Magda]

John,

I would try to get to the bottom of the issue, especially, if the case is
reproducible.

If that's not GC then check if that's the I/O (your logs show that the
checkpointing rate is high):

   - You can monitor checkpointing duration with a JMX tool
   

or
   Control Center
   

   .
   - Configure write-throttling
   

   if the checkpointing buffer fills in quickly.
   - Ideally, storage files and WALs should be stored on different SSD media
   
.
   SSDs also do garbage collection and you might hit it frequently.

As for the failureDetectionTimeout, I would set it to 15 secs until your
cluster is battle-tested and well-tuned for your use case.

-
Denis


On Tue, Aug 18, 2020 at 10:37 AM John Smith  wrote:

> I don't see why we would get such a huge pause, in fact I have provided GC
> logs before and we found nothing...
>
> All operations are in the "big" partitioned 3 million cache are put or get
> and a query on another cache which has 450 entries. There no other caches.
>
> The nodes all have 6G off heap and 26G off heap.
>
> I think it can be IO related but I can't seem to be able to correlate it
> to IO. I saw some heavy IO usage but the node failed way after.
>
> Now my question is should I put the failure detection to 60s just for the
> sake of trying it? Isn't that too high? If i put the servers to 60s how how
> high should I put the clients?
>
> On Tue., Aug. 18, 2020, 7:32 a.m. Ilya Kasnacheev, <
> ilya.kasnach...@gmail.com> wrote:
>
>> Hello!
>>
>> [13:39:53,242][WARNING][jvm-pause-detector-worker][IgniteKernal%company]
>> Possible too long JVM pause: 41779 milliseconds.
>>
>> It seems that you have too-long full GC. Either make sure it does not
>> happen, or increase failureDetectionTimeout to be longer than any expected
>> GC.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 17 авг. 2020 г. в 17:51, John Smith :
>>
>>> Hi guys it seems every couple of weeks we lose a node... Here are the
>>> logs:
>>> https://www.dropbox.com/sh/8cv2v8q5lcsju53/AAAU6ZSFkfiZPaMwHgIh5GAfa?dl=0
>>>
>>> And some extra details. Maybe I need to do more tuning then what is
>>> already mentioned below, maybe set a higher timeout?
>>>
>>> 3 server nodes and 9 clients (client = true)
>>>
>>> Performance wise the cluster is not doing any kind of high volume on
>>> average it does about 15-20 puts/gets/queries (any combination of) per
>>> 30-60 seconds.
>>>
>>> The biggest cache we have is: 3 million records distributed with 1
>>> backup using the following template.
>>>
>>>   >> class="org.apache.ignite.configuration.CacheConfiguration">
>>> 
>>> 
>>> 
>>> 
>>> >> value="READ_WRITE_SAFE"/>
>>>   
>>>
>>> Persistence is configured:
>>>
>>>   
>>> >> class="org.apache.ignite.configuration.DataStorageConfiguration">
>>>   
>>>   
>>> >> class="org.apache.ignite.configuration.DataRegionConfiguration">
>>>   
>>>
>>>   
>>>   
>>> 
>>>   
>>> 
>>>   
>>>
>>> We also followed the tuning instructions for GC and I/O
>>> if [ -z "$JVM_OPTS" ] ; then
>>> JVM_OPTS="-Xms6g -Xmx6g -server -XX:MaxMetaspaceSize=256m"
>>> fi
>>>
>>> #
>>> # Uncomment the following GC settings if you see spikes in your
>>> throughput due to Garbage Collection.
>>> #
>>> JVM_OPTS="$JVM_OPTS -XX:+UseG1GC -XX:+AlwaysPreTouch
>>> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC"
>>> sysctl -w vm.dirty_writeback_centisecs=500 sysctl -w vm
>>> .dirty_expire_centisecs=500
>>>
>>>

Re: Ignite Queue (Documentation or Code defect)?

[Denis Magda]

Could you please file a ticket in Ignite JIRA for 2.10 release? That's
definitely an overlook in the current implementation.

-
Denis


On Tue, Aug 18, 2020 at 9:57 AM Humphrey  wrote:

> Hope someone pics this up. That the code is not behaving as the
> documentation
> is saying.
> (Not throwing exception when queue can not be fetched).
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Operation block on Cluster recovery/rebalance.

[Denis Magda]

>
> But on client reconnect, doesn't it mean it will still block until the
> cluster is active even if I get new IgniteCache instance?


No, the client will be getting an exception on an attempt to get an
IgniteCache instance.

-
Denis


On Fri, Aug 14, 2020 at 4:14 PM John Smith  wrote:

> Yeah I can maybe use vertx event bus or something to do this... But now I
> have to tie the ignite instance to the IgniteCahe repository I wrote.
>
> But on client reconnect, doesn't it mean it will still block until the
> cluster is active even if I get new IgniteCache instance?
>
> On Fri, 14 Aug 2020 at 18:22, Denis Magda  wrote:
>
>> @Evgenii Zhuravlev , @Ilya Kasnacheev
>> , any thoughts on this?
>>
>> As a dirty workaround, you can update your cache references on client
>> reconnect events. You will be getting an exception by calling
>> ignite.cache(cacheName) in the time when the cluster is not activated yet.
>> Does this work for you?
>>
>> -
>> Denis
>>
>>
>> On Fri, Aug 14, 2020 at 3:12 PM John Smith 
>> wrote:
>>
>>> Is there any work around? I can't have an HTTP server block on all
>>> requests.
>>>
>>> 1- I need to figure out why I lose a server nodes every few weeks, which
>>> when rebooting the nodes cause the inactive state until they are back
>>>
>>> 2- Implement some kind of logic on the client side not to block the HTTP
>>> part...
>>>
>>> Can IgniteCache instance be notified of disconnected events so I can
>>> maybe tell the repository class I have to set a flag to skip the operation?
>>>
>>>
>>> On Fri., Aug. 14, 2020, 5:17 p.m. Denis Magda, 
>>> wrote:
>>>
>>>> My guess that it's standard behavior for all operations (SQL,
>>>> key-value, compute, etc.). But I'll let the maintainers of those modules
>>>> clarify.
>>>>
>>>> -
>>>> Denis
>>>>
>>>>
>>>> On Fri, Aug 14, 2020 at 1:44 PM John Smith 
>>>> wrote:
>>>>
>>>>> Hi Denis, so to understand it's all operations or just the query?
>>>>>
>>>>> On Fri., Aug. 14, 2020, 12:53 p.m. Denis Magda, 
>>>>> wrote:
>>>>>
>>>>>> John,
>>>>>>
>>>>>> Ok, we nailed it. That's the current expected behavior. Generally, I
>>>>>> agree with you that the platform should support an option when operations
>>>>>> fail if the cluster is deactivated. Could you propose the change by
>>>>>> starting a discussion on the dev list? You can refer to this user list
>>>>>> discussion for reference. Let me know if you need help with this.
>>>>>>
>>>>>> -
>>>>>> Denis
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 13, 2020 at 5:55 PM John Smith 
>>>>>> wrote:
>>>>>>
>>>>>>> No I, reuse the instance. The cache instance is created once at
>>>>>>> startup of the application and I pass it to my "repository" class
>>>>>>>
>>>>>>> public abstract class AbstractIgniteRepository implements 
>>>>>>> CacheRepository {
>>>>>>> public final long DEFAULT_OPERATION_TIMEOUT = 2000;
>>>>>>>
>>>>>>> private Vertx vertx;
>>>>>>> private IgniteCache cache;
>>>>>>>
>>>>>>> AbstractIgniteRepository(Vertx vertx, IgniteCache cache) {
>>>>>>> this.vertx = vertx;
>>>>>>> this.cache = cache;
>>>>>>> }
>>>>>>>
>>>>>>> ...
>>>>>>>
>>>>>>> Future> query(final String sql, final long 
>>>>>>> timeoutMs, final Object... args) {
>>>>>>> final Promise> promise = Promise.promise();
>>>>>>>
>>>>>>> vertx.setTimer(timeoutMs, l -> {
>>>>>>> promise.tryFail(new TimeoutException("Cache operation did 
>>>>>>> not complete within: " + timeoutMs + " Ms.")); // THIS FIRE IF THE BLOE 
>>>>>>> DOESN"T COMPLETE IN TIME.
>>>>>>> });
>>>>>>>
>>>>>>> vertx.>executeBlocking(code -> {
>>>>>

Re: Operation block on Cluster recovery/rebalance.

[Denis Magda]

@Evgenii Zhuravlev , @Ilya Kasnacheev
, any thoughts on this?

As a dirty workaround, you can update your cache references on client
reconnect events. You will be getting an exception by calling
ignite.cache(cacheName) in the time when the cluster is not activated yet.
Does this work for you?

-
Denis


On Fri, Aug 14, 2020 at 3:12 PM John Smith  wrote:

> Is there any work around? I can't have an HTTP server block on all
> requests.
>
> 1- I need to figure out why I lose a server nodes every few weeks, which
> when rebooting the nodes cause the inactive state until they are back
>
> 2- Implement some kind of logic on the client side not to block the HTTP
> part...
>
> Can IgniteCache instance be notified of disconnected events so I can maybe
> tell the repository class I have to set a flag to skip the operation?
>
>
> On Fri., Aug. 14, 2020, 5:17 p.m. Denis Magda,  wrote:
>
>> My guess that it's standard behavior for all operations (SQL, key-value,
>> compute, etc.). But I'll let the maintainers of those modules clarify.
>>
>> -
>> Denis
>>
>>
>> On Fri, Aug 14, 2020 at 1:44 PM John Smith 
>> wrote:
>>
>>> Hi Denis, so to understand it's all operations or just the query?
>>>
>>> On Fri., Aug. 14, 2020, 12:53 p.m. Denis Magda, 
>>> wrote:
>>>
>>>> John,
>>>>
>>>> Ok, we nailed it. That's the current expected behavior. Generally, I
>>>> agree with you that the platform should support an option when operations
>>>> fail if the cluster is deactivated. Could you propose the change by
>>>> starting a discussion on the dev list? You can refer to this user list
>>>> discussion for reference. Let me know if you need help with this.
>>>>
>>>> -
>>>> Denis
>>>>
>>>>
>>>> On Thu, Aug 13, 2020 at 5:55 PM John Smith 
>>>> wrote:
>>>>
>>>>> No I, reuse the instance. The cache instance is created once at
>>>>> startup of the application and I pass it to my "repository" class
>>>>>
>>>>> public abstract class AbstractIgniteRepository implements 
>>>>> CacheRepository {
>>>>> public final long DEFAULT_OPERATION_TIMEOUT = 2000;
>>>>>
>>>>> private Vertx vertx;
>>>>> private IgniteCache cache;
>>>>>
>>>>> AbstractIgniteRepository(Vertx vertx, IgniteCache cache) {
>>>>> this.vertx = vertx;
>>>>> this.cache = cache;
>>>>> }
>>>>>
>>>>> ...
>>>>>
>>>>> Future> query(final String sql, final long timeoutMs, 
>>>>> final Object... args) {
>>>>> final Promise> promise = Promise.promise();
>>>>>
>>>>> vertx.setTimer(timeoutMs, l -> {
>>>>> promise.tryFail(new TimeoutException("Cache operation did not 
>>>>> complete within: " + timeoutMs + " Ms.")); // THIS FIRE IF THE BLOE 
>>>>> DOESN"T COMPLETE IN TIME.
>>>>> });
>>>>>
>>>>> vertx.>executeBlocking(code -> {
>>>>> SqlFieldsQuery query = new SqlFieldsQuery(sql).setArgs(args);
>>>>> query.setTimeout((int) timeoutMs, TimeUnit.MILLISECONDS);
>>>>>
>>>>>
>>>>> try (QueryCursor> cursor = cache.query(query)) { // 
>>>>> <--- BLOCKS HERE.
>>>>> List rows = new ArrayList<>();
>>>>> Iterator> iterator = cursor.iterator();
>>>>>
>>>>> while(iterator.hasNext()) {
>>>>> List currentRow = iterator.next();
>>>>> JsonArray row = new JsonArray();
>>>>>
>>>>> currentRow.forEach(o -> row.add(o));
>>>>>
>>>>> rows.add(row);
>>>>> }
>>>>>
>>>>> code.complete(rows);
>>>>> } catch(Exception ex) {
>>>>> code.fail(ex);
>>>>> }
>>>>> }, result -> {
>>>>> if(result.succeeded()) {
>>>>> promise.tryComplete(result.result());
>>>>> } else {
>>>>

Re: Operation block on Cluster recovery/rebalance.

[Denis Magda]

My guess that it's standard behavior for all operations (SQL, key-value,
compute, etc.). But I'll let the maintainers of those modules clarify.

-
Denis


On Fri, Aug 14, 2020 at 1:44 PM John Smith  wrote:

> Hi Denis, so to understand it's all operations or just the query?
>
> On Fri., Aug. 14, 2020, 12:53 p.m. Denis Magda,  wrote:
>
>> John,
>>
>> Ok, we nailed it. That's the current expected behavior. Generally, I
>> agree with you that the platform should support an option when operations
>> fail if the cluster is deactivated. Could you propose the change by
>> starting a discussion on the dev list? You can refer to this user list
>> discussion for reference. Let me know if you need help with this.
>>
>> -
>> Denis
>>
>>
>> On Thu, Aug 13, 2020 at 5:55 PM John Smith 
>> wrote:
>>
>>> No I, reuse the instance. The cache instance is created once at startup
>>> of the application and I pass it to my "repository" class
>>>
>>> public abstract class AbstractIgniteRepository implements 
>>> CacheRepository {
>>> public final long DEFAULT_OPERATION_TIMEOUT = 2000;
>>>
>>> private Vertx vertx;
>>> private IgniteCache cache;
>>>
>>> AbstractIgniteRepository(Vertx vertx, IgniteCache cache) {
>>> this.vertx = vertx;
>>> this.cache = cache;
>>> }
>>>
>>> ...
>>>
>>> Future> query(final String sql, final long timeoutMs, 
>>> final Object... args) {
>>> final Promise> promise = Promise.promise();
>>>
>>> vertx.setTimer(timeoutMs, l -> {
>>> promise.tryFail(new TimeoutException("Cache operation did not 
>>> complete within: " + timeoutMs + " Ms.")); // THIS FIRE IF THE BLOE DOESN"T 
>>> COMPLETE IN TIME.
>>> });
>>>
>>> vertx.>executeBlocking(code -> {
>>> SqlFieldsQuery query = new SqlFieldsQuery(sql).setArgs(args);
>>> query.setTimeout((int) timeoutMs, TimeUnit.MILLISECONDS);
>>>
>>>
>>> try (QueryCursor> cursor = cache.query(query)) { // 
>>> <--- BLOCKS HERE.
>>> List rows = new ArrayList<>();
>>> Iterator> iterator = cursor.iterator();
>>>
>>> while(iterator.hasNext()) {
>>> List currentRow = iterator.next();
>>> JsonArray row = new JsonArray();
>>>
>>> currentRow.forEach(o -> row.add(o));
>>>
>>> rows.add(row);
>>>     }
>>>
>>> code.complete(rows);
>>> } catch(Exception ex) {
>>> code.fail(ex);
>>> }
>>> }, result -> {
>>> if(result.succeeded()) {
>>> promise.tryComplete(result.result());
>>> } else {
>>> promise.tryFail(result.cause());
>>> }
>>> });
>>>
>>> return promise.future();
>>> }
>>>
>>> public  T cache() {
>>> return (T) cache;
>>> }
>>> }
>>>
>>>
>>>
>>> On Thu, 13 Aug 2020 at 16:29, Denis Magda  wrote:
>>>
>>>> I've created a simple test and always getting the exception below on an
>>>> attempt to get a reference to an IgniteCache instance in cases when the
>>>> cluster is not activated:
>>>>
>>>> *Exception in thread "main" class org.apache.ignite.IgniteException:
>>>> Can not perform the operation because the cluster is inactive. Note, that
>>>> the cluster is considered inactive by default if Ignite Persistent Store is
>>>> used to let all the nodes join the cluster. To activate the cluster call
>>>> Ignite.active(true)*
>>>>
>>>> Are you trying to get a new IgniteCache reference whenever the client
>>>> reconnects successfully to the cluster? My guts feel that currently, Ignite
>>>> verifies the activation status and generates the exception above whenever
>>>> you're getting a reference to an IgniteCache or IgniteCompute. But once you
>>>> got those references and try to run some operations then those get stuck if
>>>> the cluster is not activated.
>>>> -
>>>>

Re: Change Data Capture Feature

[Denis Magda]

Hello Pavel,

GridGain supports a Kafka Connect integration that supports all
sophisticated CDC scenarios:
https://www.gridgain.com/docs/latest/integrations/kafka/cert-kafka-connect

Debezium is another option. This article covers how to capture changes
with GridGain
Kafka Connect

while
that's not the only option for sure.

Finally, continuous  queries is a way to go as well but you have to keep
track of messages that have already been pushed to a Kafka topic. So, that
last position has to be maintained in Ignite somehow.

-
Denis


On Thu, Aug 13, 2020 at 8:48 AM Pavel Strashkin 
wrote:

> Hello,
>
> Is there a Change Data Capture (CDC) feature in plans to make it possible
> to stream cache updates to Kafka for example?
>
> I've found the Continuous Queries feature, but it's not clear to me
> whether it's possible to use it for implementation. What's missing it seems
> is ability to start from the last position whenever the client restarts.
>
> Thanks.
>

Re: Operation block on Cluster recovery/rebalance.

[Denis Magda]

John,

Ok, we nailed it. That's the current expected behavior. Generally, I agree
with you that the platform should support an option when operations fail if
the cluster is deactivated. Could you propose the change by starting a
discussion on the dev list? You can refer to this user list discussion for
reference. Let me know if you need help with this.

-
Denis


On Thu, Aug 13, 2020 at 5:55 PM John Smith  wrote:

> No I, reuse the instance. The cache instance is created once at startup of
> the application and I pass it to my "repository" class
>
> public abstract class AbstractIgniteRepository implements 
> CacheRepository {
> public final long DEFAULT_OPERATION_TIMEOUT = 2000;
>
> private Vertx vertx;
> private IgniteCache cache;
>
> AbstractIgniteRepository(Vertx vertx, IgniteCache cache) {
> this.vertx = vertx;
> this.cache = cache;
> }
>
> ...
>
> Future> query(final String sql, final long timeoutMs, 
> final Object... args) {
> final Promise> promise = Promise.promise();
>
> vertx.setTimer(timeoutMs, l -> {
> promise.tryFail(new TimeoutException("Cache operation did not 
> complete within: " + timeoutMs + " Ms.")); // THIS FIRE IF THE BLOE DOESN"T 
> COMPLETE IN TIME.
> });
>
> vertx.>executeBlocking(code -> {
> SqlFieldsQuery query = new SqlFieldsQuery(sql).setArgs(args);
> query.setTimeout((int) timeoutMs, TimeUnit.MILLISECONDS);
>
>
> try (QueryCursor> cursor = cache.query(query)) { // <--- 
> BLOCKS HERE.
> List rows = new ArrayList<>();
> Iterator> iterator = cursor.iterator();
>
> while(iterator.hasNext()) {
> List currentRow = iterator.next();
> JsonArray row = new JsonArray();
>
> currentRow.forEach(o -> row.add(o));
>
> rows.add(row);
> }
>
> code.complete(rows);
> } catch(Exception ex) {
> code.fail(ex);
> }
> }, result -> {
> if(result.succeeded()) {
> promise.tryComplete(result.result());
> } else {
>     promise.tryFail(result.cause());
> }
> });
>
> return promise.future();
> }
>
> public  T cache() {
> return (T) cache;
> }
> }
>
>
>
> On Thu, 13 Aug 2020 at 16:29, Denis Magda  wrote:
>
>> I've created a simple test and always getting the exception below on an
>> attempt to get a reference to an IgniteCache instance in cases when the
>> cluster is not activated:
>>
>> *Exception in thread "main" class org.apache.ignite.IgniteException: Can
>> not perform the operation because the cluster is inactive. Note, that the
>> cluster is considered inactive by default if Ignite Persistent Store is
>> used to let all the nodes join the cluster. To activate the cluster call
>> Ignite.active(true)*
>>
>> Are you trying to get a new IgniteCache reference whenever the client
>> reconnects successfully to the cluster? My guts feel that currently, Ignite
>> verifies the activation status and generates the exception above whenever
>> you're getting a reference to an IgniteCache or IgniteCompute. But once you
>> got those references and try to run some operations then those get stuck if
>> the cluster is not activated.
>> -
>> Denis
>>
>>
>> On Thu, Aug 13, 2020 at 6:37 AM John Smith 
>> wrote:
>>
>>> The cache.query() starts to block when ignite server nodes are being
>>> restarted and there's no baseline topology yet. The server nodes do not
>>> block. It's the client that blocks.
>>>
>>> The dumpfiles are of the server nodes. The screen shot is from the
>>> client app using your kit profiler on the client side the threads are
>>> marked as red on your kit.
>>>
>>> The app is simple, make http request, it runs cache Sql query on ignite
>>> and if it succeeds does a put back to ignite.
>>>
>>> The Client disconnected exception only happens when all server nodes in
>>> the cluster are down. The blockage only happens when the cluster is trying
>>> to establish baseline topology.
>>>
>>> On Wed., Aug. 12, 2020, 6:28 p.m. Denis Magda, 
>>> wrote:
>>>
>>>> John,
>>>>
>>>> I don't see any traits of an application-caused deadlock in the thread
>>>> dum

Re: How to use Ignite Spring Data to query selected fields from Ignite ?

[Denis Magda]

Hi,

Check this section of the Ignite + Spring Data tutorial:
https://www.gridgain.com/docs/tutorials/spring/spring_ignite_tutorial#create-spring-data-repositories

-
Denis


On Wed, Aug 12, 2020 at 6:52 PM 1478814670 <1478814...@qq.com> wrote:

> Hi,
> Use Ignite Spring Data module to query some selected fields from a
> table or join query from tables, the return value is just the first field
> of the record。
> Sample code:
>  
>  public interface ObjectARepository extends IgniteRepository String> {
>   @Query(select a0, a1 from TableA where a2='xx')
>   public List findObjectA(String a2);
>
>   @Query(select a.a0, a.a1, b.b1, b.b2  from TableA as a ,TableB
> as b where a2=b2)
>   public List findData();
>  }
> -
>How to get the expected fields using the Spring Data module?
>
>Thanks!
>

Re: 2.8.1 - Loading Plugin Provider - Conflicting documentation

[Denis Magda]

Veena, thanks for the ticket! We’ll take care of it.


On Thursday, August 13, 2020, VeenaMithare  wrote:

> HI ,
>
> Raised a Documentation Improvement jira :
> IGNITE-13356
> Documentation Change needed: PluginProvider loading changed from 2.8.1
> regards,
> Veena.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>


-- 
-
Denis

Re: Enabling swapPath causes invoking shutdown hook

[Denis Magda]

If Ignite can hold all the records in memory it won't be reading pages from
disk (the page replacement). Thus, it's expected that the 12GB performs
better than the 5GB case.

Btw, are you planning to reload the whole data set into Ignite on potential
cluster restarts? Each node loses a subset of the data located in the swap
space on restarts.

-
Denis


On Thu, Aug 13, 2020 at 7:36 PM 38797715 <38797...@qq.com> wrote:

> Hi Denis,
>
> We did a test, in the same environment (8G RAM, 10G swap partition) and
> the same configuration (2G Heap, enable persistence, data volume is about
> 6G), the only difference is that the maxSize size is different, which is
> configured as 5G and 12G respectively. We found that the performance of the
> scenario with maxSize = 12G is better than that of the scenario with
> maxSize = 5G, and the write performance is improved by more than 10%.
>
> I suspect that if the memory data region is large enough to hold all the
> data, ignite's page replacement might not be enabled.
>
> Our test scenarios are limited and may not be convincing. However, I think
> that the lack of memory may be the norm. At this time, it may be a good
> practice to make full use of the swap mechanism of the OS, which takes up
> more disk space but achieves better performance.
> 在 2020/8/14 上午8:22, Denis Magda 写道:
>
> Ignite swapping is based on the swapping mechanism of the OS. So, you
> shouldn’t see any difference if enable the OS one directly some way.
>
> Generally, you should not use swapping of any form as a permanent
> persistence layer due to the performance penalty. Once the swapping kicks
> in, you should scale out your cluster and wait while the cluster rebalances
> a part of the data to a new node. When the rebalancing completes, the
> performance will be recovered and swapping won’t longer be needed.
>
> Denis
>
> On Thursday, August 13, 2020, 38797715 <38797...@qq.com> wrote:
>
>> Hi,
>>
>> We retested and found that if we configured swapPath, as the amount of
>> data increased, the write speed was actually slower and slower. If the
>> amount of data is large, on average, it is much slower than the scenario
>> where native persistence is enabled and wal is disabled.
>>
>> In this way, the use of the swapPath property has no productive value,
>> maybe it was an early development function, and now it is a bit out of date.
>>
>> What I want to ask is, in the case of small physical memory, turning on
>> persistence, and then configuring a larger maxSize (using the swap
>> mechanism of the OS), is this a solution? In other words, the swap
>> mechanism of the OS and the page replacement of Ignite, which is better?
>> 在 2020/8/6 下午9:23, Ilya Kasnacheev 写道:
>>
>> Hello!
>>
>> I think the performance of swap space should be on par with persistence
>> with disabled WAL.
>>
>> You can submit suggested updates to the documentation if you like.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> ср, 5 авг. 2020 г. в 06:00, 38797715 <38797...@qq.com>:
>>
>>> Hi Ilya,
>>>
>>> If so, there are two ways to implement ignite's swap space:
>>> 1. maxSize > physical memory, which will use the swap mechanism of the
>>> OS, can be used *vm.swappiness* Adjust.
>>> 2. Configure the *swapPath* property, which is implemented by Ignite
>>> itself, is independent of the OS and has no optimization parameters.
>>> There's a choice between these two models, right? Then I think there may
>>> be many problems in the description of the document. I hope you can check
>>> it again:
>>> https://apacheignite.readme.io/docs/swap-space
>>>
>>> After our initial testing, the performance of swap space is much better
>>> than native persistence, so I think this pattern is valuable in some
>>> scenarios.
>>> 在 2020/8/4 下午10:16, Ilya Kasnacheev 写道:
>>>
>>> Hello!
>>>
>>> From the docs:
>>>
>>> To avoid this situation with the swapping capabilities, you need to :
>>>
>>>- Set maxSize = bigger_ than_RAM_size, in which case, the OS will
>>>take care of the swapping.
>>>- Enable swapping by setting the DataRegionConfiguration.swapPath
>>> property.
>>>
>>>
>>> I actually think these are either-or. You should either do the first
>>> (and configure OS swapping) or the second part.
>>>
>>> Having said that, I recommend setting proper Native Persistence instead.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>

Re: Enabling swapPath causes invoking shutdown hook

[Denis Magda]

Ignite swapping is based on the swapping mechanism of the OS. So, you
shouldn’t see any difference if enable the OS one directly some way.

Generally, you should not use swapping of any form as a permanent
persistence layer due to the performance penalty. Once the swapping kicks
in, you should scale out your cluster and wait while the cluster rebalances
a part of the data to a new node. When the rebalancing completes, the
performance will be recovered and swapping won’t longer be needed.

Denis

On Thursday, August 13, 2020, 38797715 <38797...@qq.com> wrote:

> Hi,
>
> We retested and found that if we configured swapPath, as the amount of
> data increased, the write speed was actually slower and slower. If the
> amount of data is large, on average, it is much slower than the scenario
> where native persistence is enabled and wal is disabled.
>
> In this way, the use of the swapPath property has no productive value,
> maybe it was an early development function, and now it is a bit out of date.
>
> What I want to ask is, in the case of small physical memory, turning on
> persistence, and then configuring a larger maxSize (using the swap
> mechanism of the OS), is this a solution? In other words, the swap
> mechanism of the OS and the page replacement of Ignite, which is better?
> 在 2020/8/6 下午9:23, Ilya Kasnacheev 写道:
>
> Hello!
>
> I think the performance of swap space should be on par with persistence
> with disabled WAL.
>
> You can submit suggested updates to the documentation if you like.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 5 авг. 2020 г. в 06:00, 38797715 <38797...@qq.com>:
>
>> Hi Ilya,
>>
>> If so, there are two ways to implement ignite's swap space:
>> 1. maxSize > physical memory, which will use the swap mechanism of the
>> OS, can be used *vm.swappiness* Adjust.
>> 2. Configure the *swapPath* property, which is implemented by Ignite
>> itself, is independent of the OS and has no optimization parameters.
>> There's a choice between these two models, right? Then I think there may
>> be many problems in the description of the document. I hope you can check
>> it again:
>> https://apacheignite.readme.io/docs/swap-space
>>
>> After our initial testing, the performance of swap space is much better
>> than native persistence, so I think this pattern is valuable in some
>> scenarios.
>> 在 2020/8/4 下午10:16, Ilya Kasnacheev 写道:
>>
>> Hello!
>>
>> From the docs:
>>
>> To avoid this situation with the swapping capabilities, you need to :
>>
>>- Set maxSize = bigger_ than_RAM_size, in which case, the OS will
>>take care of the swapping.
>>- Enable swapping by setting the DataRegionConfiguration.swapPath
>> property.
>>
>>
>> I actually think these are either-or. You should either do the first (and
>> configure OS swapping) or the second part.
>>
>> Having said that, I recommend setting proper Native Persistence instead.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> сб, 25 июл. 2020 г. в 04:49, 38797715 <38797...@qq.com>:
>>
>>> Hi,
>>>
>>> https://apacheignite.readme.io/docs/swap-space
>>>
>>> According to the above document, if the physical memory is small, you
>>> can solve this problem by opening the swap space,The specific method is to
>>> configure maxSize to a larger value (i.e. larger than the physical memory),
>>> and the swapPath property needs to be configured.
>>>
>>> But from the test results, the node is terminated.
>>>
>>> I think the correct result should be that even if the amount of data
>>> exceeds the physical memory, the node should still be able to run normally,
>>> but the data is exchanged to the disk.
>>>
>>> I want to know what parameters affect the behavior of this
>>> configuration? *vm.swappiness* or others?
>>> 在 2020/7/24 下午9:55, aealexsandrov 写道:
>>>
>>> Hi,
>>>
>>> Can you please clarify your expectations? You expected that JVM process will
>>> be killed instead of gracefully stopping? What you are going to achieve?
>>>
>>> BR,
>>> Andrei
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>>

-- 
-
Denis

Re: Operation block on Cluster recovery/rebalance.

[Denis Magda]

I've created a simple test and always getting the exception below on an
attempt to get a reference to an IgniteCache instance in cases when the
cluster is not activated:

*Exception in thread "main" class org.apache.ignite.IgniteException: Can
not perform the operation because the cluster is inactive. Note, that the
cluster is considered inactive by default if Ignite Persistent Store is
used to let all the nodes join the cluster. To activate the cluster call
Ignite.active(true)*

Are you trying to get a new IgniteCache reference whenever the client
reconnects successfully to the cluster? My guts feel that currently, Ignite
verifies the activation status and generates the exception above whenever
you're getting a reference to an IgniteCache or IgniteCompute. But once you
got those references and try to run some operations then those get stuck if
the cluster is not activated.
-
Denis


On Thu, Aug 13, 2020 at 6:37 AM John Smith  wrote:

> The cache.query() starts to block when ignite server nodes are being
> restarted and there's no baseline topology yet. The server nodes do not
> block. It's the client that blocks.
>
> The dumpfiles are of the server nodes. The screen shot is from the client
> app using your kit profiler on the client side the threads are marked as
> red on your kit.
>
> The app is simple, make http request, it runs cache Sql query on ignite
> and if it succeeds does a put back to ignite.
>
> The Client disconnected exception only happens when all server nodes in
> the cluster are down. The blockage only happens when the cluster is trying
> to establish baseline topology.
>
> On Wed., Aug. 12, 2020, 6:28 p.m. Denis Magda,  wrote:
>
>> John,
>>
>> I don't see any traits of an application-caused deadlock in the thread
>> dumps. Please elaborate on the following:
>>
>> 7- Restart 1st node, run operation, operation fails with
>>> ClientDisconectedException but application still able to complete it's
>>> request.
>>
>>
>> What's the IP address of the server node the client app uses to join the
>> cluster? If that's not the address of the 1st node, that is already
>> restarted, then the client couldn't join the cluster and it's expected that
>> it fails with the ClientDisconnectedException.
>>
>> 8- Start 2nd node, run operation, from here on all operations just block.
>>
>>
>> Are the operations unblocked and completed successfully when the third
>> node joins the cluster and the cluster gets activated automatically?
>>
>> -
>> Denis
>>
>>
>> On Wed, Aug 12, 2020 at 11:08 AM John Smith 
>> wrote:
>>
>>> Ok Denis here they are...
>>>
>>> 3 nodes and I capture a yourlit screenshot of what it thinks are
>>> deadlocks on the client app.
>>>
>>> https://www.dropbox.com/sh/2cxjkngvx0ubw3b/AADa--HQg-rRsY3RBo2vQeJ9a?dl=0
>>>
>>> On Wed, 12 Aug 2020 at 11:07, John Smith  wrote:
>>>
>>>> Hi Denis. I will asap but you I think you were right it is the query
>>>> that blocks.
>>>>
>>>> My application first first runs a select on the cache and then does a
>>>> put to cache.
>>>>
>>>> On Tue, 11 Aug 2020 at 19:22, Denis Magda  wrote:
>>>>
>>>>> John,
>>>>>
>>>>> It sounds like a deadlock caused by the application logic. Is there
>>>>> any chance that the operation you run on step 8 accesses several keys in
>>>>> one order while the other operations work with the same keys but in a
>>>>> different order. The deadlocks are possible when you use Ignite 
>>>>> Transaction
>>>>> API or simply execute bulk operations such as cache.readAll() or
>>>>> cache.writeAll(..).
>>>>>
>>>>> Please take and attach thread dumps from all the cluster nodes for
>>>>> analysis if we need to dig deeper.
>>>>>
>>>>> -
>>>>> Denis
>>>>>
>>>>>
>>>>> On Mon, Aug 10, 2020 at 6:23 PM John Smith 
>>>>> wrote:
>>>>>
>>>>>> Hi Denis, I think you are right. It's the query that blocks the other
>>>>>> k/v operations are ok.
>>>>>>
>>>>>> Any thoughts on this?
>>>>>>
>>>>>> On Mon, 10 Aug 2020 at 15:28, John Smith 
>>>>>> wrote:
>>>>>>
>>>>>>> I tried with 2.8.1, same issue. Operations block indefinitely...
>>>>>>>
>>>>>>> 1- Start 3 node 

Re: Operation block on Cluster recovery/rebalance.

[Denis Magda]

John,

I don't see any traits of an application-caused deadlock in the thread
dumps. Please elaborate on the following:

7- Restart 1st node, run operation, operation fails with
> ClientDisconectedException but application still able to complete it's
> request.


What's the IP address of the server node the client app uses to join the
cluster? If that's not the address of the 1st node, that is already
restarted, then the client couldn't join the cluster and it's expected that
it fails with the ClientDisconnectedException.

8- Start 2nd node, run operation, from here on all operations just block.


Are the operations unblocked and completed successfully when the third node
joins the cluster and the cluster gets activated automatically?

-
Denis


On Wed, Aug 12, 2020 at 11:08 AM John Smith  wrote:

> Ok Denis here they are...
>
> 3 nodes and I capture a yourlit screenshot of what it thinks are deadlocks
> on the client app.
>
> https://www.dropbox.com/sh/2cxjkngvx0ubw3b/AADa--HQg-rRsY3RBo2vQeJ9a?dl=0
>
> On Wed, 12 Aug 2020 at 11:07, John Smith  wrote:
>
>> Hi Denis. I will asap but you I think you were right it is the query that
>> blocks.
>>
>> My application first first runs a select on the cache and then does a put
>> to cache.
>>
>> On Tue, 11 Aug 2020 at 19:22, Denis Magda  wrote:
>>
>>> John,
>>>
>>> It sounds like a deadlock caused by the application logic. Is there any
>>> chance that the operation you run on step 8 accesses several keys in one
>>> order while the other operations work with the same keys but in a different
>>> order. The deadlocks are possible when you use Ignite Transaction API or
>>> simply execute bulk operations such as cache.readAll() or
>>> cache.writeAll(..).
>>>
>>> Please take and attach thread dumps from all the cluster nodes for
>>> analysis if we need to dig deeper.
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Mon, Aug 10, 2020 at 6:23 PM John Smith 
>>> wrote:
>>>
>>>> Hi Denis, I think you are right. It's the query that blocks the other
>>>> k/v operations are ok.
>>>>
>>>> Any thoughts on this?
>>>>
>>>> On Mon, 10 Aug 2020 at 15:28, John Smith 
>>>> wrote:
>>>>
>>>>> I tried with 2.8.1, same issue. Operations block indefinitely...
>>>>>
>>>>> 1- Start 3 node cluster
>>>>> 2- Start client application client = true with Ignition.start()
>>>>> 3- Run some cache operations, everything ok...
>>>>> 4- Shut down one node, run operation, still ok
>>>>> 5- Shut down 2nd node, run operation, still ok
>>>>> 6- Shut down 3rd node, run operation, still ok... Operations start
>>>>> failing with ClientDisconectedException...
>>>>> 7- Restart 1st node, run operation, operation fails
>>>>> with ClientDisconectedException but application still able to complete 
>>>>> it's
>>>>> request.
>>>>> 8- Start 2nd node, run operation, from here on all operations just
>>>>> block.
>>>>>
>>>>> Basically the client application is an HTTP Server on each HTTP
>>>>> request does cache exception.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, 7 Aug 2020 at 19:46, John Smith 
>>>>> wrote:
>>>>>
>>>>>> No, everything blocks... Also using 2.7.0 just in case.
>>>>>>
>>>>>> Only time I get exception is if the cluster is completely off, then I
>>>>>> get ClientDisconectedException...
>>>>>>
>>>>>> On Fri, 7 Aug 2020 at 18:52, Denis Magda  wrote:
>>>>>>
>>>>>>> If I'm not mistaken, key-value operations (cache.get/put) and
>>>>>>> compute calls fail with an exception if the cluster is deactivated. Do
>>>>>>> those fail on your end?
>>>>>>>
>>>>>>> As for the async and SQL operations, let's see what other community
>>>>>>> members say.
>>>>>>>
>>>>>>> -
>>>>>>> Denis
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Aug 7, 2020 at 1:06 PM John Smith 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi any thoughts on this?
>>>>>>>>
>>>>>>>> On Thu, 6 Aug 2020 at 23:33, John Smith 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Here is another example where it blocks.
>>>>>>>>>
>>>>>>>>> SqlFieldsQuery query = new SqlFieldsQuery(
>>>>>>>>> "select * from my_table")
>>>>>>>>> .setArgs(providerId, carrierCode);
>>>>>>>>> query.setTimeout(1000, TimeUnit.MILLISECONDS);
>>>>>>>>>
>>>>>>>>> try (QueryCursor> cursor = cache.query(query))
>>>>>>>>>
>>>>>>>>> cache.query just blocks even with the timeout set.
>>>>>>>>>
>>>>>>>>> Is there a way to timeout and at least have the application
>>>>>>>>> continue and respond with an appropriate message?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, 6 Aug 2020 at 23:06, John Smith 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi running 2.7.0
>>>>>>>>>>
>>>>>>>>>> When I reboot a node and it begins to rejoin the cluster or the
>>>>>>>>>> cluster is not yet activated with baseline topology operations seem 
>>>>>>>>>> to
>>>>>>>>>> block forever, operations that are supposed to return IgniteFuture. 
>>>>>>>>>> I.e:
>>>>>>>>>> putAsync, getAsync etc... They just block, until the cluster 
>>>>>>>>>> resolves it's
>>>>>>>>>> state.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>

Re: Operation block on Cluster recovery/rebalance.

[Denis Magda]

John,

It sounds like a deadlock caused by the application logic. Is there any
chance that the operation you run on step 8 accesses several keys in one
order while the other operations work with the same keys but in a different
order. The deadlocks are possible when you use Ignite Transaction API or
simply execute bulk operations such as cache.readAll() or
cache.writeAll(..).

Please take and attach thread dumps from all the cluster nodes for analysis
if we need to dig deeper.

-
Denis


On Mon, Aug 10, 2020 at 6:23 PM John Smith  wrote:

> Hi Denis, I think you are right. It's the query that blocks the other k/v
> operations are ok.
>
> Any thoughts on this?
>
> On Mon, 10 Aug 2020 at 15:28, John Smith  wrote:
>
>> I tried with 2.8.1, same issue. Operations block indefinitely...
>>
>> 1- Start 3 node cluster
>> 2- Start client application client = true with Ignition.start()
>> 3- Run some cache operations, everything ok...
>> 4- Shut down one node, run operation, still ok
>> 5- Shut down 2nd node, run operation, still ok
>> 6- Shut down 3rd node, run operation, still ok... Operations start
>> failing with ClientDisconectedException...
>> 7- Restart 1st node, run operation, operation fails
>> with ClientDisconectedException but application still able to complete it's
>> request.
>> 8- Start 2nd node, run operation, from here on all operations just block.
>>
>> Basically the client application is an HTTP Server on each HTTP request
>> does cache exception.
>>
>>
>>
>>
>>
>>
>> On Fri, 7 Aug 2020 at 19:46, John Smith  wrote:
>>
>>> No, everything blocks... Also using 2.7.0 just in case.
>>>
>>> Only time I get exception is if the cluster is completely off, then I
>>> get ClientDisconectedException...
>>>
>>> On Fri, 7 Aug 2020 at 18:52, Denis Magda  wrote:
>>>
>>>> If I'm not mistaken, key-value operations (cache.get/put) and compute
>>>> calls fail with an exception if the cluster is deactivated. Do those fail
>>>> on your end?
>>>>
>>>> As for the async and SQL operations, let's see what other community
>>>> members say.
>>>>
>>>> -
>>>> Denis
>>>>
>>>>
>>>> On Fri, Aug 7, 2020 at 1:06 PM John Smith 
>>>> wrote:
>>>>
>>>>> Hi any thoughts on this?
>>>>>
>>>>> On Thu, 6 Aug 2020 at 23:33, John Smith 
>>>>> wrote:
>>>>>
>>>>>> Here is another example where it blocks.
>>>>>>
>>>>>> SqlFieldsQuery query = new SqlFieldsQuery(
>>>>>> "select * from my_table")
>>>>>> .setArgs(providerId, carrierCode);
>>>>>> query.setTimeout(1000, TimeUnit.MILLISECONDS);
>>>>>>
>>>>>> try (QueryCursor> cursor = cache.query(query))
>>>>>>
>>>>>> cache.query just blocks even with the timeout set.
>>>>>>
>>>>>> Is there a way to timeout and at least have the application continue
>>>>>> and respond with an appropriate message?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, 6 Aug 2020 at 23:06, John Smith 
>>>>>> wrote:
>>>>>>
>>>>>>> Hi running 2.7.0
>>>>>>>
>>>>>>> When I reboot a node and it begins to rejoin the cluster or the
>>>>>>> cluster is not yet activated with baseline topology operations seem to
>>>>>>> block forever, operations that are supposed to return IgniteFuture. I.e:
>>>>>>> putAsync, getAsync etc... They just block, until the cluster resolves 
>>>>>>> it's
>>>>>>> state.
>>>>>>>
>>>>>>>
>>>>>>>

Re: [BLOG] Apache Ignite Extensions

[Denis Magda]

Hi Saikat,

The weekly routines of an Apache Ignite committer :) I believe that Ignite
application developers would love to learn insights from this article and
eventually get modular-based Ignite integrations. So, with(out) your
permission sharing the article via @user list.

Excellent article and the summary of this undergoing project. Keep it up!

-
Denis


On Sat, Aug 8, 2020 at 1:37 PM Saikat Maitra 
wrote:

> Hi,
>
> I have published a blog on Apache Ignite Extensions and about migration of
> existing integration modules to new Ignite Extensions repository.
>
> https://samaitra.blogspot.com/2020/08/apache-ignite-extensions.html
>
> Please review and share if you have any feedback.
>
> Regards,
> Saikat
>

Re: What does all partition owners have left the grid, partition data has been lost mean?

[Denis Magda]

John,

This page covers all the aspects of backups in Ignite (what are those, how
to configure, etc.):
https://www.gridgain.com/docs/latest/developers-guide/configuring-caches/configuring-backups

As for the 5 nodes cluster with the persistence, yes, all those have to be
added to be in the same baseline topology.

-
Denis


On Thu, Aug 6, 2020 at 8:26 AM John Smith  wrote:

> Ok if I have 5 nodes with persistence then all nodes need to be in
> baseline?
>
> Also what are the docs for backup to make sure I have it right?
>
>
> On Thu, 6 Aug 2020 at 10:08, Ilya Kasnacheev 
> wrote:
>
>> Hello!
>>
>> You are confusing baseline with backups here.
>>
>> You should have 1 backup to afford losing a node.
>>
>> You should have all data nodes in the baseline.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> ср, 5 авг. 2020 г. в 17:56, John Smith :
>>
>>> I mean I have 3 nodes and the baseline is set to 3. Does it mean if I
>>> put 2 as baseline then I can lose at least 1? If I remove one node from
>>> baseline does it mean it will not store data?
>>>
>>> Or is it better to have 3 baseline nodes and add a 4th node? In that
>>> case if I still lose a baseline node will I still be able to do
>>> operations on the cache?
>>>
>>> On Wed, 5 Aug 2020 at 08:21, John Smith  wrote:
>>>
 I have 3 nodes and baseline topology is 3 so if I lose 1 I guess it's
 enough... Should it be 2?

 On Tue., Aug. 4, 2020, 10:57 a.m. Ilya Kasnacheev, <
 ilya.kasnach...@gmail.com> wrote:

> Hello!
>
> What is your baseline topology at this moment? It means just that: you
> have lost enough nodes of your distributed grid that data is nowhere to be
> found now.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 3 авг. 2020 г. в 19:12, John Smith :
>
>> I get the below exception on my client...
>>
>> #1 I rebooted the cache nodes error still continued.
>> #2 restarted the client node error went away.
>> #3 this seems to happen every few weeks.
>> #4 is there some sort of timeout values and retries I can put?
>> #5 cache operations seem to block when rebooting the nodes (I have 3
>> nodes). Is there a way not to block?
>>
>> javax.cache.CacheException: class
>> org.apache.ignite.internal.processors.cache.CacheInvalidStateException:
>> Failed to execute cache operation (all partition owners have left the 
>> grid,
>> partition data has been lost) [cacheName=xx, part=273, 
>> key=16479796986]
>> at
>> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
>> at
>> org.apache.ignite.internal.processors.cache.IgniteCacheFutureImpl.convertException(IgniteCacheFutureImpl.java:62)
>> at
>> org.apache.ignite.internal.util.future.IgniteFutureImpl.get(IgniteFutureImpl.java:157)
>> at
>> com.xx.common.vertx.ext.data.impl.IgniteCacheRepository.lambda$executeAsync$394d953f$1(IgniteCacheRepository.java:59)
>> at
>> org.apache.ignite.internal.util.future.AsyncFutureListener$1.run(AsyncFutureListener.java:53)
>> at
>> com.xx.common.vertx.ext.data.impl.VertxIgniteExecutorAdapter.lambda$execute$0(VertxIgniteExecutorAdapter.java:18)
>> at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:369)
>> at
>> io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
>> at
>> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
>> at
>> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>> at
>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by:
>> org.apache.ignite.internal.processors.cache.CacheInvalidStateException:
>> Failed to execute cache operation (all partition owners have left the 
>> grid,
>> partition data has been lost) [cacheName=xx, part=273, 
>> key=16479796986]
>> at
>> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTopologyFutureAdapter.validatePartitionOperation(GridDhtTopologyFutureAdapter.java:161)
>> at
>> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTopologyFutureAdapter.validateCache(GridDhtTopologyFutureAdapter.java:116)
>> at
>> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:417)
>> at
>> 

Re: Operation block on Cluster recovery/rebalance.

[Denis Magda]

If I'm not mistaken, key-value operations (cache.get/put) and compute calls
fail with an exception if the cluster is deactivated. Do those fail on your
end?

As for the async and SQL operations, let's see what other community members
say.

-
Denis


On Fri, Aug 7, 2020 at 1:06 PM John Smith  wrote:

> Hi any thoughts on this?
>
> On Thu, 6 Aug 2020 at 23:33, John Smith  wrote:
>
>> Here is another example where it blocks.
>>
>> SqlFieldsQuery query = new SqlFieldsQuery(
>> "select * from my_table")
>> .setArgs(providerId, carrierCode);
>> query.setTimeout(1000, TimeUnit.MILLISECONDS);
>>
>> try (QueryCursor> cursor = cache.query(query))
>>
>> cache.query just blocks even with the timeout set.
>>
>> Is there a way to timeout and at least have the application continue and
>> respond with an appropriate message?
>>
>>
>>
>> On Thu, 6 Aug 2020 at 23:06, John Smith  wrote:
>>
>>> Hi running 2.7.0
>>>
>>> When I reboot a node and it begins to rejoin the cluster or the cluster
>>> is not yet activated with baseline topology operations seem to block
>>> forever, operations that are supposed to return IgniteFuture. I.e:
>>> putAsync, getAsync etc... They just block, until the cluster resolves it's
>>> state.
>>>
>>>
>>>

Re: Enabling swapPath causes invoking shutdown hook

[Denis Magda]

Agree with Ilya that the performance should be comparable if you disable
the WAL of the Ignite persistence.

Anyway, the swapping and Ignite persistence pursue different goals. The
swapping is one of the out-of-memory protection techniques - if you run out
of DRAM, then the OS will start swapping in/out Ignite memory pages to
avoid a node outage. But, the swap space is not a durable storage layer. If
you restart the cluster, then all the swapped pages will evaporate. While,
Ignite persistence is your durable disk tier that survives cluster restarts
and, thus, uses more sophisticated algorithms to ensure data consistency
and durability. Just select what suits you best. I put more thoughts on
this in this article:
https://www.gridgain.com/resources/blog/out-of-memory-apache-ignite-cluster-handling-techniques

-
Denis


On Thu, Aug 6, 2020 at 6:23 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> I think the performance of swap space should be on par with persistence
> with disabled WAL.
>
> You can submit suggested updates to the documentation if you like.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 5 авг. 2020 г. в 06:00, 38797715 <38797...@qq.com>:
>
>> Hi Ilya,
>>
>> If so, there are two ways to implement ignite's swap space:
>> 1. maxSize > physical memory, which will use the swap mechanism of the
>> OS, can be used *vm.swappiness* Adjust.
>> 2. Configure the *swapPath* property, which is implemented by Ignite
>> itself, is independent of the OS and has no optimization parameters.
>> There's a choice between these two models, right? Then I think there may
>> be many problems in the description of the document. I hope you can check
>> it again:
>> https://apacheignite.readme.io/docs/swap-space
>>
>> After our initial testing, the performance of swap space is much better
>> than native persistence, so I think this pattern is valuable in some
>> scenarios.
>> 在 2020/8/4 下午10:16, Ilya Kasnacheev 写道:
>>
>> Hello!
>>
>> From the docs:
>>
>> To avoid this situation with the swapping capabilities, you need to :
>>
>>- Set maxSize = bigger_ than_RAM_size, in which case, the OS will
>>take care of the swapping.
>>- Enable swapping by setting the DataRegionConfiguration.swapPath
>> property.
>>
>>
>> I actually think these are either-or. You should either do the first (and
>> configure OS swapping) or the second part.
>>
>> Having said that, I recommend setting proper Native Persistence instead.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> сб, 25 июл. 2020 г. в 04:49, 38797715 <38797...@qq.com>:
>>
>>> Hi,
>>>
>>> https://apacheignite.readme.io/docs/swap-space
>>>
>>> According to the above document, if the physical memory is small, you
>>> can solve this problem by opening the swap space,The specific method is to
>>> configure maxSize to a larger value (i.e. larger than the physical memory),
>>> and the swapPath property needs to be configured.
>>>
>>> But from the test results, the node is terminated.
>>>
>>> I think the correct result should be that even if the amount of data
>>> exceeds the physical memory, the node should still be able to run normally,
>>> but the data is exchanged to the disk.
>>>
>>> I want to know what parameters affect the behavior of this
>>> configuration? *vm.swappiness* or others?
>>> 在 2020/7/24 下午9:55, aealexsandrov 写道:
>>>
>>> Hi,
>>>
>>> Can you please clarify your expectations? You expected that JVM process will
>>> be killed instead of gracefully stopping? What you are going to achieve?
>>>
>>> BR,
>>> Andrei
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>>

Re: Best first steps for performance tuning

[Denis Magda]

Devin,

Please consider watching this talk by Ivan Rakov, one of main contributors
to the native persistence. Rewind to 28 minutes 14 seconds for performance
consideration: https://www.youtube.com/watch?v=f2ArcJPH4iU

Let us know if any piece of advice was helpful and we’ll put the knowledge
on paper then for better discoverability.

Denis

On Thursday, August 6, 2020, Evgenii Zhuravlev 
wrote:

> Hi Devin,
>
> Yes, you're right, the first step could be increasing the amount of
> offheap memory used for data(data region size). By default, Ignite uses 20%
> of available RAM.
>
> After that, I would recommend finding where the bottleneck is for your
> system - you can check CPU, disk and network to find it.
>
> Best Regards,
> Evgenii
>
> чт, 6 авг. 2020 г. в 12:49, Devin Bost :
>
>> While looking at the docs, there are a lot of available parameters for
>> performance tuning.
>> We have several high-velocity Ignite operations, and Ignite is having
>> trouble keeping up. We are using native persistence, and I suspect we will
>> get more value from increasing the amount of memory used since I think the
>> default memory utilization is low if I remember correctly.
>>
>> We're wondering what the first few things should be for us to look at for
>> performance tuning and wondering if anyone has some guidance so we know
>> what to start with.
>>
>> Thanks,
>>
>> Devin G. Bost
>>
>

-- 
-
Denis

Re: read-though tutorial for a big table

[Denis Magda]

Alex,

Please share a bit more details on what you’re struggling with. It sounds
like you are looking for a specific piece of advice rather than generic
performance suggestions.

Denis

On Thursday, August 6, 2020, Alex Panchenko 
wrote:

> Hello Vladimir,
>
> are there some key things you can share with us? Some checklist with the
> most important configuration params or things we need to review/check?
> anything would be helpful
>
> I've been playing with Ignite for the last few months, performance still
> low.
> I have to decide whether to switch from Ignite to some another solution or
> improve the performance ASAP.
>
> Thanks
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>


-- 
-
Denis

Re: run ignitevisorcmd in k8s and docker？

[Denis Magda]

Hi,

Then Visor CMD should work fine. Please share your configs and logs for
analysis.

As for the alternate UI tools, yes, you can deploy those in Docker. Just
google for respective documentation.

Denis

On Thursday, August 6, 2020, bbweb  wrote:

> Hi Denis, Actaully we are running visor CMD in the same container with
> Ignite server, and using the same configuration file with
> KubernetesIPFinder that ignite server use. Should this mode be supported?
> And for the link provided, does it mean that web console can also be used?
> Thanks! - Bright
>
> - 原始邮件 -
>
> *发件人：*Denis Magda
>
> *发送时间：*2020-08-07 11:53:11
>
> *收件人：*user
>
> *主 题：*Re: run ignitevisorcmd in k8s and docker？
>
> Hi there,
>
> Visor CMD uses a legacy connection mode (daemon node) that might be tricky
> to deal with in K8S environments. Try to start it inside the same K8S
> network with your cluster and use a configuration with KubernetesIPFinder
> you used for the server nodes.
>
> Alternatively, go for an alternate monitoring tool that is better suited
> for K8S:
> https://ignite.apache.org/features/management-and-monitoring.html
>
> -
> Denis
>
>
> On Thu, Aug 6, 2020 at 6:43 PM bbweb < bb...@sohu.com> wrote:
>
> Hi,
>
> we aremeeting problem when we run ignitevisorcmd in K8S and docker
> environment. Afterwe start a cluster in K8S and run ignitevisorcmd in the
> node, it just can'tfind any node when running "top" in it, it just show
> empty topology.
>
> Is thissupported in K8S and docker? Or are there some configurations
> should be done?
>
> Thanks for any response!
>
>
>
>
>
> <https://a.app.qq.com/o/simple.jsp?pkgname=com.sohu.mail.client.cordova>
>
>
>
>
>
>
> <https://a.app.qq.com/o/simple.jsp?pkgname=com.sohu.mail.client.cordova>
>
>
>
>

-- 
-
Denis

Re: run ignitevisorcmd in k8s and docker？

[Denis Magda]

Hi there,

Visor CMD uses a legacy connection mode (daemon node) that might be tricky
to deal with in K8S environments. Try to start it inside the same K8S
network with your cluster and use a configuration with KubernetesIPFinder
you used for the server nodes.

Alternatively, go for an alternate monitoring tool that is better suited
for K8S:
https://ignite.apache.org/features/management-and-monitoring.html

-
Denis


On Thu, Aug 6, 2020 at 6:43 PM bbweb  wrote:

> Hi,
>
> we are meeting problem when we run ignitevisorcmd in K8S and docker
> environment. After we start a cluster in K8S and run ignitevisorcmd in the
> node, it just can't find any node when running "top" in it, it just show
> empty topology.
>
> Is this supported in K8S and docker? Or are there some configurations
> should be done?
>
> Thanks for any response!
>
>
>
>
>
> 
>
>
>
>

Re: Call for presentations for ApacheCon North America 2020 now open

[Denis Magda]

Wesley,

Definitely follow the advice of Saikat and submit the talk to the IMC
Summit.

Also, we’ll be happy to host you for Ignite virtual meetup. Many community
folks will be glad to learn how you use Ignite ML in production. Are you
ready?
https://www.meetup.com/Apache-Ignite-Virtual-Meetup/

Denis

On Wednesday, August 5, 2020, Wesley Peng  wrote:

> Congrats. We could prepare a talking for "machine learning application
> with Ignite", as we did store feature engineering data into ignite for
> large-scale and fast access.
>
> regards.
>
>
> Saikat Maitra wrote:
>
>> Congrats!!!
>>
>> It looks like both of our talks are on same day, Tuesday, September 29th
>>
>> https://apachecon.com/acah2020/tracks/ignite.html
>>
>

-- 
-
Denis

Re: Call for presentations for ApacheCon North America 2020 now open

[Denis Magda]

Congrats, Saikat! I received a similar message that my talk (In-memory
computing essentials for software engineers) was accepted as well. So, at
least two talks by the Ignite folks.

Once you share the time your presentation is scheduled for, I'll go ahead
and update on the events' page on the website.
https://ignite.apache.org/events.html

-
Denis


On Tue, Aug 4, 2020 at 6:40 PM Saikat Maitra 
wrote:

> Hi,
>
> I learned that my proposal for talk about Data Streaming using Apache
> Ignite
> and Apache Flink has been accepted.
>
> I have not yet received the schedule yet. I will share as soon as I have
> it.
>
> Regards,
> Saikat
>
> On Wed, Jan 22, 2020 at 8:09 PM Saikat Maitra 
> wrote:
>
> > Hi Denis,
> >
> > Thank you for your email. I have submitted a talk about Data Streaming
> > using Apache Ignite and Apache Flink.
> >
> > I would also like to co-present on Ignite internals and usecases deep
> dive.
> >
> > Regards,
> > Saikat
> >
> > On Wed, Jan 22, 2020 at 6:22 PM Denis Magda  wrote:
> >
> >> Igniters,
> >>
> >> I was submitting an Ignite talk to the conference and found out that
> this
> >> time ASF created a separate category for Ignite-specific proposals. You
> can
> >> see an abstract submitted by me:
> >>
> https://drive.google.com/file/d/1woaEOWaIFxN8UIJ7nvFbYoc53mYsUsSN/view?usp=sharing
> >>
> >> Who else is ready to submit? It can be a deep-dive about Ignite
> internals
> >> or Ignite use case overview. We can co-present. Also, GridGain is ready
> to
> >> sponsor your trip if the talk is accepted.
> >>
> >>
> >>
> >> -
> >> Denis
> >>
> >>
> >> On Tue, Jan 21, 2020 at 5:22 PM Denis Magda  wrote:
> >>
> >>> Ignite folks, just bringing to your attention and encourage anybody
> >>> interested to submit an Ignite-related session. Both Alex Zinoviev and
> I
> >>> presented at ApacheCons the previous year -- the events are worth
> attending
> >>> and speaking at.
> >>>
> >>> -
> >>> Denis
> >>>
> >>>
> >>> -- Forwarded message -
> >>> From: Rich Bowen 
> >>> Date: Tue, Jan 21, 2020 at 7:06 AM
> >>> Subject: Call for presentations for ApacheCon North America 2020 now
> open
> >>> To: plann...@apachecon.com 
> >>>
> >>>
> >>> Dear Apache enthusiast,
> >>>
> >>> (You’re receiving this message because you are subscribed to one or
> more
> >>> project mailing lists at the Apache Software Foundation.)
> >>>
> >>> The call for presentations for ApacheCon North America 2020 is now open
> >>> at https://apachecon.com/acna2020/cfp
> >>>
> >>> ApacheCon will be held at the Sheraton, New Orleans, September 28th
> >>> through October 2nd, 2020.
> >>>
> >>> As in past years, ApacheCon will feature tracks focusing on the various
> >>> technologies within the Apache ecosystem, and so the call for
> >>> presentations will ask you to select one of those tracks, or “General”
> >>> if the content falls outside of one of our already-organized tracks.
> >>> These tracks are:
> >>>
> >>> Karaf
> >>> Internet of Things
> >>> Fineract
> >>> Community
> >>> Content Delivery
> >>> Solr/Lucene (Search)
> >>> Gobblin/Big Data Integration
> >>> Ignite
> >>> Observability
> >>> Cloudstack
> >>> Geospatial
> >>> Graph
> >>> Camel/Integration
> >>> Flagon
> >>> Tomcat
> >>> Cassandra
> >>> Groovy
> >>> Web/httpd
> >>> General/Other
> >>>
> >>> The CFP will close Friday, May 1, 2020 8:00 AM (America/New_York time).
> >>>
> >>> Submit early, submit often, at https://apachecon.com/acna2020/cfp
> >>>
> >>> Rich, for the ApacheCon Planners
> >>>
> >>
>

Re: Serialize a char array member as part of class serialization into cache - C++ API

[Denis Magda]

Igor,

Would you please join the discussion and help Data tackle the issue?

-
Denis


On Thu, Jul 30, 2020 at 1:03 AM Dana Milan  wrote:

> Hi,
>
> I couldn't find an answer anywhere else, hopefully you can help me.
>
> I have the following class:
>
> class Pair {
> friend struct ignite::binary::BinaryType;
> public:
> Pair() {
> _len = 0;
> _buff = nullptr;
> }
>
> Pair(char* buff, int len) {
> _len = len;
> _buff = new char[len];
> for (int i = 0; i < len; i++) {
> _buff[i] = buff[i];
> }
> }
>
> ~ Pair() {
> delete[] _buff;
> }
>
> private:
> char* _buff;
> int _len;
> };
>
> I try to serialize the class into cache in the following manner:
>
> template<>
> struct ignite::binary::BinaryType
> {
> static int32_t GetTypeId()
> {
> return GetBinaryStringHashCode("Pair");
> }
>
> static void GetTypeName(std::string& name)
> {
> name = "Pair";
> }
>
> static int32_t GetFieldId(const char* name)
> {
> return GetBinaryStringHashCode(name);
> }
>
> static bool IsNull(const Pair& obj)
> {
> return false;
> }
>
> static void GetNull(Pair& dst)
> {
> dst = Pair();
> }
>
> static void Write(BinaryWriter& writer, const Pair& obj)
> {
> BinaryRawWriter rawWriter = writer.RawWriter();
>
> int len = obj._len;
> char* buff = obj._buff;
>
> rawWriter.WriteInt32(len);
>
> auto binWriter = rawWriter.WriteArray();
> for (int i = 0; i < len; i++) {
> binWriter.Write(buff[i]);
> }
> binWriter.Close();
> }
>
> static void Read(BinaryReader& reader, Pair& dst)
> {
> BinaryRawReader rawReader = reader.RawReader();
>
> dst._len = rawReader.ReadInt32();
>
> dst._buff = new char[dst._len];
> auto binReader = rawReader.ReadArray();
> for (int i = 0; i < dst._len; i++) {
> dst._buff[i] = binReader.GetNext();
> }
> }
> };
>
> When I try to compile I get errors as following:
> [image: image.png]
> If I comment out the parts of reading and writing the array it compiles
> successfully.
>
> Does this happen because I also need to serialize the write of 'char'? (If
> so, how do I do it?)
> Am I using the ReadArray and WriteArray correctly?
> Is there another way of storing the char buffer in cache?
>
> If someone can provide a working code snippet that would be amazing, but
> any help would be appreciated.
> Thanks a lot!
>
>

Re: Ignite Web Console sign up. Not found.

[Denis Magda]

The Ignite community stopped supporting Ignite Web Console some time ago
and recently made a decision to discontinue the tool officially:
https://issues.apache.org/jira/browse/IGNITE-13038

Please select an alternate tool. We've put together a list of potential
candidates:
https://ignite.apache.org/features/management-and-monitoring.html

-
Denis


On Sat, Aug 1, 2020 at 5:46 PM chp  wrote:

> Hello,
>
> trying to sign up with stand-alone installed web console. And receive
> unclear error Not found. Could it possible to insert first user into DB
> without sign up?
>
> Regards,
> chp
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Effective Monitoring of Rebalancing

[Denis Magda]

John,

If it's still relevant for you, now you can monitor rebalancing progress
with a special widget of Control Center:
https://www.gridgain.com/docs/control-center/latest/monitoring/configuring-widgets#rebalance-widget

-
Denis


On Fri, Jan 17, 2020 at 12:33 AM John Gardner 
wrote:

> I've found a similar request from Evgenii Zhuravlev in December 2019, and
> some solutions, but I was hoping that these metrics were available via the
> REST API?
>
> John
>
> On Fri, 17 Jan 2020 at 08:19, John Gardner 
> wrote:
>
>> Hi everyone
>>
>> We've been using Ignite (actually GridGain) in AWS for a couple of years,
>> but in a very basic way.  Now we're in a position to use a lot more of the
>> functionality we need to start monitoring valid metrics more and
>> subsequently alert on issues.
>>
>> We're trying to automate our processes as much as possible using CI and
>> CodeDeploy, but we really need to understand when the entire cluster is
>> rebalanced successfully, because only then can we trigger further
>> autonomous operations.
>>
>> What tricks and techniques is everybody using for this?
>>
>> Thanks in advance.
>>
>> John
>>
>>
>
> --
>
> 
> John Gardner
> Infrastructure Manager
>
> W: www.salecycle.com 
> [image: Email Marketing in a Privacy Conscious World - Download Report]
> 
>

Re: Work Directory Usage with Persistence Disabled

[Denis Magda]

Hello Mitchell,

Ignite saves there classes of objects which your application puts in caches
or executes as compute tasks. For instance, if you put a MyObject instance
in a cache, then Ignite will persist the info about MyObject in the
marshaller directory and then will be able to deserialize this object
whenever your application reads in from the cache.

-
Denis


On Wed, Jul 29, 2020 at 12:52 PM Mitchell Rathbun (BLOOMBERG/ 731 LEX) <
mrathb...@bloomberg.net> wrote:

> In the documentation, Ignite is described as a pure in-memory solution
> when persistence is disabled. However, even with persistence disabled, I am
> noticing that the Ignite work directory must be set to an existing
> directory, and that this directory has the 'marshaller/' directory created
> with files written to it. Why is this needed if persistence is off? Is
> there anyway to avoid writing to the file system completely?
>

Re: [MEETUP] Apache Ignite Best Practices for Native Persistence and Data Recovery

[Denis Magda]

Vasiliy,

You can find a link to the recording in the comments section here:
https://www.meetup.com/ru-RU/Apache-Ignite-Virtual-Meetup/events/271602652/

-
Denis


On Wed, Jul 29, 2020 at 2:36 AM Vasily Laktionov 
wrote:

> Hi,
> Please share slides and/or video of meetup.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Kubernetes : Connection timed out (Connection timed out)

[Denis Magda]

Steve,

Excellent, glad the problem is gone. The community is working on the new
documentation and, as far as I know, the Kubernetes instructions will be
brought up-to-date. Thanks for pointing that out.

-
Denis


On Wed, Jul 29, 2020 at 2:53 AM steve.hostettler 
wrote:

> Hello Dennis,
>
> so it works again, it was apparently a transient Azure problem when
> creating
> new clusters.
> FYI the documentation and
> https://github.com/apache/ignite/tree/master/modules/kubernetes/config/az
> is not up to date and gave errors when executing them (from a K8S
> perspective)
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Fwd: The Apache(R) Software Foundation Announces Annual Report for 2020 Fiscal Year

[Denis Magda]

Igniters,

The board of directors published the annual report and even though Ignite
is not mentioned in the press release, you can find encouraging
datapoints by looking into the full report:
https://www.apache.org/foundation/docs/FY2020AnnualReport.pdf

Remember that Ignite is compared to other 340+ top-level project. So, as
the community, we've achieved a lot.

Ignite is the 14th by most active sources (visits).
Ignite is the 12 largest project by the lines of code.

Top 5 user lists:

   1. Flink
   2. Lucene
   3. *Ignite*
   4. Tomcat
   5. Kafka

Top 10 dev lists:

   1. Flink
   2. Tomcat
   3. Royale
   4. Beam
   5. James
   6. Kafka
   7. Netbeans
   8. *Ignite*
   9. Arrow
   10. Airflow


Top mailing lists (user + dev combined):

   1. Flink (user)
   2. Flink (dev)
   3. Tomcat (dev)
   4. Royale (dev)
   5. Beam (dev)
   6. Lucene (user)
   7. James (dev)
   8. Kafka (dev)
   9. Netbeans (dev)
   10. *Ignite (user)*
   11. *Ignite (dev)*

Congrats!
-
Denis


-- Forwarded message -
From: Sally Khudairi 
Date: Wed, Jul 29, 2020 at 5:07 AM
Subject: The Apache® Software Foundation Announces Annual Report for 2020
Fiscal Year
To: Apache Announce List 


[this announcement is available online at
https://s.apache.org/FY2020AnnualReport-PR ]

World's largest Open Source foundation provides 227M+ lines of code, valued
at more than $20B, to the public-at-large at 100% no cost.

Wakefield, MA —29 July 2020— The Apache® Software Foundation (ASF), the
all-volunteer developers, stewards, and incubators of more than 350 Open
Source projects and initiatives, announced today the availability of the
annual report for its 2020 fiscal year (1 May 2019 - 30 April 2020).

Now in its 21st year, the world's largest Open Source foundation’s "Apache
Way" of community-driven development is the proven process behind thousands
of developers successfully collaborating on hundreds of Apache projects.
The Apache Way has directly influenced the InnerSource methodology of
applying Open Source and open development principles to an organization.
The Apache Way has been adopted by countless organizations, including
Capital One, Comcast, Ericsson, HP, IBM, Google, Microsoft, PayPal, SAP,
T-Mobile, Walmart, and countless others.

Valued at more than $20B —and provided to the public-at-large at 100% no
cost— Apache software is used in every Internet-connected country on the
planet.

Apache software comprises 227M+ lines of code, is integral to nearly every
end user computing device, manages exabytes of data, executes teraflops of
operations, and stores billions of objects in virtually every industry.
Countless mission-critical projects in Artificial Intelligence, Big Data,
build management, Cloud Computing, content management, DevOps, Deep
Learning, IoT and Edge computing, mobile, servers, Web frameworks, and many
other categories are powered by Apache. [Learn more about the ASF’s reach
and influence at https://s.apache.org/ApacheEverywhere ]

Every Apache Top-Level Project (and its sub-projects, if applicable) is
overseen by a Project Management Committee (PMC) that guides its day-to-day
operations, including community development and product releases.

At the close of FY2020, 201 PMCs managed 339 Top-Level Projects and dozens
of sub-projects; 9 projects were newly-graduated Top-Level Projects from
the Apache Incubator, and 8 projects retired to the Apache Attic. Apache
Incubator PMCs mentored 45 projects under development in the Apache
Incubator; 6 projects were new entrants to the Incubator, and 3 were
retired.

FY2020 highlights include:

 - ~8M lines of Apache code added, valued at approximately $600M worth of
work; total code value exceeding $20B (CoCoMo model)
 - Stewardship of 227M+ lines of code in the Apache repositories;
 - Foundation operations supported by contributions from 10 Platinum
Sponsors, 9 Gold Sponsors, 11 Silver Sponsors, 25 Bronze Sponsors, 6
Platinum Targeted Sponsors, 5 Gold Targeted Sponsors, 3 Silver Targeted
Sponsors, 10 Bronze Targeted Sponsors, and more than 500 individual donors;
 - 34 new individual ASF Members elected, totalling 813;
 - Exceeded 7,700 code Committers;
 - 206 Top-Level communities overseeing 339+ Apache projects, plus dozens
of sub-projects and initiatives;
 - 9 newly-graduated Top-Level Projects from the Apache Incubator;
 - 45 projects currently undergoing development in the Apache Incubator;
 - Web requests received from every Internet-connected country on the
planet;
 - 35M+ page views per week across apache.org;
 - ~2 Petabytes source code downloads from Apache mirrors;
 - Top 5 most active/visited Apache projects: Kafka, Hadoop, Lucene, POI,
ZooKeeper;
 - Top 5 Apache repositories by number of commits: Camel, Flink, Beam,
HBase, Lucene Solr;
 - Top 5 Apache repositories by lines of code: NetBeans, OpenOffice, Flex
(combined), Mynewt (combined), Trafodion;
 - 2,892 Committers changed 60,132,710 lines of code over 174,889 commits;
 - 12,413 people created 

Re: Kubernetes : Connection timed out (Connection timed out)

[Denis Magda]

Steve, Jose,

Try to follow this instruction created specifically for Azure Kubernetes
Service (AKS):
https://www.gridgain.com/docs/latest/installation-guide/kubernetes/azure-deployment

Not, if the cluster (server nodes) are planned to be deployed as virtual
machines (on-premises) while the thick clients will be running in AKS then
keep this configuration option in mind (it's already supported by GridGain
and will be available in Ignite 2.9):
https://www.gridgain.com/docs/latest/developers-guide/clustering/running-client-nodes-behind-nat

-
Denis


On Tue, Jul 28, 2020 at 12:03 PM jose farfan  wrote:

> Hi
>
> Also I am using this file ignite-rbac.yaml. Maybe can help you,..
> I am still learning how to deploy ignite in the cluster.
>
> BR
> Jose
>
>
> On Tue, 28 Jul 2020 at 20:53, jose farfan  wrote:
>
>> Hi Steve
>>
>> No, I am using a on-premise cluster.
>>
>> can you access a pod using the shell, and try to ping
>> kubernetes.default.svc.cluster.local.
>>
>> BR
>> Jose
>>
>>
>>
>> On Tue, 28 Jul 2020 at 16:05, steve.hostettler <
>> steve.hostett...@gmail.com> wrote:
>>
>>> Running
>>>
>>>  curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H
>>> "Authorization: Bearer $(cat
>>> /var/run/secrets/kubernetes.io/serviceaccount/token)" -H "Acept:
>>> application/json"
>>>
>>> https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/default/endpoints/processing-engine-pe-v1-ignite
>>>
>>> On my local K8S works but not on Azure. On Azure I end up with a Timeout.
>>>
>>> Any Idea?
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>

Re: Load balanced CQ listener/Catch up missed events

[Denis Magda]

The client can create a queue, and the servers will keep items of that
queue. Those items will be stored in memory and disk tiers of the servers.
And, yes, physically the records will be persisted in the database folder
(that is under the work folder by default). Hope this helps.

-
Denis


On Fri, Jul 24, 2020 at 9:36 AM Devakumar J 
wrote:

> Hi Denis,
>
> Thanks for the reply.
>
> I mean to ask, is it possible to create distributed queue at client node
> and
> if so where it will be persisted when native persistence enabled.
>
> Should it be under server folder's work folder?
>
> Thanks,
> Devakumar J
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: 3rd party persistence: Mapping with RDBMS tables that don't have primary keys

[Denis Magda]

Hi Alex,


> I'm not going to write any new records or update existing ones for such
> tables through Ignite. So I can skip the overrides in CacheStore
> implementation, right? (I'm using default implementation at the moment)


That's right.

https://apacheignite.readme.io/docs/id-generator
> Thi may work. Can you point me to the example where Ignite config is in XML
> file


I don't have any XML example that shows how to create an AtomicSequence
bean. I tend to initialize it programmatically in code. Anyway, just create
it as a standard bean if it needs to be defined in the XML format.

-
Denis


On Wed, Jul 22, 2020 at 12:46 AM Alex Panchenko 
wrote:

> Hi Denis,
> thanks for your reply
>
> >>How will you access such records in Ignite? SQL lookups?
> Yes, using SQL lookups
>
> >>The default CacheStore implementation that writes down changes to a
> relational database needs to be overridden.
> I'm not going to write any new records or update existing ones for such
> tables through Ignite. So I can skip the overrides in CacheStore
> implementation, right? (I'm using default implementation at the moment)
>
> >>Obviously, Ignite still requires a primary key and that can be an integer
> number incremented by your application:
> https://apacheignite.readme.io/docs/id-generator
> Thi may work. Can you point me to the example where Ignite config is in XML
> file
>
> Also at the moment I'm using the following workaround. It seems like
> working, but I'm still testing it
>
> "keyType"="com.some.package.KeyValuesTableKey"
>
> KeyValuesTableKey {
> key: UUID
> value: String
> }
>
>
> Thank you
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Apache Ignite with istio/envoy mTLS

[Denis Magda]

To support this deployment model you would need to have a Kubernetes
Service per Ignite server node so that remote thick clients can connect to
any server from outside. Also, you would need to enable the following
setting on the clients side (the setting will be released in Ignite 2.9):
https://www.gridgain.com/docs/latest/developers-guide/clustering/running-client-nodes-behind-nat

Thin clients might be a much better choice for those applications that
connect from outside. It's easier to set up and manage. While inside K8 you
can keep using the thick clients.

-
Denis


On Thu, Jul 23, 2020 at 11:36 AM akorensh  wrote:

> Hi,
>At this point(2.8.1), Apache Ignite does not support Thick clients being
> outside of the K8 cluster.
>while servers being in the cluster.
> Thanks, Alex
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>