Re: Adding experimental support for Intel Optane DC Persistent Memory

[Ivan Pavlukhin]

Denis,

I suppose we are using not optimal strategy here. And motivation is simple:
1. Code review is "expensive", benchmarking (usually) are "cheap".
2. Code review is "subjective", benchmarking are "objective".

And my personal opinion here is that there could be a performance win
with Optane in proposed implementation in particular cases. It depends
on how much slower is Optane performance comparing to RAM. But we do
not know it.

ср, 16 окт. 2019 г. в 01:10, Denis Magda :
>
> Alex, thanks for an extensive summary and proposing different
> implementation trails. Presently, the JEP-352 approach seems the most
> reasonable - JDK folks will take care of the integration and will be
> testing/maintaining the feature going forward, while the Ignite community
> will reuse what JDK provides. However, let's see what Eric and Mulugeta of
> Intel (both copied) think about this and other approaches.
>
> Ivan, I'm not sure if any performance testing was done for the given
> pull-request. Anyway, it might be premature considering Alex's summary.
>
> -
> Denis
>
>
> On Mon, Oct 14, 2019 at 11:50 PM Ivan Pavlukhin  wrote:
>
> > By the way, did we have a change to run some existing benchmarks
> > against proposed implementation?
> >
> > пт, 11 окт. 2019 г. в 17:49, Alexey Goncharuk  > >:
> > >
> > > Hello Mulugeta, Igniters.
> > >
> > > Thanks for your interest and efforts in integrating the persistent memory
> > > to Ignite. Here are my thoughts on the PR:
> > >
> > > First of all I was questioning whether we want to use the integration
> > with
> > > the pmem library via JNI. Java 14 will have native support for NVME via
> > > native mapped byte buffers [1] since the corresponding tickets are
> > already
> > > resolved [2],[3]. The suggested integration uses JNI calls quite often
> > (if
> > > I read the PR right, there will be tens of JNI calls per a single
> > > operation), which may and most likely will undermine the benefits of
> > using
> > > PM. On the other hand, there is a PoC/project [4] which successfully
> > > demonstrates how the new integration can be used to natively access
> > > persistent memory from Java. Additionally, new Unsafe have several
> > > performance issues - each put* methods has a map lookup, and CAS method
> > > uses a global lock. Removing these issues will change the PR and
> > > architecture dramatically.
> > >
> > > Other than performance and an extra library dependency, there are
> > > additional reasons to use custom implementation of PM persistence
> > > primitives:
> > >
> > >- Ignite significantly benefits from having WAL as a single point of
> > >data for both failure recovery and historical rebalance. As far as I
> > can
> > >guess, the suggested library also uses some sort of journaling in
> > order to
> > >support crash recovery. Since we do not want give up on the historical
> > >rebalance, we would need to write an additional journal, which then
> > raises
> > >a question of how to transact between this journal.
> > >- Ignite is supposed to work with large indexes, a I not aware how
> > LLPL
> > >handles them. Several researches [5],[6] show that since PM memory has
> > >larger access latency than regular RAM, it is beneficial to buffer
> > index
> > >pages in regular memory and store leaf pages in PM. There is a generic
> > >approach to convert in-memory indexes to PM ones [7], I think it makes
> > >sense to investigate whether our existing indexes can be converted to
> > PM.
> > >- There is an active IEP which is targeted for a file-based
> > rebalancing.
> > >In the suggested implementation this optimization would not work. A
> > better
> > >integration should allow to take a snapshot of a partition and
> > transfer it
> > >to another node for fast rebalancing.
> > >
> > > Overall, I think we may take the following path:
> > >
> > >- Target JEP352 APIs; for now keep using the whole region msync for
> > flush
> > >- Introduce PM-based isolated components to support WAL iterator and
> > >transaction management. We can start with WAL, then implement a
> > PM-based
> > >index based on current research, then a tuple heap with free space
> > >management. Each of the components may be developed and tested
> > independently
> > >- Do Ignite internals refactoring to properly abstract storage layer
> > >from transactions/communication. Currently, there is an abstraction
> > leak in
> > >the storage layer, which makes it aware of particular Ignite
> > implementation
> > >details (Binary Objects, partition eviction, etc)
> > >- Once the storage is separated, we should be able to plug the new
> > >components instead of the "classical" ones
> > >
> > > WDYT? I believe we should create a detailed IEP with concrete proposals
> > on
> > > both PM structures design and Ignite internals changes before making any
> > > further code changes.
> > >
> > > [1] 

Re: [DISCUSS] Pub/Sub Streamer Implementation

[Denis Magda]

Emmanouil, Saikat,

After contemplating on this, let me propose another way with this
particular integration that is aligned with our modularization endeavor.
The modularization [1] defines the Ignite core, Ignite modules, and
independent integrations. The first two are to be located in Ignite
repositories and to be tested/updated by the community for every release.
The modules are a subset of existing integrations that are adopted widely
and require community attention.

As for the independent integrations [1], those are supposed to be located
in a separate GitHub repo. I would advise us to start implementing this
concept with the Pub/Sub integration. Even though it will be located in a
separate Github repo, it will be listed among officially available
integrations on the Ignite website featuring Emmanouil as a contributor.

What do you think?


[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations



-
Denis


On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra 
wrote:

> Hi Emmanouil,
>
> The changes looks good to me. I wanted to check if you are also able to
> validate that the once data is added to Ignite cluster you are also able to
> access it using another Ignite client or using rest endpoints.
>
> We use Teamcity for CI process, the details are as follows
>
> Trigger validation of those test suites that have been affected by your
> changes on TeamCity :  >
>
>- Locate a test suite you have to check, press button named "..." that
>is located on the left of "Run" button. "Run custom build" dialog will
>appear;
>- Go to "Changes" tab and choose "pull//head" in
>"Build branch" dropdown list;
>- Press "Run build" button and monitor tests results.
>
>
>
> @Roman rsht...@yahoo.com - can you also please take a look?
>
> Regards,
> Saikat
>
> On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
> gkatzio...@gmail.com>
> wrote:
>
> > Hi @Saikat Maitra
> >
> > I have my implementation on an Ignite fork on my personal GitHub account
> > [1]
> > It is based on the Kafka implementation which is of a polling nature.
> > GCP does not provide a service like Kafka. Instead it provides us with
> > Pub/Sub which is just like Kafka.
> > Therefore using the Kafka streamer as a guideline made more sense to me.
> >
> > Apart from the implementation I have also added a unit test.
> > It definitely needs some polishing and a double check on my side to make
> > the review smooth.
> > For the tests a local Pub/Sub [2] server has to be setup however I am not
> > sure which is the best way to do so with regards to your CI/CD.
> >
> > [1] https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> > [2] https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> >
> > *Emmanouil Gkatziouras*
> > https://egkatzioura.com/ |
> > https://www.linkedin.com/in/gkatziourasemmanouil/
> > https://github.com/gkatzioura
> >
> >
> > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra 
> > wrote:
> >
> > > Hello Emmanouil,
> > >
> > > Can you please share more info about the implementation?
> > >
> > > We have similar implementation for data streamers with Kafka and
> RocketMQ
> > >
> > > https://apacheignite-mix.readme.io/docs/kafka-streamer
> > >
> > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> > >
> > > Please review and share if you have any questions.
> > >
> > > Regards,
> > > Saikat
> > >
> > >
> > >
> > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
> > > gkatzio...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1]
> > > > Yes it is indeed cloud specific. It is the messaging service provided
> > by
> > > > GCP.
> > > > I have made the integration with it, but I haven't published it yet
> (I
> > am
> > > > waiting for the discussion/ticket and proceed with the contribution).
> > > >
> > > > Kind regards,
> > > > Emmanouil
> > > >
> > > > [1] https://cloud.google.com/pubsub/docs/overview
> > > >
> > > > *Emmanouil Gkatziouras*
> > > > https://egkatzioura.com/ |
> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > https://github.com/gkatzioura
> > > >
> > > >
> > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda  wrote:
> > > >
> > > > > Hello Emmanouil,
> > > > >
> > > > > Are you referring to any specific implementation tailored for a
> > cloud?
> > > If
> > > > > I'm not mistaken you were mentioning AWS/GCE. Is there any Github
> > > source
> > > > > code?
> > > > >
> > > > > Here is a list of existing integrations with streaming and pub/sub
> > > > > frameworks:
> > > > > https://apacheignite-mix.readme.io/docs/overview
> > > > >
> > > > > with the source code in Ignite master:
> > > > > https://github.com/apache/ignite/tree/master/modules
> > > > >
> > > > > Check out those resources to see what are the most suitable Ignite
> > APIs
> > > > for
> > > > > such integrations.
> > > > >
> 

Re: Adding experimental support for Intel Optane DC Persistent Memory

[Denis Magda]

Alex, thanks for an extensive summary and proposing different
implementation trails. Presently, the JEP-352 approach seems the most
reasonable - JDK folks will take care of the integration and will be
testing/maintaining the feature going forward, while the Ignite community
will reuse what JDK provides. However, let's see what Eric and Mulugeta of
Intel (both copied) think about this and other approaches.

Ivan, I'm not sure if any performance testing was done for the given
pull-request. Anyway, it might be premature considering Alex's summary.

-
Denis


On Mon, Oct 14, 2019 at 11:50 PM Ivan Pavlukhin  wrote:

> By the way, did we have a change to run some existing benchmarks
> against proposed implementation?
>
> пт, 11 окт. 2019 г. в 17:49, Alexey Goncharuk  >:
> >
> > Hello Mulugeta, Igniters.
> >
> > Thanks for your interest and efforts in integrating the persistent memory
> > to Ignite. Here are my thoughts on the PR:
> >
> > First of all I was questioning whether we want to use the integration
> with
> > the pmem library via JNI. Java 14 will have native support for NVME via
> > native mapped byte buffers [1] since the corresponding tickets are
> already
> > resolved [2],[3]. The suggested integration uses JNI calls quite often
> (if
> > I read the PR right, there will be tens of JNI calls per a single
> > operation), which may and most likely will undermine the benefits of
> using
> > PM. On the other hand, there is a PoC/project [4] which successfully
> > demonstrates how the new integration can be used to natively access
> > persistent memory from Java. Additionally, new Unsafe have several
> > performance issues - each put* methods has a map lookup, and CAS method
> > uses a global lock. Removing these issues will change the PR and
> > architecture dramatically.
> >
> > Other than performance and an extra library dependency, there are
> > additional reasons to use custom implementation of PM persistence
> > primitives:
> >
> >- Ignite significantly benefits from having WAL as a single point of
> >data for both failure recovery and historical rebalance. As far as I
> can
> >guess, the suggested library also uses some sort of journaling in
> order to
> >support crash recovery. Since we do not want give up on the historical
> >rebalance, we would need to write an additional journal, which then
> raises
> >a question of how to transact between this journal.
> >- Ignite is supposed to work with large indexes, a I not aware how
> LLPL
> >handles them. Several researches [5],[6] show that since PM memory has
> >larger access latency than regular RAM, it is beneficial to buffer
> index
> >pages in regular memory and store leaf pages in PM. There is a generic
> >approach to convert in-memory indexes to PM ones [7], I think it makes
> >sense to investigate whether our existing indexes can be converted to
> PM.
> >- There is an active IEP which is targeted for a file-based
> rebalancing.
> >In the suggested implementation this optimization would not work. A
> better
> >integration should allow to take a snapshot of a partition and
> transfer it
> >to another node for fast rebalancing.
> >
> > Overall, I think we may take the following path:
> >
> >- Target JEP352 APIs; for now keep using the whole region msync for
> flush
> >- Introduce PM-based isolated components to support WAL iterator and
> >transaction management. We can start with WAL, then implement a
> PM-based
> >index based on current research, then a tuple heap with free space
> >management. Each of the components may be developed and tested
> independently
> >- Do Ignite internals refactoring to properly abstract storage layer
> >from transactions/communication. Currently, there is an abstraction
> leak in
> >the storage layer, which makes it aware of particular Ignite
> implementation
> >details (Binary Objects, partition eviction, etc)
> >- Once the storage is separated, we should be able to plug the new
> >components instead of the "classical" ones
> >
> > WDYT? I believe we should create a detailed IEP with concrete proposals
> on
> > both PM structures design and Ignite internals changes before making any
> > further code changes.
> >
> > [1] https://openjdk.java.net/jeps/352
> > [2] https://bugs.openjdk.java.net/browse/JDK-8221397
> > [3] https://bugs.openjdk.java.net/browse/JDK-8221696
> > [4] https://github.com/jhalliday/mashona
> > [5] https://www.usenix.org/system/files/conference/fast17/fast17-lee.pdf
> > [6] http://www.vldb.org/pvldb/vol8/p786-chen.pdf
> > [7] https://www.cs.utexas.edu/~vijay/papers/sosp19-recipe.pdf
> > [8]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-16%3A+Optimization+of+rebalancing
> >
> > вт, 8 окт. 2019 г. в 08:15, Alexey Goncharuk  >:
> >
> > > Igniters,
> > >
> > > I would like to resurrect this discussion and will review the change
> again
> > > shortly. If anyone want to join the 

Re: Broken image links in under-the-hood articles in wiki

[Dmitriy Pavlov]

Yes, it may be that something is wrong. I don't remember exactly.

But if this was another image, it was definitely lost (it was just png, no
sources available).


вт, 15 окт. 2019 г. в 18:29, Ivan Pavlukhin :

> Dmitriy,
>
> Thank you!
>
> Funny thing, I added exactly the same picture first. But later
> realized that there is an almost same picture in section "BPlus Tree
> Structure" on the same page [1]. So, I decided that there is something
> wrong =)
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
>
> вт, 15 окт. 2019 г. в 17:34, Dmitriy Pavlov :
> >
> > Hi Ivan,
> >
> > thank you for doing all these changes. I've attached the last one.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > вт, 15 окт. 2019 г. в 15:56, Ivan Pavlukhin :
> >
> > > Dmitriy,
> > >
> > > I edited pages [1, 2] by myself. Unfortunately failed to find a
> > > suitable image for [1] in section "Pages & links" (left a note).
> > > Please check when you have time.
> > >
> > > Just for a protocol, original images are at [3].
> > >
> > > [1]
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
> > > [2]
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
> > > [3]
> > >
> https://drive.google.com/drive/folders/1eRIG2WVf9MmmwCcBpZEaR1dEalusDxhR
> > >
> > > вт, 15 окт. 2019 г. в 15:06, Dmitriy Pavlov :
> > > >
> > > > I've shared in ASF slack, Ignite channel.
> > > >
> > > > I didn't manage to do it on the weekend, but this change is on my
> list.
> > > >
> > > > пн, 14 окт. 2019 г. в 18:23, Ivan Pavlukhin :
> > > >
> > > > > Dmitriy,
> > > > >
> > > > > > PS, I've shared links to new images sources in the Slack channel.
> > > > >
> > > > > Cannot find links. Which Slack channel was meant?
> > > > >
> > > > > чт, 10 окт. 2019 г. в 00:42, Dmitriy Pavlov :
> > > > > >
> > > > > > Hi Ivan, thanks for bringing this here, it is definitely needed
> to be
> > > > > fixed.
> > > > > >
> > > > > > There is the same issue for one picture here, as well,
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
> > > > > >
> > > > > > I'll replace images in the wiki as well, hopefully, by the end
> of the
> > > > > > weekend.
> > > > > >
> > > > > > Sincerely,
> > > > > > Dmitriy Pavlov
> > > > > >
> > > > > > PS, I've shared links to new images sources in the Slack channel.
> > > > > >
> > > > > > ср, 9 окт. 2019 г. в 15:19, Ivan Pavlukhin  >:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I faced that "Ignite Durable Memory - under the hood" [1] page
> has
> > > > > > > some broken image links.
> > > > > > >
> > > > > > > Dmitriy Pavlov, perhaps you as an author know how to deal with
> it.
> > > > > > >
> > > > > > > Folks, if you saw other broken links do not hesitate to write
> to
> > > this
> > > > > > > thread.
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
> > > > > > >
> > > > > > > --
> > > > > > > Best regards,
> > > > > > > Ivan Pavlukhin
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Ivan Pavlukhin
> > > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

[MTCGA]: new failures in builds [4698426] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 *New Critical Failure in master Platform .NET (Inspections)* 
https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PlatformNetInspections?branch=%3Cdefault%3E
 No changes in the build

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 22:38:56 15-10-2019 

[jira] [Created] (IGNITE-12293) .NET thin client: config serialization should use OpCodes

[Alexandr Shapkin (Jira)]

Alexandr Shapkin created IGNITE-12293:
-

 Summary: .NET thin client: config serialization should use OpCodes
 Key: IGNITE-12293
 URL: https://issues.apache.org/jira/browse/IGNITE-12293
 Project: Ignite
  Issue Type: Improvement
  Components: thin client
Affects Versions: 2.7.6
Reporter: Alexandr Shapkin
Assignee: Alexandr Shapkin
 Fix For: 2.9


Thin client configuration could be partially serialized via 
ClientCacheConfigurationSerializer.java

But this #OpCode based serialization is limited only to .NET -> Java direction.

Let's use OpCodes as a default implementation for all supported platforms, this 
will help us to add a new configuration section only for a specific platform.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12292) Java thin client: In some cases txId intersection (tx started on different nodes) leads to errors

[Aleksey Plekhanov (Jira)]

Aleksey Plekhanov created IGNITE-12292:
--

 Summary: Java thin client: In some cases txId intersection (tx 
started on different nodes) leads to errors
 Key: IGNITE-12292
 URL: https://issues.apache.org/jira/browse/IGNITE-12292
 Project: Ignite
  Issue Type: Bug
  Components: thin client
Affects Versions: 2.8
Reporter: Aleksey Plekhanov
Assignee: Aleksey Plekhanov
 Fix For: 2.8


Reproducer:
{code:java}
public void testTxWithIdIntersection() throws Exception {
int CLUSTER_SIZE = 2;

try (LocalIgniteCluster cluster = LocalIgniteCluster.start(CLUSTER_SIZE);
 IgniteClient client = Ignition.startClient(new ClientConfiguration()
 .setAddresses(cluster.clientAddresses().toArray(new 
String[CLUSTER_SIZE])))
) {
ClientCache cache = client.createCache(new 
ClientCacheConfiguration().setName("cache")
.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL));

CyclicBarrier barrier = new CyclicBarrier(2);

GridTestUtils.runAsync(() -> {
try {
// Another thread starts transaction here.
barrier.await(1, TimeUnit.SECONDS);

for (int i = 0; i < CLUSTER_SIZE; i++)
dropAllThinClientConnections(Ignition.allGrids().get(i));

ClientTransaction tx = client.transactions().txStart();

barrier.await(1, TimeUnit.SECONDS);

// Another thread puts to cache here.
barrier.await(1, TimeUnit.SECONDS);

tx.commit();

barrier.await(1, TimeUnit.SECONDS);
}
catch (Exception e) {
log.error("Unexpected error", e);
}
});

ClientTransaction tx = client.transactions().txStart();

barrier.await(1, TimeUnit.SECONDS);

// Another thread drops connections and create new transaction here, 
which started on another node with the
// same transaction id as we started in this thread.
barrier.await(1, TimeUnit.SECONDS);

try {
cache.put(0, 0);

fail("Exception expected");
}
catch (ClientException expected) {
// No-op.
}

tx.close();

barrier.await(1, TimeUnit.SECONDS);

// Another thread commit transaction here.
barrier.await(1, TimeUnit.SECONDS);

assertFalse(cache.containsKey(0));
}
}
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Broken image links in under-the-hood articles in wiki

[Ivan Pavlukhin]

Dmitriy,

Thank you!

Funny thing, I added exactly the same picture first. But later
realized that there is an almost same picture in section "BPlus Tree
Structure" on the same page [1]. So, I decided that there is something
wrong =)

[1] 
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood

вт, 15 окт. 2019 г. в 17:34, Dmitriy Pavlov :
>
> Hi Ivan,
>
> thank you for doing all these changes. I've attached the last one.
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 15 окт. 2019 г. в 15:56, Ivan Pavlukhin :
>
> > Dmitriy,
> >
> > I edited pages [1, 2] by myself. Unfortunately failed to find a
> > suitable image for [1] in section "Pages & links" (left a note).
> > Please check when you have time.
> >
> > Just for a protocol, original images are at [3].
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
> > [2]
> > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
> > [3]
> > https://drive.google.com/drive/folders/1eRIG2WVf9MmmwCcBpZEaR1dEalusDxhR
> >
> > вт, 15 окт. 2019 г. в 15:06, Dmitriy Pavlov :
> > >
> > > I've shared in ASF slack, Ignite channel.
> > >
> > > I didn't manage to do it on the weekend, but this change is on my list.
> > >
> > > пн, 14 окт. 2019 г. в 18:23, Ivan Pavlukhin :
> > >
> > > > Dmitriy,
> > > >
> > > > > PS, I've shared links to new images sources in the Slack channel.
> > > >
> > > > Cannot find links. Which Slack channel was meant?
> > > >
> > > > чт, 10 окт. 2019 г. в 00:42, Dmitriy Pavlov :
> > > > >
> > > > > Hi Ivan, thanks for bringing this here, it is definitely needed to be
> > > > fixed.
> > > > >
> > > > > There is the same issue for one picture here, as well,
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
> > > > >
> > > > > I'll replace images in the wiki as well, hopefully, by the end of the
> > > > > weekend.
> > > > >
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > >
> > > > > PS, I've shared links to new images sources in the Slack channel.
> > > > >
> > > > > ср, 9 окт. 2019 г. в 15:19, Ivan Pavlukhin :
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I faced that "Ignite Durable Memory - under the hood" [1] page has
> > > > > > some broken image links.
> > > > > >
> > > > > > Dmitriy Pavlov, perhaps you as an author know how to deal with it.
> > > > > >
> > > > > > Folks, if you saw other broken links do not hesitate to write to
> > this
> > > > > > thread.
> > > > > >
> > > > > > [1]
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > > Ivan Pavlukhin
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >



-- 
Best regards,
Ivan Pavlukhin

[jira] [Created] (IGNITE-12291) Create controllable paged query requests / responses for TextQuery similar to current SQL result processing

[Yuriy Shuliha (Jira)]

Yuriy Shuliha  created IGNITE-12291:
---

 Summary: Create controllable paged query requests / responses for 
TextQuery similar to current SQL result processing
 Key: IGNITE-12291
 URL: https://issues.apache.org/jira/browse/IGNITE-12291
 Project: Ignite
  Issue Type: Improvement
Reporter: Yuriy Shuliha 
Assignee: Yuriy Shuliha 


For now query initiator node sends 1 _GridCacheQueryRequest_ and can get 
multiple _GridCacheQueryResponse_-s per remote.

TextQuery processing finishes when all remotes send _GridCacheQueryResponse_ 
with 'finished' flag.

_TextQuery_ processing  should be reworked like it was done for SQL queries.

Ignite has _GridQueryNextPageRequest \ Response_ classes for SQL result 
processing.
Similar processing should be implemented for _TextQueries_:
*GridTextQueryNextPageRequest*
*GridTextQueryNextPageResponse* 

Proper _TextQuery_ response processing should be implemented in 
_GridCacheQueryFutureAdapter._

This will allow us to add correct sorting and apply limit correctly on reduce.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Broken image links in under-the-hood articles in wiki

[Dmitriy Pavlov]

Hi Ivan,

thank you for doing all these changes. I've attached the last one.

Sincerely,
Dmitriy Pavlov

вт, 15 окт. 2019 г. в 15:56, Ivan Pavlukhin :

> Dmitriy,
>
> I edited pages [1, 2] by myself. Unfortunately failed to find a
> suitable image for [1] in section "Pages & links" (left a note).
> Please check when you have time.
>
> Just for a protocol, original images are at [3].
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
> [2]
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
> [3]
> https://drive.google.com/drive/folders/1eRIG2WVf9MmmwCcBpZEaR1dEalusDxhR
>
> вт, 15 окт. 2019 г. в 15:06, Dmitriy Pavlov :
> >
> > I've shared in ASF slack, Ignite channel.
> >
> > I didn't manage to do it on the weekend, but this change is on my list.
> >
> > пн, 14 окт. 2019 г. в 18:23, Ivan Pavlukhin :
> >
> > > Dmitriy,
> > >
> > > > PS, I've shared links to new images sources in the Slack channel.
> > >
> > > Cannot find links. Which Slack channel was meant?
> > >
> > > чт, 10 окт. 2019 г. в 00:42, Dmitriy Pavlov :
> > > >
> > > > Hi Ivan, thanks for bringing this here, it is definitely needed to be
> > > fixed.
> > > >
> > > > There is the same issue for one picture here, as well,
> > > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
> > > >
> > > > I'll replace images in the wiki as well, hopefully, by the end of the
> > > > weekend.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > PS, I've shared links to new images sources in the Slack channel.
> > > >
> > > > ср, 9 окт. 2019 г. в 15:19, Ivan Pavlukhin :
> > > >
> > > > > Hi,
> > > > >
> > > > > I faced that "Ignite Durable Memory - under the hood" [1] page has
> > > > > some broken image links.
> > > > >
> > > > > Dmitriy Pavlov, perhaps you as an author know how to deal with it.
> > > > >
> > > > > Folks, if you saw other broken links do not hesitate to write to
> this
> > > > > thread.
> > > > >
> > > > > [1]
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Ivan Pavlukhin
> > > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

Re: Broken image links in under-the-hood articles in wiki

[Ivan Pavlukhin]

Dmitriy,

I edited pages [1, 2] by myself. Unfortunately failed to find a
suitable image for [1] in section "Pages & links" (left a note).
Please check when you have time.

Just for a protocol, original images are at [3].

[1] 
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
[2] 
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
[3] https://drive.google.com/drive/folders/1eRIG2WVf9MmmwCcBpZEaR1dEalusDxhR

вт, 15 окт. 2019 г. в 15:06, Dmitriy Pavlov :
>
> I've shared in ASF slack, Ignite channel.
>
> I didn't manage to do it on the weekend, but this change is on my list.
>
> пн, 14 окт. 2019 г. в 18:23, Ivan Pavlukhin :
>
> > Dmitriy,
> >
> > > PS, I've shared links to new images sources in the Slack channel.
> >
> > Cannot find links. Which Slack channel was meant?
> >
> > чт, 10 окт. 2019 г. в 00:42, Dmitriy Pavlov :
> > >
> > > Hi Ivan, thanks for bringing this here, it is definitely needed to be
> > fixed.
> > >
> > > There is the same issue for one picture here, as well,
> > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
> > >
> > > I'll replace images in the wiki as well, hopefully, by the end of the
> > > weekend.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > PS, I've shared links to new images sources in the Slack channel.
> > >
> > > ср, 9 окт. 2019 г. в 15:19, Ivan Pavlukhin :
> > >
> > > > Hi,
> > > >
> > > > I faced that "Ignite Durable Memory - under the hood" [1] page has
> > > > some broken image links.
> > > >
> > > > Dmitriy Pavlov, perhaps you as an author know how to deal with it.
> > > >
> > > > Folks, if you saw other broken links do not hesitate to write to this
> > > > thread.
> > > >
> > > > [1]
> > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >



-- 
Best regards,
Ivan Pavlukhin

Re: Broken image links in under-the-hood articles in wiki

[Dmitriy Pavlov]

I've shared in ASF slack, Ignite channel.

I didn't manage to do it on the weekend, but this change is on my list.

пн, 14 окт. 2019 г. в 18:23, Ivan Pavlukhin :

> Dmitriy,
>
> > PS, I've shared links to new images sources in the Slack channel.
>
> Cannot find links. Which Slack channel was meant?
>
> чт, 10 окт. 2019 г. в 00:42, Dmitriy Pavlov :
> >
> > Hi Ivan, thanks for bringing this here, it is definitely needed to be
> fixed.
> >
> > There is the same issue for one picture here, as well,
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
> >
> > I'll replace images in the wiki as well, hopefully, by the end of the
> > weekend.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > PS, I've shared links to new images sources in the Slack channel.
> >
> > ср, 9 окт. 2019 г. в 15:19, Ivan Pavlukhin :
> >
> > > Hi,
> > >
> > > I faced that "Ignite Durable Memory - under the hood" [1] page has
> > > some broken image links.
> > >
> > > Dmitriy Pavlov, perhaps you as an author know how to deal with it.
> > >
> > > Folks, if you saw other broken links do not hesitate to write to this
> > > thread.
> > >
> > > [1]
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

Re: [DISCUSSION] Single point in API for changing cluster state.

[Sergey Antonov]

Hi, Alexei!

Thank you for reply!

> The states ACTIVE, INACTIVE, READ-ONLY look confusing. Actually read-only
cluster is active too.
How about INACTIVE, ACTIVE, ACTIVE_READ-ONLY states?

> Also it would be useful to allow users wait for re-balance which could
happen after activation in read-only mode to achieve really idle grid.
Read-only mode doesn't affects rebalance process.

> I would suggest adding new property to Ignite configuration like
setActivationOptions(ActivationOption... options) which should be mutable
in runtime.
I'm not sure that it's good idea. @Alexey Goncharuk  I'd
like to know your opinion about activation option and storing them on PDS.

> This proposal also better regarding backward compatibility.
Which kind of compatibility did you mean? New cluster mode doesn't affects
PDS compatibility.

ср, 25 сент. 2019 г. в 13:26, Alexei Scherbakov <
alexey.scherbak...@gmail.com>:

> Sergey Antonov,
>
> The states ACTIVE, INACTIVE, READ-ONLY look confusing.
> Actually read-only cluster is active too.
>
> I would suggest adding new property to Ignite configuration like
> setActivationOptions(ActivationOption... options) which should be mutable
> in runtime.
>
> For control.sh should be the same options for activate command.
>
> Also it would be useful to allow users wait for re-balance which could
> happen after activation in read-only mode to achieve really idle grid.
>
> So, available options list in my opinion: READ_ONLY, WAIT_FOR_REBALANCE.
>
> This proposal also better regarding backward compatibility.
>
> вт, 24 сент. 2019 г. в 20:35, Sergey Antonov :
>
> > Maxim,
> >
> > > I think from the user point the INACTIVE -> READ-ONLY transition [1]
> > should be allowed prior to adding a new `state` command [2] to avoid
> > unnecessary error messages.
> > Yes. I plan complete both tickets till the code freeze of 2.8 release.
> >
> > >   I also think we can avoid the word 'set` in this command.
> > I don't think so. We already have command control.sh --state command for
> > getting current state. I think we shouldn't "overload" commands in
> > control.sh.
> >
> > > What about cluster deactivation confirmation? Will it be used for all
> the
> > cluster state changes?
> > Yes. I think we should confirm any cluster state transition.
> >
> > вт, 24 сент. 2019 г. в 19:07, Maxim Muzafarov :
> >
> > > Sergey,
> > >
> > > +1, I like your idea.
> > >
> > > I think from the user point the INACTIVE -> READ-ONLY transition [1]
> > > should be allowed prior to adding a new `state` command [2] to avoid
> > > unnecessary error messages. I also think we can avoid the word 'set`
> > > in this command.
> > >
> > > Example: control.sh --state ACTIVE [--yes]
> > >
> > >
> > > What about cluster deactivation confirmation? Will it be used for all
> > > the cluster state changes?
> > >   Deactivate cluster:
> > > control.(sh|bat) --deactivate [--yes]
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-11866
> > > [2] https://issues.apache.org/jira/browse/IGNITE-12225
> > >
> > >
> > >
> > > On Tue, 24 Sep 2019 at 16:50, Sergey Antonov <
> antonovserge...@gmail.com>
> > > wrote:
> > > >
> > > > Andrey,
> > > >
> > > > > What are state transitions valid?
> > > > Now all transitions are valid, except INACTIVE -> READ-ONLY. This
> > > > transition will be fixed under [1]
> > > >
> > > > > Regarding state names, as I understand, all transitions are valid
> > from
> > > > any to any of 3 states.
> > > > Yes, see my comment above.
> > > >
> > > > > But, regarding on console.sh command it is not obvious.
> > > > Yes. It's one of points why we should have single command in
> > control.sh.
> > > >
> > > > > What effect will --read-only-on and --read-only-off commands have
> if
> > > > current state is INACTIVE ?
> > > > --read-only-on - cluster will be activated in read-only mode
> > > > --read-only-off - cluster will be activated. I.e --read-only-off will
> > > have
> > > > the same effect as --activate
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-11866
> > > >
> > > > вт, 24 сент. 2019 г. в 16:40, Andrey Mashenkov <
> > > andrey.mashen...@gmail.com>:
> > > >
> > > > > Sergey,
> > > > >
> > > > > What are state transitions valid?
> > > > > For now we have only 2 states (active and inactive) and possible
> > > > > transitions are obvious Active <--> Inactive.
> > > > >
> > > > > Regarding state names, as I understand, all transitions are valid
> > from
> > > any
> > > > > to any of 3 states.
> > > > > But, regarding on console.sh command it is not obvious.
> > > > > What effect will --read-only-on and --read-only-off commands have
> if
> > > > > current state is INACTIVE ?
> > > > >
> > > > >
> > > > > On Tue, Sep 24, 2019 at 4:23 PM Sergey Antonov <
> > > antonovserge...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Also, I would add IGNITE-12225
> > > > > >  ticket to
> 2.8
> > > > > release
> > > > > > scope.
> > > > > >
> > > > > > вт, 24 

Write access to Apache Ignite Confluence

[Denis Garus]

Hi, Igniters!

I'd like to add a new wiki page with IEP for the Sandbox feature.

Could someone grant me (Denis Garus) write permission?

Re: Adding experimental support for Intel Optane DC Persistent Memory

[Ivan Pavlukhin]

By the way, did we have a change to run some existing benchmarks
against proposed implementation?

пт, 11 окт. 2019 г. в 17:49, Alexey Goncharuk :
>
> Hello Mulugeta, Igniters.
>
> Thanks for your interest and efforts in integrating the persistent memory
> to Ignite. Here are my thoughts on the PR:
>
> First of all I was questioning whether we want to use the integration with
> the pmem library via JNI. Java 14 will have native support for NVME via
> native mapped byte buffers [1] since the corresponding tickets are already
> resolved [2],[3]. The suggested integration uses JNI calls quite often (if
> I read the PR right, there will be tens of JNI calls per a single
> operation), which may and most likely will undermine the benefits of using
> PM. On the other hand, there is a PoC/project [4] which successfully
> demonstrates how the new integration can be used to natively access
> persistent memory from Java. Additionally, new Unsafe have several
> performance issues - each put* methods has a map lookup, and CAS method
> uses a global lock. Removing these issues will change the PR and
> architecture dramatically.
>
> Other than performance and an extra library dependency, there are
> additional reasons to use custom implementation of PM persistence
> primitives:
>
>- Ignite significantly benefits from having WAL as a single point of
>data for both failure recovery and historical rebalance. As far as I can
>guess, the suggested library also uses some sort of journaling in order to
>support crash recovery. Since we do not want give up on the historical
>rebalance, we would need to write an additional journal, which then raises
>a question of how to transact between this journal.
>- Ignite is supposed to work with large indexes, a I not aware how LLPL
>handles them. Several researches [5],[6] show that since PM memory has
>larger access latency than regular RAM, it is beneficial to buffer index
>pages in regular memory and store leaf pages in PM. There is a generic
>approach to convert in-memory indexes to PM ones [7], I think it makes
>sense to investigate whether our existing indexes can be converted to PM.
>- There is an active IEP which is targeted for a file-based rebalancing.
>In the suggested implementation this optimization would not work. A better
>integration should allow to take a snapshot of a partition and transfer it
>to another node for fast rebalancing.
>
> Overall, I think we may take the following path:
>
>- Target JEP352 APIs; for now keep using the whole region msync for flush
>- Introduce PM-based isolated components to support WAL iterator and
>transaction management. We can start with WAL, then implement a PM-based
>index based on current research, then a tuple heap with free space
>management. Each of the components may be developed and tested 
> independently
>- Do Ignite internals refactoring to properly abstract storage layer
>from transactions/communication. Currently, there is an abstraction leak in
>the storage layer, which makes it aware of particular Ignite implementation
>details (Binary Objects, partition eviction, etc)
>- Once the storage is separated, we should be able to plug the new
>components instead of the "classical" ones
>
> WDYT? I believe we should create a detailed IEP with concrete proposals on
> both PM structures design and Ignite internals changes before making any
> further code changes.
>
> [1] https://openjdk.java.net/jeps/352
> [2] https://bugs.openjdk.java.net/browse/JDK-8221397
> [3] https://bugs.openjdk.java.net/browse/JDK-8221696
> [4] https://github.com/jhalliday/mashona
> [5] https://www.usenix.org/system/files/conference/fast17/fast17-lee.pdf
> [6] http://www.vldb.org/pvldb/vol8/p786-chen.pdf
> [7] https://www.cs.utexas.edu/~vijay/papers/sosp19-recipe.pdf
> [8]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-16%3A+Optimization+of+rebalancing
>
> вт, 8 окт. 2019 г. в 08:15, Alexey Goncharuk :
>
> > Igniters,
> >
> > I would like to resurrect this discussion and will review the change again
> > shortly. If anyone want to join the review - you are welcome!
> >
> > ср, 22 авг. 2018 г. в 18:49, Denis Magda :
> >
> >> Hi Dmitry,
> >>
> >> That's a BSD-3-Clause license if to believe this statement
> >> "SPDX-License-Identifier: BSD-3-Clause":
> >> https://github.com/pmem/llpl/blob/master/LICENSE
> >>
> >> This license can be used with ASF software:
> >> https://www.apache.org/legal/resolved.html#category-a
> >>
> >> --
> >> Denis
> >>
> >> On Wed, Aug 22, 2018 at 9:28 AM Dmitriy Pavlov 
> >> wrote:
> >>
> >> > Hi Denis,
> >> >
> >> > Could you please double check if we can refer to any library licensed to
> >> > Intel. Can we develop code only version of this support (without
> >> shipping
> >> > it in release)?
> >> >
> >> > https://github.com/apache/ignite/pull/4381 is quite huge change,
> >> > including 128 files changed, patch review