Re: Pessimistic Mode and Transactions and 2PC

[John Wilson]

No they are not. I just want to understand.

On Wednesday, August 8, 2018, Dmitriy Pavlov  wrote:

> Hi John,
>
> Are these questions related to some contribution?
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 8 авг. 2018 г. в 3:18, John Wilson :
>
> > Hi,
> >
> > Assume the following:
> >
> >
> >- I have a transaction coordinator and two primary nodes with 0 backup
> >nodes.
> >- Persistence store is enabled.
> >- I'm running a transaction in pessimistic mode with serializable
> >isolation.
> >
> > I have these questions:
> >
> >1. What exactly happens during the prepare phase? Only acquiring locks
> >on the two primary nodes? Or do the primary nodes themselves, in
> > addition
> >to acquiring locks, write to their respective WAL a TxRecord with a
> > "begin
> >prepare" info?
> >2. Assume locks have been acquired successfully, would the nodes then
> >write a "prepared" TxRecord to WAL before returning a "Yes" vote to
> >coordinator?
> >3. When the coordinator sends a commit message, would each node write
> >the key-values to the DataRecord and a commit to the TxRecord before
> >returning to coordinator?
> >
> >
> > Overall, I'm trying to understand what happens exactly during prepare and
> > commit phases and when the key-values involved in the transaction are
> > actually written; as well as the exact updates that are written to the
> WAL
> > files in each phase.
> >
> > appreciate your response.
> >
> > Thanks,
> >
>

Pessimistic Mode and Transactions and 2PC

[John Wilson]

Hi,

Assume the following:


   - I have a transaction coordinator and two primary nodes with 0 backup
   nodes.
   - Persistence store is enabled.
   - I'm running a transaction in pessimistic mode with serializable
   isolation.

I have these questions:

   1. What exactly happens during the prepare phase? Only acquiring locks
   on the two primary nodes? Or do the primary nodes themselves, in addition
   to acquiring locks, write to their respective WAL a TxRecord with a "begin
   prepare" info?
   2. Assume locks have been acquired successfully, would the nodes then
   write a "prepared" TxRecord to WAL before returning a "Yes" vote to
   coordinator?
   3. When the coordinator sends a commit message, would each node write
   the key-values to the DataRecord and a commit to the TxRecord before
   returning to coordinator?


Overall, I'm trying to understand what happens exactly during prepare and
commit phases and when the key-values involved in the transaction are
actually written; as well as the exact updates that are written to the WAL
files in each phase.

appreciate your response.

Thanks,

Folks, how do I generate tar.gz release for Ignite?

[John Wilson]

Hi,

How do I generate tar.gz for Ignite from source?

Thanks,

Quick question on data and index pages

[John Wilson]

Hi,

1. What are direct and indirect count in data page header used for? What is
the difference?

[
https://cwiki-test.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Freelists
]

2. Are data pages organized in a B+ tree structure or index pages only?

3. Is there any difference between internal and leaf nodes in the B+ tree
structure?


Thanks,

Quick questions on B+ Trees and Partitions

[John Wilson]

Hi,


   1. B+ tree initialization, BPlusTree.initTree, seems to be called for
   every partition. Why?
   
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/tree/BPlusTree.java
   2. The documentation here,
   https://apacheignite.readme.io/docs/memory-architecture, also states
   that for each SQL index, Ignite instantiates and manages a dedicated B+
   Tree. So, is the number of B+ trees determined by partition number of # of
   indexes defined?
   3. Assume I have a Person cache and an Organization cache. How many B+
   trees are defined for each cache
   4. What differentiates one B+ tree from another B+ tree? Just the cache
   it represents?


Thanks,
John

Re: Dirty Reads and READ_COMMITTED

[John Wilson]

And no. I'm not describing REPEATABLE_READ. I'm describing
https://en.wikipedia.org/wiki/Isolation_(database_systems)#Dirty_reads and
how READ_COMMITTED isolation can avoid dirty reads.

On Wed, Jul 25, 2018 at 11:20 AM, John Wilson 
wrote:

> I agree with your description. But the documentation  https://
> apacheignite.readme.io/docs/transactions is misleading in stating that no
> read locks are acquired (says "READ_COMMITTED - Data is read without a lock
> and is never cached in the transaction itself."). Which should be wrong.
> Read locks are acquired but they are released as soon as the read is
> complete (and they are not held until the transaction commits or rolls
> back).
>
> Thanks,
>
> On Wed, Jul 25, 2018 at 10:33 AM, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
>> Hi John,
>>
>> Read committed isolation typically implies that lock on read is not held
>> throughout the transaction lifecycle, i.e. released right after read is
>> completed (in Ignite this just means that no lock is required at all).
>> This
>> semantic allows second read to get an updated value that was already
>> committed by another transaction. See Wikipedia description for example:
>> https://en.wikipedia.org/wiki/Isolation_(database_systems)#Read_committed
>>
>> What you're describing is REPEATABLE_READ isolation which guarantees that
>> subsequent reads return the same value. In Ignite it's achieved by
>> acquiring lock on read and releasing it only on commit/rollback.
>>
>> -Val
>>
>> On Wed, Jul 25, 2018 at 10:12 AM John Wilson 
>> wrote:
>>
>> > Hi,
>> >
>> > Consider the following transaction where we read key 1 twice.
>> >
>> > try (Transaction tx = Ignition.ignite().transactions
>> ().txStart(PESSIMISTIC,
>> > READ_COMMITTED)) {
>> > cache.get(1);
>> > //...
>> > cache.get(1);
>> > tx.commit();
>> > }
>> >
>> > According to the documentation here,
>> > https://apacheignite.readme.io/docs/transactions, data is read without
>> a
>> > lock and is never cached. If that is the case, then how do we avoid a
>> dirty
>> > read on the second cache.get(1)? Another uncommitted transaction may
>> update
>> > the key between the first and second reads.
>> >
>> > In most RDMS, a READ_COMMITTED isolation level, acquires locks for both
>> > read and writes. The read lock is released after a read while the write
>> > lock is held until the transaction completes. So for the above example,
>> I
>> > expect a read lock on each cache.get(1).
>> >
>> >
>> > Thanks,
>> >
>>
>
>

Re: Dirty Reads and READ_COMMITTED

[John Wilson]

I agree with your description. But the documentation  https://apacheignite.
readme.io/docs/transactions is misleading in stating that no read locks are
acquired (says "READ_COMMITTED - Data is read without a lock and is never
cached in the transaction itself."). Which should be wrong. Read locks are
acquired but they are released as soon as the read is complete (and they
are not held until the transaction commits or rolls back).

Thanks,

On Wed, Jul 25, 2018 at 10:33 AM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Hi John,
>
> Read committed isolation typically implies that lock on read is not held
> throughout the transaction lifecycle, i.e. released right after read is
> completed (in Ignite this just means that no lock is required at all). This
> semantic allows second read to get an updated value that was already
> committed by another transaction. See Wikipedia description for example:
> https://en.wikipedia.org/wiki/Isolation_(database_systems)#Read_committed
>
> What you're describing is REPEATABLE_READ isolation which guarantees that
> subsequent reads return the same value. In Ignite it's achieved by
> acquiring lock on read and releasing it only on commit/rollback.
>
> -Val
>
> On Wed, Jul 25, 2018 at 10:12 AM John Wilson 
> wrote:
>
> > Hi,
> >
> > Consider the following transaction where we read key 1 twice.
> >
> > try (Transaction tx = Ignition.ignite().transactions().txStart(
> PESSIMISTIC,
> > READ_COMMITTED)) {
> > cache.get(1);
> > //...
> > cache.get(1);
> > tx.commit();
> > }
> >
> > According to the documentation here,
> > https://apacheignite.readme.io/docs/transactions, data is read without a
> > lock and is never cached. If that is the case, then how do we avoid a
> dirty
> > read on the second cache.get(1)? Another uncommitted transaction may
> update
> > the key between the first and second reads.
> >
> > In most RDMS, a READ_COMMITTED isolation level, acquires locks for both
> > read and writes. The read lock is released after a read while the write
> > lock is held until the transaction completes. So for the above example, I
> > expect a read lock on each cache.get(1).
> >
> >
> > Thanks,
> >
>

Dirty Reads and READ_COMMITTED

[John Wilson]

Hi,

Consider the following transaction where we read key 1 twice.

try (Transaction tx = Ignition.ignite().transactions().txStart(PESSIMISTIC,
READ_COMMITTED)) {
cache.get(1);
//...
cache.get(1);
tx.commit();
}

According to the documentation here,
https://apacheignite.readme.io/docs/transactions, data is read without a
lock and is never cached. If that is the case, then how do we avoid a dirty
read on the second cache.get(1)? Another uncommitted transaction may update
the key between the first and second reads.

In most RDMS, a READ_COMMITTED isolation level, acquires locks for both
read and writes. The read lock is released after a read while the write
lock is held until the transaction completes. So for the above example, I
expect a read lock on each cache.get(1).


Thanks,

Re: Does Ignite Txs Protect against Phantom Reads?

[John Wilson]

Hi Val,

Thanks, that answers my question.

John.

On Tue, Jul 24, 2018 at 5:13 PM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Hi John,
>
> Looks like pictures can't be attached directly. Try uploading it somewhere
> and providing a link.
>
> However, since you're talking about ranges, my guess would be that you're
> using SQL which is currently NOT transactional. This support currently in
> development though, probably other members of the community can provide
> more details on the progress.
>
> -Val
>
> On Tue, Jul 24, 2018 at 4:52 PM John Wilson 
> wrote:
>
> > Hi,
> >
> > Do Ignite transactions with serializable isolation level protect against
> > phantom reads? For example, in the below example, does Ignite, in
> > pessimistic mode, lock all entries in the range 10 to 30?
> >
> > Thanks.
> >
> >
> >
> >
>

Does Ignite Txs Protect against Phantom Reads?

[John Wilson]

Hi,

Do Ignite transactions with serializable isolation level protect against
phantom reads? For example, in the below example, does Ignite, in
pessimistic mode, lock all entries in the range 10 to 30?

Thanks.

Re: Quick questions on segments and page map buckets

[John Wilson]

I'm talking about PageMemoryNostoreImpl here:
https://github.com/apache/ignite/blob/ce8e31e28e16c7c22fe88ab6b1a1304b14f6881d/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java#L109
and
https://github.com/apache/ignite/blob/ce8e31e28e16c7c22fe88ab6b1a1304b14f6881d/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java#L208

Thanks,

On Thu, Jul 12, 2018 at 10:31 AM, Dmitry Pavlov 
wrote:

> Hi, I've double checked code regarding question 1:
>
> PageMemoryImpl#segments segments count (len) comes from sizes array len.
> Sizes array come to page memory from
> GridCacheDatabaseSharedManager.calculateFragmentSizes(). This method which
> gets availableProcessors from runtime (CPU count). I didn't find any shift
> to SEG_BITS.
>
> It can be chaged by DataStorageConfiguration.setConcurrencyLevel() but by
> default is the same with #CPU.
>
> Which is why I mentioned segments count = CPU count by default in wiki.
>
> Recenly some member of community proposed update to offical doc about
> segments count. So now it states #segments=#CPU.
>
> So I guess for 8 CPU we will have 8 segments.
>
> Please correct me if I missing something.
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 27 июн. 2018 г. в 15:30, Eduard Shangareev <
> eduard.shangar...@gmail.com
> >:
>
> > Dmitry,
> >
> > I mean Chugunov, but I have questioned him. His answer was the A.
> Goncharuk
> > should now the answer.
> >
> > On Wed, Jun 27, 2018 at 2:58 PM, Dmitry Pavlov 
> > wrote:
> >
> > > John, is this question still actual?
> > >
> > > пт, 22 июн. 2018 г. в 15:18, Dmitry Pavlov :
> > >
> > > > Hi Ed,
> > > >
> > > > Which Sergey do you mean? I know a number of contributors.
> > > >
> > > > Sincerely,
> > > >
> > > >
> > > > вт, 19 июн. 2018 г. в 22:15, Eduard Shangareev <
> > > > eduard.shangar...@gmail.com>:
> > > >
> > > >> Hi,
> > > >>
> > > >> 1. It looks weird, yeah. Need to ask Sergey, who has changed it last
> > > time.
> > > >>
> > > >> 2. Because we could reuse memory. For example, after cache destroy
> or
> > > >> something like that.
> > > >>
> > > >> On Tue, Jun 19, 2018 at 9:58 PM, John Wilson <
> sami.hailu...@gmail.com
> > >
> > > >> wrote:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > Two quick questions:
> > > >> >
> > > >> >
> > > >> >1. The design documentation here,
> > > >> >https://cwiki.apache.org/confluence/display/IGNITE/
> > > >> > Ignite+Durable+Memory+-+under+the+hood,
> > > >> >states that the default segment count is equal to the number of
> > > >> logical
> > > >> >cores available in the underlying machine. However, the
> segments
> > > >> array
> > > >> > in
> > > >> >PageMemory indicates that the maximum number of segments is: 1
> <<
> > > >> > SEG_BITS.
> > > >> >Since SEG_BITS = 4, the max # segments is 16. Did I miss
> > something
> > > >> here?
> > > >> >2. Reading the code in PageMemoryNoStoreImp, it looks like
> pages
> > > are
> > > >> >allocated segment sequentially in a bump-the-pointer strategy
> > where
> > > >> the
> > > >> >first 8 bytes of a segment hold a pointer to the index of the
> > last
> > > >> >allocated page. If this is true, then I don't understand the
> > point
> > > of
> > > >> >having a page map buckets. Why not use a simple arithmetic
> index
> > *
> > > >> > pageSize
> > > >> >to get the offset of a page?
> > > >> >
> > > >> > Thanks.
> > > >> > John
> > > >> >
> > > >>
> > > >
> > >
> >
>

Optimistic mode with serializable vs read_committed

[John Wilson]

Hi,

I was reading this documentation,
https://www.gridgain.com/resources/blog/apache-ignite-transactions-architecture-concurrency-modes-and-isolation-levels,
to understand the difference between Optimistic mode with Serializable vs
read_committed. The only difference I see from the explanation given is
that for Serializable isolation, the "*primary nodes manage the transaction
requests internally*" before an ACK is sent to the coordinator while for
read_committed, the ACK is sent immediately and the nodes "*manage the
transaction requests internally*" later.

My question:


   1. What does "*primary nodes manage the transaction requests
internally"* exactly
   mean?
   2. What are the specific things a node has to do to manage transaction
   requests internally?

Thanks,
John

Re: A quick question on cluster topology and partitions?

[John Wilson]

Regarding 1. I'm referring to the documentation here,
https://apacheignite.readme.io/docs/topology-validation, which states
"*Topology
validator is used to verify that cluster topology is valid for further
cache operations*." and "*If topology validator is not configured, then the
cluster topology is always considered to be valid.*"

Thanks,

On Mon, Jul 2, 2018 at 4:21 PM, Dmitriy Setrakyan 
wrote:

> On Mon, Jul 2, 2018 at 4:03 PM, John Wilson 
> wrote:
>
> > Hi,
> >
> >
> >1. What exactly is a cluster topology? What makes a cluster topology
> >invalid for further cache operations?
> >
>
> Cluster topology is a set of Ignite nodes in the cluster. I do not think a
> cluster topology could be invalid on its own. Perhaps you are asking about
> a situation when after a certain number of node failures/stops we can be in
> a situation where all primary and backup copies become inaccessible. In
> that case, the cluster should enter a read-only state for the lost
> partitions.
>
>
> >2.  Why do we have the concept of partitions in Ignite? Why don't we
> >have a key-to-node mapping rather than a key-to-partition and a
> >partition-to-node mapping?
> >
>
> Main reason is because there is a finite number of partitions and there is
> an infinite number of keys. Whenever ignite topology changes, Ignite must
> rebalance data to the new nodes (or to the existing nodes). In this case,
> Ignite needs to know when a certain partition is moved to another node. If
> there were no partitions, then it would be impossible to tell when to
> finish the rebalancing process.
>
> D.
>

A quick question on cluster topology and partitions?

[John Wilson]

Hi,


   1. What exactly is a cluster topology? What makes a cluster topology
   invalid for further cache operations?
   2.  Why do we have the concept of partitions in Ignite? Why don't we
   have a key-to-node mapping rather than a key-to-partition and a
   partition-to-node mapping?

Thanks,
John

Two quick questions on index page and data page

[John Wilson]

Hi,


   1. An index page contains the hash value of key and a link; where a link
   is page_id + offset. Question: what is this offset? Is it the offset to the
   item in the data page? In other words, Ignite locates the page and the item
   within the page and finally gets the key-value pair by following the item?
   2. The design doc below states that the item serves as an internal
   reference (offset) to the key value pair. Does the item has information
   about the size of the key-value is it pointing to? If not, how does it now
   the size of the key-value.

https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood

Thanks,
John

Quick questions on segments and page map buckets

[John Wilson]

Hi,

Two quick questions:


   1. The design documentation here,
   
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood,
   states that the default segment count is equal to the number of logical
   cores available in the underlying machine. However, the segments array in
   PageMemory indicates that the maximum number of segments is: 1 << SEG_BITS.
   Since SEG_BITS = 4, the max # segments is 16. Did I miss something here?
   2. Reading the code in PageMemoryNoStoreImp, it looks like pages are
   allocated segment sequentially in a bump-the-pointer strategy where the
   first 8 bytes of a segment hold a pointer to the index of the last
   allocated page. If this is true, then I don't understand the point of
   having a page map buckets. Why not use a simple arithmetic index * pageSize
   to get the offset of a page?

Thanks.
John

Re: How does Ignite garbage collect unused pages?

[John Wilson]

Thanks!

On Thu, Jun 14, 2018 at 2:00 AM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> John,
>
> The page is moved between FreeList buckets synchronously during the
> corresponding cache entry update, so it is the part of cache.put() or
> cache.remove() operation.
>
> ср, 13 июн. 2018 г. в 2:32, Denis Magda :
>
> > Whenever you add or remove an entry, it changes the size of a page which
> > can lead to page movement between free list. According to this page, the
> > page defragmentation/compaction happens in the background and when a
> > threshold is met:
> > https://apacheignite.readme.io/docs/memory-defragmentation
> >
> > Hope Ignite persistence experts can shine more light on this.
> >
> > --
> > Denis
> >
> > On Tue, Jun 12, 2018 at 3:12 PM John Wilson 
> > wrote:
> >
> > > thanks. But *when* does that happen - i.e. when is the decision made to
> > > move pages? Is this part of the cache.put path or a separate thread?
> > >
> > > On Tue, Jun 12, 2018 at 1:03 PM, Denis Magda 
> wrote:
> > >
> > > > A page is moved between free lists that used to track pages of
> similar
> > > free
> > > > space left:
> > > >
> > >
> > https://apacheignite.readme.io/docs/memory-architecture#
> section-free-lists
> > > >
> > > > --
> > > > Denis
> > > >
> > > >
> > > > On Tue, Jun 12, 2018 at 12:35 PM John Wilson <
> sami.hailu...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > How does Ignite free unused pages? Is there some kind of background
> > > > thread
> > > > > process that scans unused pages?
> > > > >
> > > > > Thanks,
> > > > >
> > > >
> > >
> >
>

Re: How does Ignite garbage collect unused pages?

[John Wilson]

thanks. But *when* does that happen - i.e. when is the decision made to
move pages? Is this part of the cache.put path or a separate thread?

On Tue, Jun 12, 2018 at 1:03 PM, Denis Magda  wrote:

> A page is moved between free lists that used to track pages of similar free
> space left:
> https://apacheignite.readme.io/docs/memory-architecture#section-free-lists
>
> --
> Denis
>
>
> On Tue, Jun 12, 2018 at 12:35 PM John Wilson 
> wrote:
>
> > Hi,
> >
> > How does Ignite free unused pages? Is there some kind of background
> thread
> > process that scans unused pages?
> >
> > Thanks,
> >
>

How does Ignite garbage collect unused pages?

[John Wilson]

Hi,

How does Ignite free unused pages? Is there some kind of background thread
process that scans unused pages?

Thanks,

Re: Optimistic Locking and the Prepare Phase

[John Wilson]

Hi  Vladimir,

Your answer is what is depicted in the graphics and makes perfect sense to
me. I guess what I'm confused about is what a "prepare" phase means and
what "*In optimistic transactions, locks are acquired on primary nodes
during the "prepare" phase* " means.

My understanding of a "prepare" phase based on the blog here (
https://www.gridgain.com/resources/blog/apache-ignite-transactions-architecture-2-phase-commit-protocol)
is that it is the phase where we acquire all the necessary locks (in
pessimistic locking) before we start the commit phase.


   1. In the context of *pessimistic* locking, at the end of the prepare
   phase but before we start commit, we would have acquired all locks. True?
   2. In the context of *optimistic* locking, a prepare phase would not
   request for or acquire locks. True?
   3. In the context of *optimistic* locking, at the end of the prepare
   phase but before we start commit, we have stored the current version of the
   keys in the transaction coordinator but we have not yet requested or
   acquired any locks. Locks will be acquired during the commit phase. True?

Thanks!

On Tue, Feb 13, 2018 at 12:54 AM, Vladimir Ozerov <voze...@gridgain.com>
wrote:

> Hi John,
>
> 1) In PESSIMISTIC mode locks are obtained either on first update
> (READ_COMMITTED) or even read (REPEATABLE_READ). I.e. they obtained before
> prepare phase and are held for the duration of transaction. In OPTIMISTIC
> mode locks are obtained only after you call IgniteTransaction.commit().
> 2) It means that transaction will fail if enlisted entries have been
> changed after they were accessed by current transaction, but before this
> transaction is committed.
>
> On Tue, Feb 13, 2018 at 9:49 AM, John Wilson <sami.hailu...@gmail.com>
> wrote:
>
> > Hi,
> >
> > The design doc below states:
> >
> > *" In optimistic transactions, locks are acquired on primary nodes during
> > the "prepare" phase, then promoted to backup nodes and released once the
> > transaction is committed. Depending on an isolation level, if Ignite
> > detects that a version of an entry has been changed since the time it was
> > requested by a transaction, then the transaction will fail at the
> "prepare"
> > phase and it will be up to an application to decide whether to restart
> the
> > transaction or not."*
> >
> > Two questions:
> >
> >
> >1. If locks are acquired during the prepare phase, why do we state
> that
> >lock acquisition for optimistic locking is delayed (as compared
> against
> >pessimistic locking)?
> >2. If "*ignite detects the version has changed since last request by
> >transaction, it will fail at prepare phase*". Very confusing. What is
> >the last request? I thought the "last request" means the "prepare"
> phase
> >and if so why we say it may fail during prepare phase?
> >
> > The graphic make sense to me - i.e. locks for optimistic locking are
> > acquired on the commit phase and not on the prepare phase.
> >
> > https://cwiki.apache.org/confluence/display/IGNITE/
> > Ignite+Key-Value+Transactions+Architecture
> >
> > Please help clarify.
> >
> > Thanks.
> >
>

Re: Page Locking vs Entry-level Locking

[John Wilson]

Hi Pavlov,

Thanks for your explanation. However, I still have these questions:


   1. If locking is per page-level and there are no entry-level locks, then
   why does the documentation here talk about having entry-level transaction
   locks? https://apacheignite.readme.io/docs/transactions
   2. I'm not clear with what "locking a durable memory region segment "
   means, what is a segment? Does a page contain multiple segments or a
   segment contains multiple pages? The mental picture I have is: a memory
   region is divided into pages (which can be meta data pages, index, or data
   pages). Index pages hold links to data page id and offset for key-value
   pairs while data pages contain the actual key-value pairs. So, what really
   is a segment and what does locking a memory segment means? I understanding
   page-locks, what is segment lock?

Appreciate your response!!!

On Tue, Feb 13, 2018 at 2:32 AM, Dmitry Pavlov <dpavlov@gmail.com>
wrote:

> Hi John,
>
> 1. No, content modification require lock holding to page to provide
> consitency in multithreaded environment.
> 2. Page is locked for read before reading its content, and unlocked after.
> Same for lock for write for writting. 1 writer or N readers allowed for
> page. On unlock write lock, dirty flag for page may be set if there data
> was actually modified.
> 3. Lock has per-page basis, additional striping by any offest within page
> is not required accoring to tests.
>
> Only one contention is observed sometimes in high load test, it is
> contention of threads to lock to durable memory region segment. But this
> situation can be handled by setting concurrenclyLevel in
> DataStorageConfiguration.
>
> Sincerely,
> Dmitriy Pavlov
>
>
> вт, 13 февр. 2018 г. в 9:56, John Wilson <sami.hailu...@gmail.com>:
>
> > Hi,
> >
> > Ignite documentation talks about entry-level locks and the page structure
> > has a LOCK_OFFSET that I assume is used to store tag info. I have these
> > questions.
> >
> >1. Does Ignite use a lock-free implementation to lock pages and/or
> >entries?
> >2. When is a page locked and when is it released?
> >3. When an entry is inserted/modified in a page, is the page locked
> >(forbidding other threads from inserting entries in the page)? or only
> > the
> >entry's offset is locked? (allowing other threads to insert/modify
> other
> >items)
> >
> > Thanks!
> >
>

Page Locking vs Entry-level Locking

[John Wilson]

Hi,

Ignite documentation talks about entry-level locks and the page structure
has a LOCK_OFFSET that I assume is used to store tag info. I have these
questions.

   1. Does Ignite use a lock-free implementation to lock pages and/or
   entries?
   2. When is a page locked and when is it released?
   3. When an entry is inserted/modified in a page, is the page locked
   (forbidding other threads from inserting entries in the page)? or only the
   entry's offset is locked? (allowing other threads to insert/modify other
   items)

Thanks!

Optimistic Locking and the Prepare Phase

[John Wilson]

Hi,

The design doc below states:

*" In optimistic transactions, locks are acquired on primary nodes during
the "prepare" phase, then promoted to backup nodes and released once the
transaction is committed. Depending on an isolation level, if Ignite
detects that a version of an entry has been changed since the time it was
requested by a transaction, then the transaction will fail at the "prepare"
phase and it will be up to an application to decide whether to restart the
transaction or not."*

Two questions:


   1. If locks are acquired during the prepare phase, why do we state that
   lock acquisition for optimistic locking is delayed (as compared against
   pessimistic locking)?
   2. If "*ignite detects the version has changed since last request by
   transaction, it will fail at prepare phase*". Very confusing. What is
   the last request? I thought the "last request" means the "prepare" phase
   and if so why we say it may fail during prepare phase?

The graphic make sense to me - i.e. locks for optimistic locking are
acquired on the commit phase and not on the prepare phase.

https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Key-Value+Transactions+Architecture

Please help clarify.

Thanks.

Re: What happens if Primary Node fails during the Commit Phase

[John Wilson]

I got the answer for #3 here
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Pages.
I will post the remaining questions in a separate thread.

On Mon, Feb 12, 2018 at 8:03 PM, John Wilson <sami.hailu...@gmail.com>
wrote:

> You're always helpful Val. Thanks!
>
>
> I have a question regarding Optimistic Locking
>
>
>1. The documentation here, https://cwiki.apache.
>org/confluence/display/IGNITE/Ignite+Key-Value+Transactions+
>Architecture
>
> <https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Key-Value+Transactions+Architecture>,
>states that locks, for optimistic locking, are acquired during the
>"prepare" phase. But the graphic depicted there and the tutorial here,
>https://www.gridgain.com/resources/blog/apache-ignite-transactions-
>architecture-concurrency-modes-and-isolation-levels
>
> <https://www.gridgain.com/resources/blog/apache-ignite-transactions-architecture-concurrency-modes-and-isolation-levels>,
>clearly indicate that locks are acquired during the commit phase; with a
>version information passed along with the key by the coordinator to the
>primary nodes. Can you please explain the discrepancy?
>
> And two questions regarding pages and page locking?
>
>1. Does Ignite use a lock-free algorithm for its B+ tree structure?
>Looking at the source code and the use of a tag field to avoid the ABA
>problem suggests that.
>2. Ignite documentation talks about entry-level locks and page locks.
>When exactly is a page locked and released? Also, when an entry is
>inserted/modified in a page, is the page locked, forbidding other threads
>from inserting other entries in the page, or only the entry's offset is
>locked allowing other threads to insert other entries in the page?
>3. What is the the difference between a directCount and indirectCount
>for a page?
>
> Thanks
>
> On Mon, Feb 12, 2018 at 7:33 PM, vkulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
>> Hi John,
>>
>> 1. True.
>>
>> 2. The blog actually provides the answer:
>>
>> When the Backup Nodes detect the failure, they will notify the Transaction
>> coordinator that they committed the transaction successfully. In this
>> scenario, there is no data loss because the data are backed up and can
>> still
>> be accessed and used by applications.
>>
>> In other words, if primary node fails, backups will not wait for a
>> message,
>> but instead will commit right away and send an ack to the coordinator.
>> Once
>> coordinator gets all required acs, transaction completes.
>>
>> -Val
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>

Re: What happens if Primary Node fails during the Commit Phase

[John Wilson]

You're always helpful Val. Thanks!


I have a question regarding Optimistic Locking


   1. The documentation here,
   
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Key-Value+Transactions+Architecture,
   states that locks, for optimistic locking, are acquired during the
   "prepare" phase. But the graphic depicted there and the tutorial here,
   
https://www.gridgain.com/resources/blog/apache-ignite-transactions-architecture-concurrency-modes-and-isolation-levels,
   clearly indicate that locks are acquired during the commit phase; with a
   version information passed along with the key by the coordinator to the
   primary nodes. Can you please explain the discrepancy?

And two questions regarding pages and page locking?

   1. Does Ignite use a lock-free algorithm for its B+ tree structure?
   Looking at the source code and the use of a tag field to avoid the ABA
   problem suggests that.
   2. Ignite documentation talks about entry-level locks and page locks.
   When exactly is a page locked and released? Also, when an entry is
   inserted/modified in a page, is the page locked, forbidding other threads
   from inserting other entries in the page, or only the entry's offset is
   locked allowing other threads to insert other entries in the page?
   3. What is the the difference between a directCount and indirectCount
   for a page?

Thanks

On Mon, Feb 12, 2018 at 7:33 PM, vkulichenko 
wrote:

> Hi John,
>
> 1. True.
>
> 2. The blog actually provides the answer:
>
> When the Backup Nodes detect the failure, they will notify the Transaction
> coordinator that they committed the transaction successfully. In this
> scenario, there is no data loss because the data are backed up and can
> still
> be accessed and used by applications.
>
> In other words, if primary node fails, backups will not wait for a message,
> but instead will commit right away and send an ack to the coordinator. Once
> coordinator gets all required acs, transaction completes.
>
> -Val
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

What happens if Primary Node fails during the Commit Phase

[John Wilson]

Hi,

Assume the Prepare phase has completed and that the primary node has
received a commit message from the coordinator.

Two questions:

   1. A primary node commits a transaction before it forwards a commit
   message to the backup nodes. True?
   2. What happens if a Primary Node fails while it is committing but
   before the commit message is sent to backup nodes? Do the backup nodes
   commit after some timeout or will they send a fail message to the
   coordinator?

The doc below provides a nice description but doesn't exactly answer my
question.

https://www.gridgain.com/resources/blog/apache-ignite-transactions-architecture-failover-and-recovery

Thanks,

What is the purpose of a binary schema and schema registry?

[John Wilson]

Hi,

When objects are marshaled, Ignites adds a schema (BinarySchema) to the
BinarySchemaRegistry. Moreover, the documentation says that an object can
have a few different schemas.

My question:

   1. What does it mean for an object to have multiple schemas? (e.g. for a
   simple person object Person obj = new Person())
   2. What is the purpose of the binary schema registry?

Thanks,

Ignite Marshalling & Serialization

[John Wilson]

Hi,

Ignite marshals data before it writes it to the off-heap data regions. Can
someone please explain to me the difference between marshaling and
serialization, in the context of Ignite?

Thanks,