Re: Partition Id for a Data key in case of data colocation

[Stanislav Lukyanov]

What do you mean by "store the data in appropriate bucket"? Affinity key is
used to locate the partition, but the (regular) key is used to store the
entry itself, so entries don't override each other since each entry has a
unique key.

Stan


prasadbhalerao1983 wrote
> Hi,
> 
> With affinity key ignite locates the partition and from partition it
> locates the node using partition to node mapping. Now in this partition (I
> believe that partition is nothing but a Map like data structure) to store
> the data in appropriate bucket ignite must be using some thing apart from
> affinity key. If every time is uses the affinity key then multiple records
> with same affinity key will land in same map bucket resulting in
> overwriting the data.
> 
> So in my case if it just uses assetGroupId to locate the bucket inside
> partition then it will simply overwrite the previous values. That's why I
> asked, does ignite use any other properties from Key object to locate the
> appropriate bucket in a given partition?
> 
> Thanks,
> Prasad
> 
> On Mon, Mar 12, 2018 at 9:31 PM, Stanislav Lukyanov 

> stanlukyanov@

> 
> wrote:
> 
>> There may be some misunderstanding.
>> What I meant was that the data most likely WILL have good distribution
>> with
>> assetGroupId being @AffinityMappedId. To have a good distribution it is
>> generally enough to have a lot of data groups, so that it is likely that
>> each partition stores more or less equal number of groups. You have 100K
>> groups - that seems to be a large enough number to me.
>>
>> Also, no, Ignite doesn't use all the fields to define where to put the
>> data.
>> Only affinity matters (which is in your case defined by assetGroupId).
>>
>> Thanks,
>> Stan
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Partition Id for a Data key in case of data colocation

[Stanislav Lukyanov]

There may be some misunderstanding.
What I meant was that the data most likely WILL have good distribution with
assetGroupId being @AffinityMappedId. To have a good distribution it is
generally enough to have a lot of data groups, so that it is likely that
each partition stores more or less equal number of groups. You have 100K
groups - that seems to be a large enough number to me.

Also, no, Ignite doesn't use all the fields to define where to put the data.
Only affinity matters (which is in your case defined by assetGroupId).

Thanks,
Stan



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Partition Id for a Data key in case of data colocation

[Prasad Bhalerao]

Hi Stan,

Thank you for the clarification. Now it is clear that partition id is
always selected based on a field annotated with @Affinitykeymapped
annotations.

Now the question is how does ignite locate the data bucket in a given
partition?
Does it use all the properties defined in key object and hashes it, in this
case id and assetGroupId?

I understand that data colocation on assetGroupId will not distribute the
data evenly across the cluster.

But the thing my data access pattern is going on be as follows.

Select * from Cache where assetGroupId =? And ipStart <= :minIp and ipEnd
>=:maxIp

Also, I will be having approx unique 100k assetGroupIds and each
assetGroupId will have 1 to N(approx. 1  to 2 million records). If I do not
colocate the data on assetGroupId then my SQL might get very slow.

Can you please suggest any better solution?

On Mon, Mar 12, 2018, 6:48 PM Stanislav Lukyanov 
wrote:

> Hi,
>
> Your code for getting a partition ID is correct.
>
> Ignite can't use your "id" field to calculate partition - it just doesn't
> know anything about it. It doesn't depend on the name, so calling it "id"
> doesn't help, and it doesn't depend on the backing Oracle DB, so it doesn't
> know that "id" is an autoincremented primary key.
>
> You're correct about that with your setup all objects with the same
> assetGroupId will end up in the same partition. That's just the semantics
> of
> the @AffinityKeyMapped - "put all data with the same value in the same
> partition". It isn't a bad thing per se - if you have 100K unique
> assetGroupId values, it should be enough to have a decent data distribution
> across the cluster.
>
> Thanks,
> Stan
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Partition Id for a Data key in case of data colocation

[Stanislav Lukyanov]

Hi,

Your code for getting a partition ID is correct.

Ignite can't use your "id" field to calculate partition - it just doesn't
know anything about it. It doesn't depend on the name, so calling it "id"
doesn't help, and it doesn't depend on the backing Oracle DB, so it doesn't
know that "id" is an autoincremented primary key.

You're correct about that with your setup all objects with the same
assetGroupId will end up in the same partition. That's just the semantics of
the @AffinityKeyMapped - "put all data with the same value in the same
partition". It isn't a bad thing per se - if you have 100K unique
assetGroupId values, it should be enough to have a decent data distribution
across the cluster.

Thanks,
Stan



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/