[jira] [Commented] (IGNITE-1770) Portables: implement constant-time field lookup.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982490#comment-14982490
 ] 

ASF GitHub Bot commented on IGNITE-1770:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/183


> Portables: implement constant-time field lookup.
> 
>
> Key: IGNITE-1770
> URL: https://issues.apache.org/jira/browse/IGNITE-1770
> Project: Ignite
>  Issue Type: Task
>  Components: general, interop
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>Priority: Blocker
> Fix For: 1.5
>
>
> See 
> https://cwiki.apache.org/confluence/display/IGNITE/Portable+object+constant-time+field+lookup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-1770) Portables: implement constant-time field lookup.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980358#comment-14980358
 ] 

ASF GitHub Bot commented on IGNITE-1770:


GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/182

IGNITE-1770 Portables: implement constant-time field lookup

Adopt protocol in .NET

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptupitsyn/ignite ignite-1770-dotnet

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/182.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #182


commit 4121b549462fd14272b341843935c4f1dbad343d
Author: vozerov-gridgain 
Date:   2015-10-23T08:16:37Z

IGNITE-1665: Avoid field ID re-calculation during reads.

commit f15d66c2203c46c8951b051690e1b7b662fde8c2
Author: vozerov-gridgain 
Date:   2015-10-23T12:02:42Z

IGNITE-1770: WIP.

commit b9422dfa16cf3674d4d0c7587330bd309cbb21f6
Author: vozerov-gridgain 
Date:   2015-10-23T12:03:00Z

Merge branch 'ignite-1282' into ignite-1770

commit d218714087d64369bbe0f8c855d7c9cd3f2b8ef2
Author: vozerov-gridgain 
Date:   2015-10-23T12:36:49Z

IGNITE-1770: Added 1 byte to flags.

commit 1e62f46721edcf59671337df7a7df2180e470aa7
Author: vozerov-gridgain 
Date:   2015-10-23T13:35:29Z

IGNITE-1770: Simplifications on writer side.

commit 844ff1e829b0071d129b332d3a6748960dec66b4
Author: vozerov-gridgain 
Date:   2015-10-23T14:19:30Z

IGNITE-1770: Better raw mode handling.

commit cd931d3e2d67d527e8f493c86eaa8248b0356327
Author: vozerov-gridgain 
Date:   2015-10-23T14:20:16Z

IGNITE-1770: Better raw mode handling.

commit 03e1f8d25eb257e7cfcc3fd2c8d7f427b3a470c8
Author: vozerov-gridgain 
Date:   2015-10-23T14:21:42Z

IGNITE-1770: Better raw mode handling.

commit 2a8b75a269f71f2487bd9eb24457df2e1c53d3c1
Author: vozerov-gridgain 
Date:   2015-10-23T14:26:03Z

IGNITE-1770: Better raw mode handling.

commit 7cf21365f124c16560eba05d14507e7e124a838b
Author: vozerov-gridgain 
Date:   2015-10-23T14:27:10Z

IGNITE-1770: Better raw mode handling.

commit d5327836941fd4805ca441de51a81e427484bfb6
Author: vozerov-gridgain 
Date:   2015-10-23T14:28:53Z

IGNITE-1770: Better raw mode handling.

commit 80bbc7def3c32ce3daac7901731ef35c2d5bfde7
Author: vozerov-gridgain 
Date:   2015-10-23T14:31:04Z

IGNITE-1770: Better raw mode handling.

commit 46e1734d26a33342870246c034090119e1aa7be9
Author: vozerov-gridgain 
Date:   2015-10-23T14:37:11Z

IGNITE-1770: Better tests.

commit 90cb58842627855eac14621398e4a1619c704878
Author: vozerov-gridgain 
Date:   2015-10-23T14:41:13Z

IGNITE-1770: Better tests.

commit 0a024013f0b23fc1108b610a1ccb955163ee4d31
Author: vozerov-gridgain 
Date:   2015-10-23T14:53:39Z

IGNITE-1770: Added space for schema offset.

commit c4ce99736e054f52696be8387aa91e8ec9ca0e23
Author: vozerov-gridgain 
Date:   2015-10-23T14:58:13Z

IGNITE-1770: Field info is written to the tail.

commit 7bd743c3697e4e3886e7b3e12a5836907f877fec
Author: vozerov-gridgain 
Date:   2015-10-26T07:42:36Z

IGNITE-1770: Offset then ID in the tail.

commit 96d08a5312f1c6270abb755ed65784b60ac660b4
Author: vozerov-gridgain 
Date:   2015-10-26T07:51:50Z

IGNITE-1770: Do not write offset for the first field.

commit d6ba7185dca7f0f160d63a67486995e80ef5792a
Author: vozerov-gridgain 
Date:   2015-10-26T07:56:14Z

IGNITE-1770: Write raw offset if needed.

commit 8c039fb1d25ca1c1e874c813e2877945cd4c63e6
Author: vozerov-gridgain 
Date:   2015-10-26T08:26:59Z

IGNITE-1770: Reader refactoring.

commit 47e54b2db4415aa3cd52ad09edcad54089f96ea0
Author: vozerov-gridgain 
Date:   2015-10-26T08:53:02Z

IGNITE-1770: Reader refactoring.

commit 1fc0757b73b81d1a56fd6bf14632418872e4b88a
Author: vozerov-gridgain 
Date:   2015-10-26T08:59:49Z

IGNITE-1770: Reader refactoring (2).

commit 5bfb9a052ddec80de52e1db639862f58ad967cd3
Author: vozerov-gridgain 
Date:   2015-10-26T09:10:22Z

IGNITE-1770: Reader refactoring (3).

commit 31945688407a87d9041b0ab90e62399feaac799d
Author: vozerov-gridgain 
Date:   2015-10-26T09:16:33Z

IGNITE-1770: Reader refactoring (3).

commit 0489b145a20cfca85e30561b783272101ad1dc68
Author: vozerov-gridgain 

[jira] [Commented] (IGNITE-1770) Portables: implement constant-time field lookup.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980643#comment-14980643
 ] 

ASF GitHub Bot commented on IGNITE-1770:


GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/183

IGNITE-1770 Portables: implement constant-time field lookup.

.NET part

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptupitsyn/ignite ignite-1770-dotnet

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/183.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #183


commit 43be01e2b81320cd4532f0f5ad9e33e7be02c083
Author: Pavel Tupitsyn 
Date:   2015-10-28T09:57:53Z

Ignite-1770 wip

commit b861086c37179ec02693a00129cb38740f5d864f
Author: Pavel Tupitsyn 
Date:   2015-10-28T10:11:09Z

Renaming

commit 4b6dc0ced524249e2bd3fceb642cc78fee082c6a
Author: Pavel Tupitsyn 
Date:   2015-10-28T10:52:40Z

Wip header reading

commit 8f00b91ae2f5dc0aaf2f5f82ddd13b0971c2665c
Author: Pavel Tupitsyn 
Date:   2015-10-28T12:03:23Z

wip

commit ed924914acef7bc154a39259ebe4313dd1c2e112
Author: Pavel Tupitsyn 
Date:   2015-10-28T12:07:53Z

wip

commit 905841892d6a2da6d09067da74675629af275455
Author: Pavel Tupitsyn 
Date:   2015-10-28T12:10:33Z

wip

commit 080578226d39b10e2a5355f0f039b52246b834e6
Author: Pavel Tupitsyn 
Date:   2015-10-28T12:58:50Z

wip writer

commit c07eae8fa8ce5bef363c93daf8aa205af59f6444
Author: Pavel Tupitsyn 
Date:   2015-10-28T13:17:29Z

wip writer

commit 4ecbbeead8eaa4b964aa6988d4c192bc5c8c4154
Author: Pavel Tupitsyn 
Date:   2015-10-28T13:37:37Z

writer done?

commit 10ae3cef15070038328f8b05e4d6508ed8214340
Author: Pavel Tupitsyn 
Date:   2015-10-28T13:42:24Z

wip

commit ecd3163efc136ca6440bdb832de5cf3b0994e69e
Author: Pavel Tupitsyn 
Date:   2015-10-28T13:48:45Z

fixing raw offset

commit cc0c81a25b95c567bc70b8acb4abfd0c7c5c5482
Author: Pavel Tupitsyn 
Date:   2015-10-28T13:53:57Z

fixing raw offset

commit 6b9d4cbd530e99d49eaf2244bc2398d79bc17263
Author: Pavel Tupitsyn 
Date:   2015-10-28T14:00:01Z

wip

commit 224aeda51c8a259b894145c374355390bff65c46
Author: Pavel Tupitsyn 
Date:   2015-10-28T14:04:15Z

reader done

commit 6191baabf9095f64d28b26bf16a931b85e5ecbc2
Author: Pavel Tupitsyn 
Date:   2015-10-28T14:11:36Z

wip

commit 76a26064ed1d457f9655641c0b3e5d24037936ba
Author: Pavel Tupitsyn 
Date:   2015-10-28T14:18:10Z

wip

commit 45c76b4c902d62638eb1f72a3a177f20fba4cf10
Author: Pavel Tupitsyn 
Date:   2015-10-28T14:22:30Z

fixing

commit d362ec0bdc02bf3e79b52dd92bc607e3c7f75a63
Author: Pavel Tupitsyn 
Date:   2015-10-28T14:27:22Z

wip

commit 2eb98b377ecbc22a5f41ee702793f1b9d559696f
Author: Pavel Tupitsyn 
Date:   2015-10-28T14:37:04Z

wip

commit b3669d3f715e03d95ad0f0fe99822cf228d4e04c
Author: Pavel Tupitsyn 
Date:   2015-10-28T14:40:39Z

wip

commit 302d94659cc1671531a22270c36b4735ef94d5d8
Author: Pavel Tupitsyn 
Date:   2015-10-28T14:50:50Z

Merge remote-tracking branch 'remotes/upstream/ignite-1770' into 
ignite-1770-dotnet

commit 1cf37391089760546d3d9f9c256f8a35a016f1dc
Author: Pavel Tupitsyn 
Date:   2015-10-28T14:52:16Z

wip

commit ec5f5f0f957bea9d577ddaab979054e9009cd063
Author: Pavel Tupitsyn 
Date:   2015-10-28T14:53:41Z

wip

commit 15473c4529de8279231901cac61d09199c860595
Author: Pavel Tupitsyn 
Date:   2015-10-28T15:29:14Z

fixing portable objects

commit d6b08850d14ba7b1854c532e7b1f3dac7a765fce
Author: Pavel Tupitsyn 
Date:   2015-10-28T15:30:41Z

Merge remote-tracking branch 'remotes/upstream/ignite-1770' into 
ignite-1770-dotnet

commit f6c91c8cfdbf5fe0888d6a9af0444d12c803f65f
Author: Pavel Tupitsyn 
Date:   2015-10-28T16:17:08Z

wip

commit 543a67ce2f8540dabf4877a09b8a9effa9a63193
Author: Pavel Tupitsyn 
Date:   2015-10-28T16:34:26Z

wip

commit ec7caecaace4bd1222abb4b1791b85fc9bda339e
Author: Pavel Tupitsyn 
Date:   2015-10-28T16:34:51Z

wip

commit c46399e14e6cb39175a3d07451fed20c01ee93b8
Author: Pavel Tupitsyn 
Date:   

[jira] [Commented] (IGNITE-1770) Portables: implement constant-time field lookup.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980642#comment-14980642
 ] 

ASF GitHub Bot commented on IGNITE-1770:


Github user ptupitsyn closed the pull request at:

https://github.com/apache/ignite/pull/182


> Portables: implement constant-time field lookup.
> 
>
> Key: IGNITE-1770
> URL: https://issues.apache.org/jira/browse/IGNITE-1770
> Project: Ignite
>  Issue Type: Task
>  Components: general, interop
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Pavel  Tupitsyn
>Priority: Blocker
> Fix For: 1.5
>
>
> See 
> https://cwiki.apache.org/confluence/display/IGNITE/Portable+object+constant-time+field+lookup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-1770) Portables: implement constant-time field lookup.

[Vladimir Ozerov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980678#comment-14980678
 ] 

Vladimir Ozerov commented on IGNITE-1770:
-

Java and .Net are ready.

> Portables: implement constant-time field lookup.
> 
>
> Key: IGNITE-1770
> URL: https://issues.apache.org/jira/browse/IGNITE-1770
> Project: Ignite
>  Issue Type: Task
>  Components: general, interop
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Pavel  Tupitsyn
>Priority: Blocker
> Fix For: 1.5
>
>
> See 
> https://cwiki.apache.org/confluence/display/IGNITE/Portable+object+constant-time+field+lookup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-1770) Portables: implement constant-time field lookup.

[Vladimir Ozerov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978530#comment-14978530
 ] 

Vladimir Ozerov commented on IGNITE-1770:
-

Java-side implementation is finished. It didn't show any speed up for now. 
Mainly because our code is not ready to take advantage of constant-field lookup 
feature. Some other optimizations are required.

> Portables: implement constant-time field lookup.
> 
>
> Key: IGNITE-1770
> URL: https://issues.apache.org/jira/browse/IGNITE-1770
> Project: Ignite
>  Issue Type: Task
>  Components: general, interop
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>Priority: Blocker
> Fix For: 1.5
>
>
> See 
> https://cwiki.apache.org/confluence/display/IGNITE/Portable+object+constant-time+field+lookup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-1770) Portables: implement constant-time field lookup.

[Vladimir Ozerov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978115#comment-14978115
 ] 

Vladimir Ozerov commented on IGNITE-1770:
-

Lets use FNV1 hashing algo as it demonstrates both good distribution and speed:
{code}
static final int FNV1_OFFSET_BASIS = 0x811C9DC5;
static final int FNV1_PRIME = 0x01000193;

int hash = FNV1_OFFSET_BASIS;

for (byte inputByte : input) {
hash = hash ^ inputByte;
hash = hash * FNV1_PRIME;
}
{code}

"input" is a sequence of field IDs.

> Portables: implement constant-time field lookup.
> 
>
> Key: IGNITE-1770
> URL: https://issues.apache.org/jira/browse/IGNITE-1770
> Project: Ignite
>  Issue Type: Task
>  Components: general, interop
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>Priority: Blocker
> Fix For: 1.5
>
>
> See 
> https://cwiki.apache.org/confluence/display/IGNITE/Portable+object+constant-time+field+lookup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-1770) Portables: implement constant-time field lookup.

[Vladimir Ozerov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978228#comment-14978228
 ] 

Vladimir Ozerov commented on IGNITE-1770:
-

I propose picking FNV1 algo for schema ID calculation. It offers pretty nice 
distribution. Using it shows minimal performance drop during object write 
(~1-2% in Java with standard Address object).

{code}
private static final int FNV1_OFFSET_BASIS = 0x811C9DC5;
private static final int FNV1_PRIME = 0x01000193;

int schemaId = FNV1_OFFSET_BASIS;

void onFieldAdded(int fieldId) {
int schemaId0 = schemaId ^ (fieldId & 0xFF);
schemaId0 = schemaId0 * FNV1_PRIME;
schemaId0 = schemaId0 ^ ((fieldId >> 8) & 0xFF);
schemaId0 = schemaId0 * FNV1_PRIME;
schemaId0 = schemaId0 ^ ((fieldId >> 16) & 0xFF);
schemaId0 = schemaId0 * FNV1_PRIME;
schemaId0 = schemaId0 ^ ((fieldId >> 24) & 0xFF);
schemaId0 = schemaId0 * FNV1_PRIME;

schemaId = schemaId0;
}
{code}

> Portables: implement constant-time field lookup.
> 
>
> Key: IGNITE-1770
> URL: https://issues.apache.org/jira/browse/IGNITE-1770
> Project: Ignite
>  Issue Type: Task
>  Components: general, interop
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>Priority: Blocker
> Fix For: 1.5
>
>
> See 
> https://cwiki.apache.org/confluence/display/IGNITE/Portable+object+constant-time+field+lookup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-1770) Portables: implement constant-time field lookup.

[Vladimir Ozerov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976513#comment-14976513
 ] 

Vladimir Ozerov commented on IGNITE-1770:
-

Final portable object structure:
0 .. 1 - Header.
1 .. 2 - Version, always 1 now.
2 .. 4 - Flags. There are two flags. "User object" - 0x1, "raw only" - 0x2. 
"Raw only" that object has no named fields and raw offset is placed on "schema 
offset" position.
4 .. 8 - Object length.
8 ..12 - Type ID.
12 .. 16 - Hash code.
16 .. 20 - Schema ID. Calculated only for reflection-based objects now. Zero in 
all other cases.
20 .. 24 - Schema offset. Contains relative offset from object head to the 
place where schema is located.

Schema is located after all object fields and raw data, and looks as follows:
[field1 ID]
[field1 offset]
[field2 ID]
[field2 offset]
...
[raw offset] - optional raw offset. Exists only if "RawWriter" was called 
during object write.


> Portables: implement constant-time field lookup.
> 
>
> Key: IGNITE-1770
> URL: https://issues.apache.org/jira/browse/IGNITE-1770
> Project: Ignite
>  Issue Type: Task
>  Components: general, interop
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>Priority: Blocker
> Fix For: 1.5
>
>
> See 
> https://cwiki.apache.org/confluence/display/IGNITE/Portable+object+constant-time+field+lookup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-1770) Portables: implement constant-time field lookup.

[Dmitriy Setrakyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977290#comment-14977290
 ] 

Dmitriy Setrakyan commented on IGNITE-1770:
---

I think that schema ID should be calculated in the same way that Java 
calculates class version IDs. They are guaranteed not to have duplicates.

> Portables: implement constant-time field lookup.
> 
>
> Key: IGNITE-1770
> URL: https://issues.apache.org/jira/browse/IGNITE-1770
> Project: Ignite
>  Issue Type: Task
>  Components: general, interop
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>Priority: Blocker
> Fix For: 1.5
>
>
> See 
> https://cwiki.apache.org/confluence/display/IGNITE/Portable+object+constant-time+field+lookup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-1770) Portables: implement constant-time field lookup.

[Vladimir Ozerov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969253#comment-14969253
 ] 

Vladimir Ozerov commented on IGNITE-1770:
-

Before digging into implementation I tried to measure optimized vs portable 
marshaller deserialization using JMH and sample "Address" class having two ints 
and two strings. Summary:

Object size:
Optimized: 60 bytes
Poratble:  108 bytes

Deserialization throughput:
Optimized: 3.0KK ops/sec
Portable:  2.6KK ops/sec

The I implemented some draft optimizations on reading side:
- Removed unnecessary ID mapper lookup on each field read;
- Optimized field seek algorithm for sequential reads;
- Delayed handles hash map creation when possible;
- Get rid of manual string lower-casing.

Results:
Optimized:   3.0KK ops/sec
Portable before: 2.6KK ops/sec
Portable after:  4.8KK ops/sec

As a next step I want to measure random field seek performance with and without 
proposed algorithm. Then I'll start implementation.

> Portables: implement constant-time field lookup.
> 
>
> Key: IGNITE-1770
> URL: https://issues.apache.org/jira/browse/IGNITE-1770
> Project: Ignite
>  Issue Type: Task
>  Components: general, interop
>Affects Versions: ignite-1.4
>Reporter: Vladimir Ozerov
>Assignee: Vladimir Ozerov
>Priority: Blocker
> Fix For: 1.5
>
>
> See 
> https://cwiki.apache.org/confluence/display/IGNITE/Portable+object+constant-time+field+lookup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)