[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506977#comment-16506977
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


Moved status to "Open", because we are waiting for a contributor who is able to 
implement CPP part.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: binary
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>Priority: Major
>  Labels: iep-2, performance
> Fix For: 2.6
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433800#comment-16433800
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


Moving to 2.6 because CPP part is not finished.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: binary
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>Priority: Major
>  Labels: iep-2, performance
> Fix For: 2.6
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Igor Sapego (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343237#comment-16343237
 ] 

Igor Sapego commented on IGNITE-5097:
-

[~daradurvs], as environment variable backward-compatibility approach has been 
rejected, there is much more effort needed now to implement it on C++ side now 
and it is harder for me now to find timespan long enough for that. I do 
remember about this task though.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: binary
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>Priority: Major
>  Labels: iep-2, performance
> Fix For: 2.5
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343056#comment-16343056
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


Hi, [~vozerov], [~isapego], could you please review Java part changes and port 
the changes to the C++ platform?

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: binary
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>Priority: Major
>  Labels: iep-2, performance
> Fix For: 2.5
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196825#comment-16196825
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


Hi, [~vozerov], [~isapego], could you please review Java part changes and port 
the changes to the C++ platform?

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: binary
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: iep-2, performance
> Fix For: 2.4
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vladimir Ozerov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185819#comment-16185819
 ] 

Vladimir Ozerov commented on IGNITE-5097:
-

Moving to 2.4 due to huge impact and additional changes required on JDBC/ODBC 
sides.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: binary
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: iep-2, performance
> Fix For: 2.4
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166495#comment-16166495
 ] 

Pavel Tupitsyn commented on IGNITE-5097:


[~daradurvs] looks good to me.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.3
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166373#comment-16166373
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


[~ptupitsyn], thank you for the notes!
I've fixed code style notes and have removed unused import.
If you have a minute, please, check it once again. Thanks!

bq. 4) Could not compile Java (mvn clean package)
It's not related to my changes and as I can see the compilation is already 
fixed in the master branch by [a 
commit|https://github.com/apache/ignite/commit/e4e0a280853b896f32ae3d85545f83faeb7aa680].

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.3
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166170#comment-16166170
 ] 

Pavel Tupitsyn commented on IGNITE-5097:


[~daradurvs]:
1) Make sure code lines fit within 120 chars (BinaryConfiguration, 
BinaryObjectBuilder, etc)
2) Method comments should be in "///" format, not "/**" (BinaryUtils). I know 
there is some old code with wrong format, but all new/updated methods should 
have proper format.
3) Unused import in BinaryEqualityComparerTest
4) Could not compile Java (mvn clean package):
{code}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile 
(default-testCompile) on project ignite-indexing: Compilation failure: Compilat
ion failure:
[ERROR] 
/C:/w/incubator-ignite/modules/indexing/src/test/java/org/apache/ignite/internal/processors/cache/index/H2DynamicColumnsAbstractBasicSelfTest.java:[272,13]
 reference
 to assertEquals is ambiguous
[ERROR] both method assertEquals(java.lang.Object,java.lang.Object) in 
junit.framework.TestCase and method assertEquals(int,int) in 
junit.framework.TestCase match
[ERROR] 
/C:/w/incubator-ignite/modules/indexing/src/test/java/org/apache/ignite/internal/processors/cache/index/H2DynamicColumnsAbstractBasicSelfTest.java:[275,13]
 reference
 to assertEquals is ambiguous
[ERROR] both method assertEquals(java.lang.Object,java.lang.Object) in 
junit.framework.TestCase and method assertEquals(int,int) in 
junit.framework.TestCase match
{code}

Other than that - good job!

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.3
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166042#comment-16166042
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


[~ptupitsyn], could you please review the new changes in .NET part?

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.3
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165876#comment-16165876
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


[~isapego], could you please port the new changes to the CPP platform?

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.3
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165858#comment-16165858
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


[~vozerov],
bq. It should be set to false by default, so that data saved on previous 
version of Ignite can be read without problems.
I agree it makes sense in such case.

I've implemented configuration parameter on {{BinaryConfiguration}}, in Java 
and .NET parts.

Sent to [ci.tests|https://ci.ignite.apache.org/viewQueued.html?itemId=828125] 
once again, after code cleanup.

Could, you please review it?

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.3
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Igor Sapego (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148761#comment-16148761
 ] 

Igor Sapego commented on IGNITE-5097:
-

[~daradurvs], TC showed one more minor issue. Please, merge changes from my 
branch again.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.3
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vladimir Ozerov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148755#comment-16148755
 ] 

Vladimir Ozerov commented on IGNITE-5097:
-

[~daradurvs],
Legacy features is part of normal product lifecycle. System properties are not 
convenient form UX perspective. Hence, we should use configuration parameter. 
It should be set to {{false}} by default, so that data saved on previous 
version of Ignite can be read without problems.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.3
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147583#comment-16147583
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


[~isapego], thanks a lot! 
Rerun [ci.tests|https://ci.ignite.apache.org/viewQueued.html?itemId=801015].

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.3
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Igor Sapego (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147544#comment-16147544
 ] 

Igor Sapego commented on IGNITE-5097:
-

[~daradurvs], I've fixed issues. Please, re-merge changes from my ticket.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.3
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Igor Sapego (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147462#comment-16147462
 ] 

Igor Sapego commented on IGNITE-5097:
-

[~daradurvs], TC shows compilation errors. I'll take a look.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.3
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147386#comment-16147386
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


Hi, [~isapego], I've merged your changes to the common PR.
Merge conflicts with the master-branch were resolved, please check it yourself. 
Thanks!
Sent to [ci.tests|https://ci.ignite.apache.org/viewQueued.html?itemId=800239].

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.3
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147239#comment-16147239
 ] 

ASF GitHub Bot commented on IGNITE-5097:


GitHub user daradurvs opened a pull request:

https://github.com/apache/ignite/pull/2549

IGNITE-5097 BinaryMarshaller should write ints in "varint" encoding where 
it makes sense



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/daradurvs/ignite ignite-5097-release

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2549.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2549


commit 779b607a783bbbedca7b9b42a915e968aa4027eb
Author: Vyacheslav Daradur 
Date:   2017-08-30T11:40:59Z

IGNITE-5097 BinaryMarshaller should write ints in "varint" encoding where 
it makes sense

commit aa5f957c9a01dcb962b1df6be2f5ba8db94e491a
Author: Igor Sapego 
Date:   2017-08-30T13:16:06Z

IGNITE-5153 CPP: Introduced varint encoding in C++




> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.3
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147240#comment-16147240
 ] 

ASF GitHub Bot commented on IGNITE-5097:


Github user daradurvs closed the pull request at:

https://github.com/apache/ignite/pull/2043


> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.3
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120600#comment-16120600
 ] 

ASF GitHub Bot commented on IGNITE-5097:


Github user daradurvs closed the pull request at:

https://github.com/apache/ignite/pull/1902


> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.2
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120271#comment-16120271
 ] 

ASF GitHub Bot commented on IGNITE-5097:


GitHub user daradurvs opened a pull request:

https://github.com/apache/ignite/pull/2425

IGNITE-5097 BinaryMarshaller should write ints in "varint" encoding where 
it makes sense



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/daradurvs/ignite ignite-5736-module

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2425.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2425


commit efbac32e650d603e18ea405bee626589ac237f61
Author: Vyacheslav Daradur 
Date:   2017-08-09T14:05:29Z

IGNITE-5732 Provide API to test compatibility with old releases (#20)

commit 0a7ffcce6527111a68568da0a6d22d4a1eb306a4
Author: Vyacheslav Daradur 
Date:   2017-08-09T14:09:08Z

IGNITE-5097 BinaryMarshaller should write ints in "varint" encoding where 
it makes sense (#21)

commit 6b34d93609bc2087b77d7b44a0665ea160965e4e
Author: Vyacheslav Daradur 
Date:   2017-08-09T17:08:50Z

ignite-5736: added VarintArraysSizeCompatibilityTest




> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.2
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Anton Vinogradov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089914#comment-16089914
 ] 

Anton Vinogradov commented on IGNITE-5097:
--

[~vozerov], 

I don't think we should extend {{BinaryConfiguration}} API by legacy support 
flags. 
We'll get rid of this compatibility support next major release.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.2
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089542#comment-16089542
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


[~vozerov], why do you think that?

As I understand the writing in varint - will be the main approach in future 
releases.
May be this task is part of design of a global feature?

BTW: Are there clear ways how to manage via BinaryConfiguration the static 
methods in the BinaryUtils?

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.2
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vladimir Ozerov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087710#comment-16087710
 ] 

Vladimir Ozerov commented on IGNITE-5097:
-

[~daradurvs], [~avinogradov],
Folks, I think it makes sense to use configuration parameter on 
{{BinaryConfiguration}} instead of system variable.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.2
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081832#comment-16081832
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


Latest 
[ci.tests|http://ci.ignite.apache.org/viewLog.html?buildId=716871=buildResultsDiv=Ignite20Tests_RunAll].

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.2
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080066#comment-16080066
 ] 

Pavel Tupitsyn commented on IGNITE-5097:


.NET part looks good to me.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.2
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080019#comment-16080019
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


I've added compatibility mode, as it had been told by [~avinogradov].

[ci.tests|http://ci.ignite.apache.org/viewLog.html?buildId=713851=buildResultsDiv=Ignite20Tests_RunAll]
 looks good in scope of the binary infrastructure and dotNET integration, but 
there are some issues with cluster build in another test-plans.
Sent to [ci.tests|http://ci.ignite.apache.org/viewQueued.html?itemId=714517] 
once again.

[~vozerov], [~ptupitsyn], please review the changes.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.2
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072717#comment-16072717
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


[~isapego],
I've rebased the branch.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.2
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Igor Sapego (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072701#comment-16072701
 ] 

Igor Sapego commented on IGNITE-5097:
-

[~daradurvs],
Also, please merge changes from master, cause it's hard to implement C++ on top 
of your ticket, cause its code base is outdated.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.2
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072682#comment-16072682
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


[~avinogradov],
Makes sense. Thank you for the note.
I've created a sub-task: IGNITE-5676
I'll implement it soon.  

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.2
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Anton Vinogradov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072676#comment-16072676
 ] 

Anton Vinogradov commented on IGNITE-5097:
--

[~daradurvs], 
seems you changing storage format, so it will be nice if you'll provide 
compatibility property to allow to keep data in old format. 

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.2
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Igor Sapego (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036800#comment-16036800
 ] 

Igor Sapego commented on IGNITE-5097:
-

[~daradurvs], yeah, I'm going to do it in the nearest time.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036753#comment-16036753
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


[~isapego], as far as I know, you've already worked on the task from the 
beginning.   
Could you port the changes to the CPP platform?

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036721#comment-16036721
 ] 

Pavel Tupitsyn commented on IGNITE-5097:


Got it. Sounds good to me then.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036716#comment-16036716
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


bq. Not sure though about int<->uint casts. 
It is equivalent of {{>>>}} unsigned right bit-shift operator in Java
This is necessary in the case of negative numbers, otherwise it may provide 
infinite loop. Covered by added unit test.
bq. 1) Does this work for negative values?
Yes it works for negative values, but we shouldn't use it for, because negative 
values always take 5 bytes.
bq. 2) Since we use this thing purely for positive values, should we use uint 
in method signature?
Not sure. I'd prefer not to use {{uint}} for compliance with the Java 
implementation.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036696#comment-16036696
 ] 

Pavel Tupitsyn commented on IGNITE-5097:


[~daradurvs], .NET changes look good to me in general, thank you.
Not sure though about {{int}}<->{{uint}} casts. 
1) Does this work for negative values?
2) Since we use this thing purely for positive values, should we use {{uint}} 
in method signature?

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036396#comment-16036396
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


I've ported the code to .NET platform.

[~ptupitsyn], please review the changes.

[ci.tests|http://ci.ignite.apache.org/viewQueued.html?itemId=646242] 

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033515#comment-16033515
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


[~vozerov], thanks for your notes.
bq. doWriteUnsignedVarint should have safe and unsafe implementations
Great idea!
bq. looks like DECIMAL is read incorrectly here
Fixed.
bq. Looks like varint applied to all arrays except of ENUM_ARR. Why?
Hmm, it has been applied already. I mean BinaryWriterExImpl#doWriteEnumArray 
and BinaryUtils#doReadBinaryEnumArray.

I've fixed that.
Sent to [ci.test|http://ci.ignite.apache.org/viewQueued.html?itemId=643274]

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vladimir Ozerov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033134#comment-16033134
 ] 

Vladimir Ozerov commented on IGNITE-5097:
-

[~daradurvs], in general fix looks good. Several comments:
1) {{BinaryWriterExImpl}} - arguments passed to {{unsafeEnsure}} must be 
changed from {{1 + 4}} to {{1 + 5}}, {{doWriteUnsignedVarint}} should have safe 
and unsafe implementations. Unsafe implementation should be used when 
{{unsafeEnsure}} is called before method call. Unsafe implementation should use 
{{out.unsafeWriteByte}}
2) {{BinaryFieldImpl.readField}} - looks like {{DECIMAL}} is read incorrectly 
here
3) Looks like varint applied to all arrays except of ENUM_ARR. Why?

Once fixed, we should ask .NET and CPP experts to port this code to these 
platforms.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030867#comment-16030867
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


I've prepared separate PR where only arrays lengths are affected:
https://github.com/apache/ignite/pull/2043

Locally, tests in IgniteBinaryObjectsTestSuite are ok.
Sent to [ci.tests|http://ci.ignite.apache.org/viewQueued.html?itemId=638454]

For me, there is 2 controversial issues:
1). Extend inteface with 'varint' methods OR just add new write-read methods.
I chose second approach, because it is used in other places in project. In 
addition, we use 'varint' for special cases, there is no sense to extend 
interface with it.
2). Implementation of sizeInVarint method, there is several ways to implement 
it.
I've prepared benchmark, looks like current implementation is a bit faster.
{code:title=Benchmark}
import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.Param;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

public class VarintSizeBenchmark {
public static void main(String[] args) throws RunnerException {
Options opt = new OptionsBuilder()
.include(VarintSizeBenchmark.class.getSimpleName())
.mode(Mode.AverageTime)
.timeUnit(TimeUnit.NANOSECONDS)
.warmupIterations(10)
.measurementIterations(30)
.forks(1)
.jvmArgs("-ea", "-Xms4g", "-Xmx4g")
.shouldFailOnError(true)
.build();

new Runner(opt).run();
}

@State(Scope.Thread)
public static class ValueState {
@Param({"1", "10", "100", "1000", "10200", "102000", "1203000", 
"10203000", "200304000", "3"})
int i;
}

@Benchmark
public int sizeInUnsignedVarint(ValueState state) {
int val = state.i;
int size = 1;

while ((val & 0xFF80) != 0L) {
val >>>= 7;
size++;
}

return size;
}

@Benchmark
public int sizeInUnsignedVarint2(ValueState state) {
int val = state.i;

if (val < 0)
return 5;

if (val <= Byte.MAX_VALUE)
return 1;

if (val <= 16383)
return 2;

if (val <= 2097151)
return 3;

if (val <= 268435455)
return 4;

return 5;
}
}
{code}
{code:title=Result}
Benchmark(i)  Mode  Cnt  Score   Error  
Units
VarintSizeBenchmark.sizeInUnsignedVarint   1  avgt   30  2,326 ± 0,042  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint  10  avgt   30  2,301 ± 0,013  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint 100  avgt   30  2,296 ± 0,008  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint1000  avgt   30  2,556 ± 0,017  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint   10200  avgt   30  2,570 ± 0,044  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint  102000  avgt   30  3,534 ± 0,021  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint 1203000  avgt   30  3,553 ± 0,075  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint10203000  avgt   30  3,569 ± 0,015  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint   200304000  avgt   30  3,564 ± 0,014  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint   3  avgt   30  3,843 ± 0,005  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint2  1  avgt   30  2,075 ± 0,068  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint2 10  avgt   30  2,032 ± 0,013  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint2100  avgt   30  2,032 ± 0,013  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint2   1000  avgt   30  2,292 ± 0,017  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint2  10200  avgt   30  2,287 ± 0,011  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint2 102000  avgt   30  2,285 ± 0,007  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint21203000  avgt   30  2,285 ± 0,007  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint2   10203000  avgt   30  2,286 ± 0,010  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint2  200304000  avgt   30  2,346 ± 0,087  
ns/op
VarintSizeBenchmark.sizeInUnsignedVarint2  3  avgt   30  2,285 ± 0,009  
ns/op
{code}

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030851#comment-16030851
 ] 

ASF GitHub Bot commented on IGNITE-5097:


GitHub user daradurvs opened a pull request:

https://github.com/apache/ignite/pull/2043

IGNITE-5097: writing arrays length in varint encoding was implemented



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/daradurvs/ignite ignite-5097_2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2043.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2043


commit 5af11ed85fe033ab2d4b70de7b44c584acb376d8
Author: Vyacheslav Daradur 
Date:   2017-05-31T08:41:56Z

ignite-5097: writing arrays length in varint encoding was implemented




> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029510#comment-16029510
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


[~vozerov], what about Decimal's bytes array?
Decimal is represented as: 1 byte(flag) + 4 bytes(scale) + bytes array 
(magnitude)

In some places Decimal has the same logic with bytes arrays, for example in 
BinarySerializedFieldComparator.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028566#comment-16028566
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


Ok, I'll rollback incompatible changes shortly.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Igor Sapego (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028478#comment-16028478
 ] 

Igor Sapego commented on IGNITE-5097:
-

Well, OK, it's all ready for C++ already then. Waiting for Java part.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vladimir Ozerov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028455#comment-16028455
 ] 

Vladimir Ozerov commented on IGNITE-5097:
-

[~isapego], yes, it makes sense for strings as well.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Igor Sapego (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028239#comment-16028239
 ] 

Igor Sapego commented on IGNITE-5097:
-

[~vozerov], what about strings?

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vladimir Ozerov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028235#comment-16028235
 ] 

Vladimir Ozerov commented on IGNITE-5097:
-

I would only do that for arrays. Applying varlens to enums and decimals would 
make them hard to analyze and parse and hence may cause slowdown in SQL and 
sereliazation and deserialization.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028233#comment-16028233
 ] 

Pavel Tupitsyn commented on IGNITE-5097:


I agree.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028232#comment-16028232
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


[~isapego], [~ptupitsyn], [~vozerov]

Let's describe discussion result:
We can use varint encoding to following cases (in the all platforms):
1) size of all arrays 
2) BigDecimal: scale and mag
3) Strings: chars array length
4) Enum ordinals

Do you agree?

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Igor Sapego (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026571#comment-16026571
 ] 

Igor Sapego commented on IGNITE-5097:
-

I mean only sizes of course.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026565#comment-16026565
 ] 

Pavel Tupitsyn commented on IGNITE-5097:


[~isapego] I agree. 

> primitive arrays
You mean array size only, or also values?

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Igor Sapego (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026554#comment-16026554
 ] 

Igor Sapego commented on IGNITE-5097:
-

[~daradurvs], [~ptupitsyn], so what is the decision, guys? I propose to only 
use varints in primitive arrays, strings, BigDecimals and Enum ordinals, where 
they could be most useful and would not interfere with platforms 
implementations.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019533#comment-16019533
 ] 

Pavel Tupitsyn commented on IGNITE-5097:


Shifting entire stream is not an option, I think. Too expensive.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019517#comment-16019517
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


bq. I pretty much sure, it's expensive in Java too, isn't it? 
Yes, it is expensive in Java too (for large arrays).

bq. serialization to binary format is implemented using streams
In that case we need to define which parts can be optimized in all platfroms.
And then I'll rollback incompatible changes.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019512#comment-16019512
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


bq. I pretty much sure, it's expensive in Java too, isn't it? 
Yes, it is very expensive in Java too.

bq. serialization to binary format is implemented using streams
In that case we need to define which parts can be optimized in all platfroms.
And then I'll rollback incompatible changes.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Igor Sapego (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019507#comment-16019507
 ] 

Igor Sapego commented on IGNITE-5097:
-

{quote}
Is resize-array or shift operations "expensive" in platfroms?
{quote}
I pretty much sure, it's expensive in Java too, isn't it? Moving large amounts 
of data is expensive. Moreover, currently, serialization to binary format is 
implemented using streams, which only imply sequent writes and does not provide 
any API to make shifts in underlying memory.
[~ptupitsyn], [~vozerov], what do you think?

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019501#comment-16019501
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


bq. the max size is going to take 5 bytes then
Yes, Integer.MAX_VALUE will took 5 bytes.

I've prepared the tests to show size limits:
BinaryUtilsSelfTest#testSizeInUnsignedVarint
BinaryUtilsSelfTest#testSizeInSignedVarint
Please, take a look:
https://github.com/apache/ignite/pull/1902/files#diff-3f0e73696aef50d25ad791438f2625f0R82

{quote}
There is an issue with implementation for Collections, Maps and Arrays of 
objects - in C++ length of these types implemented in 3 steps:
1. Open writing session;
2. Write objects one by one;
3. Close session.
{quote}
Is resize-array or shift operations "expensive" in platfroms?
Can you provide some extra step after a session closing?
Just for example:
4. Write an array size to reserved place and shift the data.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012416#comment-16012416
 ] 

Pavel Tupitsyn commented on IGNITE-5097:


[~isapego]
>  write length in varint, always using 4 bytes
Yep, that's what I meant above.

I think varlen makes sense for collection sizes. Things like {{string[]}}, 
{{int[]}}, {{byte[]}} can be quite common, and not necessary as an object field.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Igor Sapego (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012325#comment-16012325
 ] 

Igor Sapego commented on IGNITE-5097:
-

Of course, we can still write length in varint, always using 4 bytes, often 
leaving first 3 bytes to be {{0x80}}. I'm just not sure, if this makes sense. 
If you consider that collection mostly containing objects, it means that most 
collections would take pretty much memory (single object header is 24 bytes 
long). So, does 3 at most bytes really make decent traffic economy in case of 
collections?

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Igor Sapego (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012281#comment-16012281
 ] 

Igor Sapego commented on IGNITE-5097:
-

In Java collection are being written in *one* call, not in several steps, and 
with known size, so there is no such problem in Java.

{quote}
I guess we shoud just use varlen when possible, and reserve 4 bytes otherwise.
{quote}
The problem is, varint and int32 is not compatible.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Igor Sapego (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010578#comment-16010578
 ] 

Igor Sapego commented on IGNITE-5097:
-

Hi, guys,
There is an issue with implementation for Collections, Maps and Arrays of 
objects - in C++ length of these types implemented in 3 steps:
1. Open writing session;
2. Write objects one by one;
3. Close session.

As you can see, the length of the of these types is unknown upon session 
opening. Previously we were just reserving 4 bytes for length and were writing 
it on session closing. Now we can't know how much bytes to reserve, as the 
length field is now of a variable length.

[~daradurvs], [~vozerov], [~ptupitsyn], what do you guys think? Do we really 
need this optimisation for map/collection/object array?

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Igor Sapego (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006158#comment-16006158
 ] 

Igor Sapego commented on IGNITE-5097:
-

Well, on the second thought, there will be always about 6 bytes when used for 
time types, which makes it not very profitable (only saving 2 bytes per value) 
but could badly slow down marshalling. So, I guess, we should not implement 
that.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006153#comment-16006153
 ] 

Pavel Tupitsyn commented on IGNITE-5097:


For {{UUID}} it does not make sense, since it is a pair of large values at most 
cases.

As for {{Timestamp}}, I would at least write a microbenchmark to test how 
current implementation compares with varint on values which represent current 
time.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006066#comment-16006066
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


[~isapego], 

I didn't implement it, because it isn't described in the task.

If we implement a writing long/int64 in varint encoding, we will be able to use 
it at serializing for following types: {{Long}} (1 long), {{Date}} (1 long), 
{{Time}} (1 long), {{Timestamp}} (1 long), {{UUID}} (2 longs).

I think there will be less proffit than in implemented varint, because long 
values usualy is big. ({{Date}}, {{Time}}, {{Timestamp}}, {{UUID}}).
But it will allow us to use memory more effectively, it is always cool)

[~vozerov], [~ptupitsyn], any thougths?

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Igor Sapego (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005097#comment-16005097
 ] 

Igor Sapego commented on IGNITE-5097:
-

[~daradurvs], what about {{Date}} and {{Timestamp}} types? It seems like about 
2 bytes can be saved for them if we'll use varint64 for them. 

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995157#comment-15995157
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


[Varint encoding 
description.|https://developers.google.com/protocol-buffers/docs/encoding?csw=1#varints]

Implemented:
1) all arrays / collections / maps lengths
2) BigDecimal: scale and mag
3) Strings: chars array length
4) Enum ordinals

Locally, tests in IgniteBinaryObjectsTestSuite are ok.

Sent to [ci.tests|http://ci.ignite.apache.org/viewQueued.html?itemId=598270]

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995144#comment-15995144
 ] 

ASF GitHub Bot commented on IGNITE-5097:


GitHub user daradurvs opened a pull request:

https://github.com/apache/ignite/pull/1902

IGNITE-5097



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/daradurvs/ignite ignite-5097

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1902.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1902


commit 8156af3f7c9d2e05bda0dd19522af1418987886a
Author: Vyacheslav Daradur 
Date:   2017-04-28T11:11:29Z

ignite-5097: wip

commit 49f9019abfc495fc5ffaf8fe53302e8de735489e
Author: Vyacheslav Daradur 
Date:   2017-04-28T13:47:10Z

ignite-5097: writing string with varint was implemented

commit 9907d647dce33413bd1e3cedeb1efa6bdf4b8dd9
Author: Vyacheslav Daradur 
Date:   2017-04-28T13:55:10Z

ignite-5097: writing enum ordinals in varint was implemented

commit 6a7349a50e173ac964ac60e2b89da272c213c293
Author: Vyacheslav Daradur 
Date:   2017-04-28T14:23:32Z

ignite-5097: writing of decimal scale and bytes array length in varint 
format was implemented

commit c3ee1e2d54ba428a5acd7007ca309089849a877d
Author: Vyacheslav Daradur 
Date:   2017-05-02T14:44:46Z

ignite-5097: writing array/collection/map lengths in varint encoding was 
implemented

commit fba5e97ed7515718d2d87573b0059d14ea61d01e
Author: Vyacheslav Daradur 
Date:   2017-05-03T08:30:24Z

ignite-5097: refactoring

commit 6166519ed06a1d30da15046792e93a15f9bde528
Author: Vyacheslav Daradur 
Date:   2017-05-03T11:51:04Z

ignite-5097: refactoring; tests for sizeof-methods were added

commit d75dda28559926db1422b2c2f28c6b8d32290c01
Author: Vyacheslav Daradur 
Date:   2017-05-03T12:29:53Z

ignite-5097: fixes

commit 9707585a3c2b82fb03e460eee6a49b0c920acc93
Author: Vyacheslav Daradur 
Date:   2017-05-03T15:38:24Z

ignite-5097: fixes part 2




> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Vyacheslav Daradur (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15986291#comment-15986291
 ] 

Vyacheslav Daradur commented on IGNITE-5097:


[~isapego] I'll create them after implementation in Java.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-5097) BinaryMarshaller should write ints in "varint" encoding where it makes sense

[Igor Sapego (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15986266#comment-15986266
 ] 

Igor Sapego commented on IGNITE-5097:
-

This will affect platforms, so we need subtickets for C++ and .NET once the 
scope is defined.

> BinaryMarshaller should write ints in "varint" encoding where it makes sense
> 
>
> Key: IGNITE-5097
> URL: https://issues.apache.org/jira/browse/IGNITE-5097
> Project: Ignite
>  Issue Type: Task
>  Components: general
>Affects Versions: 2.0
>Reporter: Vladimir Ozerov
>Assignee: Vyacheslav Daradur
>  Labels: important, performance
> Fix For: 2.1
>
>
> There are a lot of places in the code where we write integers for some 
> special purposes. Quite often their value will be vary small, so that 
> applying "varint" format could save a lot of space at the cost of very low 
> additional CPU overhead. 
> Specifically:
> 1) Array/collection/map lengths
> 2) BigDecimal's (usually will save ~6 bytes)
> 3) Strings
> 4) Enum ordinals



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)