[jira] [Commented] (CASSANDRA-14476) ShortType and ByteType are incorrectly considered variable-length types
[ https://issues.apache.org/jira/browse/CASSANDRA-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824853#comment-17824853 ] David Capwell commented on CASSANDRA-14476: --- [~jlewandowski], when I first found out about the fixed length types being variable length I wanted to fix but this trickles down to the SSTable format as well, so requires special handling to avoid upgrade/downgrades > ShortType and ByteType are incorrectly considered variable-length types > --- > > Key: CASSANDRA-14476 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14476 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Core >Reporter: Vladimir Krivopalov >Assignee: Jacek Lewandowski >Priority: Low > Labels: lhf > Fix For: 5.0.x, 5.1 > > > The AbstractType class has a method valueLengthIfFixed() that returns -1 for > data types with a variable length and a positive value for types with a fixed > length. This is primarily used for efficient serialization and > deserialization. > > It turns out that there is an inconsistency in types ShortType and ByteType > as those are in fact fixed-length types (2 bytes and 1 byte, respectively) > but they don't have the valueLengthIfFixed() method overloaded and it returns > -1 as if they were of variable length. > > It would be good to fix that at some appropriate point, for example, when > introducing a new version of SSTables format, to keep the meaning of the > function consistent across data types. Saving some bytes in serialized format > is a minor but pleasant bonus. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14476) ShortType and ByteType are incorrectly considered variable-length types
[ https://issues.apache.org/jira/browse/CASSANDRA-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824810#comment-17824810 ] Jacek Lewandowski commented on CASSANDRA-14476: --- Here it is: https://github.com/apache/cassandra/pull/3169 > ShortType and ByteType are incorrectly considered variable-length types > --- > > Key: CASSANDRA-14476 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14476 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Core >Reporter: Vladimir Krivopalov >Assignee: Jacek Lewandowski >Priority: Low > Labels: lhf > Fix For: 5.0.x, 5.1 > > > The AbstractType class has a method valueLengthIfFixed() that returns -1 for > data types with a variable length and a positive value for types with a fixed > length. This is primarily used for efficient serialization and > deserialization. > > It turns out that there is an inconsistency in types ShortType and ByteType > as those are in fact fixed-length types (2 bytes and 1 byte, respectively) > but they don't have the valueLengthIfFixed() method overloaded and it returns > -1 as if they were of variable length. > > It would be good to fix that at some appropriate point, for example, when > introducing a new version of SSTables format, to keep the meaning of the > function consistent across data types. Saving some bytes in serialized format > is a minor but pleasant bonus. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14476) ShortType and ByteType are incorrectly considered variable-length types
[ https://issues.apache.org/jira/browse/CASSANDRA-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824785#comment-17824785 ] Jacek Lewandowski commented on CASSANDRA-14476: --- I'm going to first submit some tests which will validate serialization and comparisons, as well asl backward compatibility. > ShortType and ByteType are incorrectly considered variable-length types > --- > > Key: CASSANDRA-14476 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14476 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Core >Reporter: Vladimir Krivopalov >Assignee: Jacek Lewandowski >Priority: Low > Labels: lhf > Fix For: 5.0.x, 5.1 > > > The AbstractType class has a method valueLengthIfFixed() that returns -1 for > data types with a variable length and a positive value for types with a fixed > length. This is primarily used for efficient serialization and > deserialization. > > It turns out that there is an inconsistency in types ShortType and ByteType > as those are in fact fixed-length types (2 bytes and 1 byte, respectively) > but they don't have the valueLengthIfFixed() method overloaded and it returns > -1 as if they were of variable length. > > It would be good to fix that at some appropriate point, for example, when > introducing a new version of SSTables format, to keep the meaning of the > function consistent across data types. Saving some bytes in serialized format > is a minor but pleasant bonus. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14476) ShortType and ByteType are incorrectly considered variable-length types
[
https://issues.apache.org/jira/browse/CASSANDRA-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822240#comment-17822240
]
Jacek Lewandowski commented on CASSANDRA-14476:
---
There are more problems with type compatibility:
1. Fixed length types reported as variable length: *ByteType*, *ShortType*,
*CounterColumnType*, *SimpleDateType*, *TimeType*, and types like *TupleType*,
*UserType* when all subtypes are of fixed length
2. Value compatibility issues:
* *IntegerType* should be compatible with *ShortType*, *ByteType*,
*SimpleDateType* and *TimeType* - all of them are simple integers serialized
with Big-Endian byte order
* *LongType* is compatible with *TimestampType* and *TimestampType* is
compatible with *LongType* which makes a cycle in the type compatibility
hierarchy - I don't know if it is ok because the relation
{{isValueCompatibleWith}} is used when merging data from different sources in
order to determine the resulting type. It may end up with a result depending on
the order of data sources - is it ok for compaction and querying?
* *TimeType* is compatible with *LongType*, but it should be opposite as the
*LongType* is more generic than *TimeType*
* *SimpleDateType* is compatible with *Int32Type*, but is should be opposite as
the *Int32Type* is more generic than *SimpleDateType*
3. Painful lack of tests for this stuff
---
I haven't investigated the compatibility of complex types yet
> ShortType and ByteType are incorrectly considered variable-length types
> ---
>
> Key: CASSANDRA-14476
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14476
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Vladimir Krivopalov
>Assignee: Jacek Lewandowski
>Priority: Low
> Labels: lhf
> Fix For: 5.0.x, 5.1
>
>
> The AbstractType class has a method valueLengthIfFixed() that returns -1 for
> data types with a variable length and a positive value for types with a fixed
> length. This is primarily used for efficient serialization and
> deserialization.
>
> It turns out that there is an inconsistency in types ShortType and ByteType
> as those are in fact fixed-length types (2 bytes and 1 byte, respectively)
> but they don't have the valueLengthIfFixed() method overloaded and it returns
> -1 as if they were of variable length.
>
> It would be good to fix that at some appropriate point, for example, when
> introducing a new version of SSTables format, to keep the meaning of the
> function consistent across data types. Saving some bytes in serialized format
> is a minor but pleasant bonus.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14476) ShortType and ByteType are incorrectly considered variable-length types
[
https://issues.apache.org/jira/browse/CASSANDRA-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821308#comment-17821308
]
Jacek Lewandowski commented on CASSANDRA-14476:
---
In 5.0 the problem affects more types: {{ByteType}}, {{ShortType}},
{{SimpleDateType}}, {{TimeType}}, {{TimestampType}}. I'm going to fix it and
move the original method checking for whether the type serialization is
variable or fixed length directly to {{TypeSerializer}}. I'll also provide some
upgrade tests to make sure the old sstables can be read without problems. I
don't think we need to bump SSTable version though because it does not change
anything with serialization. It may certainly break some implicit casting in
CQL though.
> ShortType and ByteType are incorrectly considered variable-length types
> ---
>
> Key: CASSANDRA-14476
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14476
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Vladimir Krivopalov
>Assignee: Jacek Lewandowski
>Priority: Low
> Labels: lhf
>
> The AbstractType class has a method valueLengthIfFixed() that returns -1 for
> data types with a variable length and a positive value for types with a fixed
> length. This is primarily used for efficient serialization and
> deserialization.
>
> It turns out that there is an inconsistency in types ShortType and ByteType
> as those are in fact fixed-length types (2 bytes and 1 byte, respectively)
> but they don't have the valueLengthIfFixed() method overloaded and it returns
> -1 as if they were of variable length.
>
> It would be good to fix that at some appropriate point, for example, when
> introducing a new version of SSTables format, to keep the meaning of the
> function consistent across data types. Saving some bytes in serialized format
> is a minor but pleasant bonus.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14476) ShortType and ByteType are incorrectly considered variable-length types
[
https://issues.apache.org/jira/browse/CASSANDRA-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888169#comment-16888169
]
Terrence Cooke commented on CASSANDRA-14476:
I noticed in FastByteOperations.java, inside the UnsafeOperations nested class,
line 146 has the following statement:
Field f = Unsafe.class.getDeclaredField("theUnsafe");
I think this was meant to be:
Field f = UnsafeOperations.class.getDeclaredField("theUnsafe");
since the class sun.misc.Unsafe has no such field but class UnsafeOperations
does.
> ShortType and ByteType are incorrectly considered variable-length types
> ---
>
> Key: CASSANDRA-14476
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14476
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Vladimir Krivopalov
>Assignee: Jearvon Dharrie
>Priority: Low
> Labels: lhf
>
> The AbstractType class has a method valueLengthIfFixed() that returns -1 for
> data types with a variable length and a positive value for types with a fixed
> length. This is primarily used for efficient serialization and
> deserialization.
>
> It turns out that there is an inconsistency in types ShortType and ByteType
> as those are in fact fixed-length types (2 bytes and 1 byte, respectively)
> but they don't have the valueLengthIfFixed() method overloaded and it returns
> -1 as if they were of variable length.
>
> It would be good to fix that at some appropriate point, for example, when
> introducing a new version of SSTables format, to keep the meaning of the
> function consistent across data types. Saving some bytes in serialized format
> is a minor but pleasant bonus.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14476) ShortType and ByteType are incorrectly considered variable-length types
[ https://issues.apache.org/jira/browse/CASSANDRA-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742542#comment-16742542 ] Ariel Weisberg commented on CASSANDRA-14476: I think the thing to do there is have valueLengthIfFixed accept a version for, and depending on the version return a value that reflects whether it is before or after whatever we put in 4.0. You will have to make sure the version is what we expected because we have multiple versions, streaming version, messaging service version, sstable format version and inside a serializer/deserializer we have to have the right one. > ShortType and ByteType are incorrectly considered variable-length types > --- > > Key: CASSANDRA-14476 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14476 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Core >Reporter: Vladimir Krivopalov >Assignee: Jearvon Dharrie >Priority: Minor > Labels: lhf > > The AbstractType class has a method valueLengthIfFixed() that returns -1 for > data types with a variable length and a positive value for types with a fixed > length. This is primarily used for efficient serialization and > deserialization. > > It turns out that there is an inconsistency in types ShortType and ByteType > as those are in fact fixed-length types (2 bytes and 1 byte, respectively) > but they don't have the valueLengthIfFixed() method overloaded and it returns > -1 as if they were of variable length. > > It would be good to fix that at some appropriate point, for example, when > introducing a new version of SSTables format, to keep the meaning of the > function consistent across data types. Saving some bytes in serialized format > is a minor but pleasant bonus. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14476) ShortType and ByteType are incorrectly considered variable-length types
[ https://issues.apache.org/jira/browse/CASSANDRA-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734485#comment-16734485 ] Jearvon Dharrie commented on CASSANDRA-14476: - If this is still needed I would like to work on it. Can it be assigned to me please? Thanks! > ShortType and ByteType are incorrectly considered variable-length types > --- > > Key: CASSANDRA-14476 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14476 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Core >Reporter: Vladimir Krivopalov >Priority: Minor > Labels: lhf > > The AbstractType class has a method valueLengthIfFixed() that returns -1 for > data types with a variable length and a positive value for types with a fixed > length. This is primarily used for efficient serialization and > deserialization. > > It turns out that there is an inconsistency in types ShortType and ByteType > as those are in fact fixed-length types (2 bytes and 1 byte, respectively) > but they don't have the valueLengthIfFixed() method overloaded and it returns > -1 as if they were of variable length. > > It would be good to fix that at some appropriate point, for example, when > introducing a new version of SSTables format, to keep the meaning of the > function consistent across data types. Saving some bytes in serialized format > is a minor but pleasant bonus. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
