[
https://issues.apache.org/jira/browse/SPARK-19180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15819400#comment-15819400
]
yucai commented on SPARK-19180:
-------------------------------
Hi Owen,
Thanks a lot for comments, it is using unsafe API for OffHeapColumn, which
should have no align.
See codes:
{code}
@Override
public void putShorts(int rowId, int count, short value) {
long offset = data + 2 * rowId;
- for (int i = 0; i < count; ++i, offset += 4) {
+ for (int i = 0; i < count; ++i, offset += 2) {
Platform.putShort(null, offset, value);
}
}
{code}
And also, my testing:
{code}
scala> val column = ColumnVector.allocate(1024, ShortType, MemoryMode.OFF_HEAP)
column: org.apache.spark.sql.execution.vectorized.ColumnVector =
org.apache.spark.sql.execution.vectorized.OffHeapColumnVector@56fc2cea
scala> column.putShorts(0, 4, 8.toShort)
scala> column.getShort(1)
res5: Short = 18432
scala>
scala> val column = ColumnVector.allocate(1024, ShortType, MemoryMode.ON_HEAP)
column: org.apache.spark.sql.execution.vectorized.ColumnVector =
org.apache.spark.sql.execution.vectorized.OnHeapColumnVector@7fb8d720
scala> column.putShorts(0, 4, 8.toShort)
scala> column.getShort(1)
res7: Short = 8
{code}
> the offset of short is 4 in OffHeapColumnVector's putShorts
> -----------------------------------------------------------
>
> Key: SPARK-19180
> URL: https://issues.apache.org/jira/browse/SPARK-19180
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: yucai
> Fix For: 2.2.0
>
>
> the offset of short is 4 in OffHeapColumnVector's putShorts, actually it
> should be 2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
