[jira] [Updated] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15180: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the reviews and great suggestions > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15180.patch, HBASE-15180_V2.patch, > HBASE-15180_V4.patch, HBASE-15180_V6.patch, HBASE-15180_V7.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15180: --- Attachment: HBASE-15180_V7.patch > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15180.patch, HBASE-15180_V2.patch, > HBASE-15180_V4.patch, HBASE-15180_V6.patch, HBASE-15180_V7.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15180: --- Attachment: HBASE-15180_V6.patch > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15180.patch, HBASE-15180_V2.patch, > HBASE-15180_V4.patch, HBASE-15180_V6.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15180: --- Fix Version/s: (was: 1.2.1) (was: 1.3.0) > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15180.patch, HBASE-15180_V2.patch, > HBASE-15180_V4.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15180: --- Attachment: HBASE-15180_V4.patch Client side Codec also making RPC response buffer backed direct Cells. Patch for ref.. Can change this client side Cells sharing the response buffer. > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15180.patch, HBASE-15180_V2.patch, > HBASE-15180_V4.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15180: --- Affects Version/s: 0.98.0 > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15180.patch, HBASE-15180_V2.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15180: --- Fix Version/s: 1.2.1 1.3.0 > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15180.patch, HBASE-15180_V2.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15180: --- Attachment: HBASE-15180_v2.patch There was an issue in CellReadableByteArrayInputStream#readCell(int length, boolean withTags) where not advancing the position in BAIS after Cell instance is created. Thanks [~ram_krish] for noticing this. > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15180.patch, HBASE-15180_v2.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15180: --- Attachment: HBASE-15180_V2.patch > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15180.patch, HBASE-15180_V2.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15180: --- Attachment: (was: HBASE-15180_v2.patch) > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15180.patch, HBASE-15180_V2.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15180: --- Attachment: HBASE-15180.patch > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15180.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15180: --- Status: Patch Available (was: Open) > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15180.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
