[jira] [Updated] (HBASE-6868) Skip checksum is broke; are we double-checksumming by default?
[ https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6868: - Fix Version/s: (was: 0.95.0) 0.94.2 Fix up after bulk move overwrote some 0.94.2 fix versions w/ 0.95.0 (Noticed by Lars Hofhansl) Skip checksum is broke; are we double-checksumming by default? -- Key: HBASE-6868 URL: https://issues.apache.org/jira/browse/HBASE-6868 Project: HBase Issue Type: Bug Components: HFile, wal Affects Versions: 0.94.0, 0.94.1 Reporter: LiuLei Assignee: Lars Hofhansl Priority: Blocker Fix For: 0.94.2 Attachments: 6868-0.94.txt, 6868-0.96-idea.txt, 6868-0.96-v2.txt, 6868-0.96-v3.txt The HFile contains checksums for decrease the iops, so when Hbase read HFile , that dont't need to read the checksum from meta file of HDFS. But HLog file of Hbase don't contain the checksum, so when HBase read the HLog, that must read checksum from meta file of HDFS. We could add setSkipChecksum per file to hdfs or we could write checksums into WAL if this skip checksum facility is enabled -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6868) Skip checksum is broke; are we double-checksumming by default?
[ https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6868: - Status: Patch Available (was: Open) Skip checksum is broke; are we double-checksumming by default? -- Key: HBASE-6868 URL: https://issues.apache.org/jira/browse/HBASE-6868 Project: HBase Issue Type: Bug Components: HFile, wal Affects Versions: 0.94.1, 0.94.0 Reporter: LiuLei Priority: Blocker Fix For: 0.94.3, 0.96.0 Attachments: 6868-0.96-idea.txt, 6868-0.96-v2.txt, 6868-0.96-v3.txt The HFile contains checksums for decrease the iops, so when Hbase read HFile , that dont't need to read the checksum from meta file of HDFS. But HLog file of Hbase don't contain the checksum, so when HBase read the HLog, that must read checksum from meta file of HDFS. We could add setSkipChecksum per file to hdfs or we could write checksums into WAL if this skip checksum facility is enabled -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6868) Skip checksum is broke; are we double-checksumming by default?
[ https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6868: - Status: Open (was: Patch Available) Skip checksum is broke; are we double-checksumming by default? -- Key: HBASE-6868 URL: https://issues.apache.org/jira/browse/HBASE-6868 Project: HBase Issue Type: Bug Components: HFile, wal Affects Versions: 0.94.1, 0.94.0 Reporter: LiuLei Priority: Blocker Fix For: 0.94.3, 0.96.0 Attachments: 6868-0.96-idea.txt, 6868-0.96-v2.txt, 6868-0.96-v3.txt The HFile contains checksums for decrease the iops, so when Hbase read HFile , that dont't need to read the checksum from meta file of HDFS. But HLog file of Hbase don't contain the checksum, so when HBase read the HLog, that must read checksum from meta file of HDFS. We could add setSkipChecksum per file to hdfs or we could write checksums into WAL if this skip checksum facility is enabled -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6868) Skip checksum is broke; are we double-checksumming by default?
[ https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6868: - Attachment: 6868-0.94.txt Skip checksum is broke; are we double-checksumming by default? -- Key: HBASE-6868 URL: https://issues.apache.org/jira/browse/HBASE-6868 Project: HBase Issue Type: Bug Components: HFile, wal Affects Versions: 0.94.0, 0.94.1 Reporter: LiuLei Priority: Blocker Fix For: 0.94.2, 0.96.0 Attachments: 6868-0.94.txt, 6868-0.96-idea.txt, 6868-0.96-v2.txt, 6868-0.96-v3.txt The HFile contains checksums for decrease the iops, so when Hbase read HFile , that dont't need to read the checksum from meta file of HDFS. But HLog file of Hbase don't contain the checksum, so when HBase read the HLog, that must read checksum from meta file of HDFS. We could add setSkipChecksum per file to hdfs or we could write checksums into WAL if this skip checksum facility is enabled -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6868) Skip checksum is broke; are we double-checksumming by default?
[ https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6868: - Fix Version/s: (was: 0.94.3) 0.94.2 Skip checksum is broke; are we double-checksumming by default? -- Key: HBASE-6868 URL: https://issues.apache.org/jira/browse/HBASE-6868 Project: HBase Issue Type: Bug Components: HFile, wal Affects Versions: 0.94.0, 0.94.1 Reporter: LiuLei Priority: Blocker Fix For: 0.94.2, 0.96.0 Attachments: 6868-0.94.txt, 6868-0.96-idea.txt, 6868-0.96-v2.txt, 6868-0.96-v3.txt The HFile contains checksums for decrease the iops, so when Hbase read HFile , that dont't need to read the checksum from meta file of HDFS. But HLog file of Hbase don't contain the checksum, so when HBase read the HLog, that must read checksum from meta file of HDFS. We could add setSkipChecksum per file to hdfs or we could write checksums into WAL if this skip checksum facility is enabled -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6868) Skip checksum is broke; are we double-checksumming by default?
[ https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6868: - Attachment: 6868-0.96-v2.txt Patch that should work with all versions of Hadoop. Instead of reflection, just spells out the relevant config option. Skip checksum is broke; are we double-checksumming by default? -- Key: HBASE-6868 URL: https://issues.apache.org/jira/browse/HBASE-6868 Project: HBase Issue Type: Bug Components: HFile, wal Affects Versions: 0.94.0, 0.94.1 Reporter: LiuLei Priority: Blocker Fix For: 0.94.3, 0.96.0 Attachments: 6868-0.96-idea.txt, 6868-0.96-v2.txt The HFile contains checksums for decrease the iops, so when Hbase read HFile , that dont't need to read the checksum from meta file of HDFS. But HLog file of Hbase don't contain the checksum, so when HBase read the HLog, that must read checksum from meta file of HDFS. We could add setSkipChecksum per file to hdfs or we could write checksums into WAL if this skip checksum facility is enabled -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6868) Skip checksum is broke; are we double-checksumming by default?
[ https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6868: - Attachment: 6868-0.96-v3.txt This is what I meant. Now we document that: * dfs.client.read.shortcircuit should true * dfs.client.read.shortcircuit.skip.checksum should be false That way this is no longer tied to short circuiting in the code. And if short circuiting happens to be enabled it is always correct to enable dfs.client.read.shortcircuit.skip.checksum if we want HBase checksum. Sounds good? Skip checksum is broke; are we double-checksumming by default? -- Key: HBASE-6868 URL: https://issues.apache.org/jira/browse/HBASE-6868 Project: HBase Issue Type: Bug Components: HFile, wal Affects Versions: 0.94.0, 0.94.1 Reporter: LiuLei Priority: Blocker Fix For: 0.94.3, 0.96.0 Attachments: 6868-0.96-idea.txt, 6868-0.96-v2.txt, 6868-0.96-v3.txt The HFile contains checksums for decrease the iops, so when Hbase read HFile , that dont't need to read the checksum from meta file of HDFS. But HLog file of Hbase don't contain the checksum, so when HBase read the HLog, that must read checksum from meta file of HDFS. We could add setSkipChecksum per file to hdfs or we could write checksums into WAL if this skip checksum facility is enabled -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6868) Skip checksum is broke; are we double-checksumming by default?
[ https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6868: - Attachment: 6868-0.96-idea.txt So how about this? Check that short circuit is enabled for HBase checksumming. Also disables short circuit checksum for the noChecksumFs. Skip checksum is broke; are we double-checksumming by default? -- Key: HBASE-6868 URL: https://issues.apache.org/jira/browse/HBASE-6868 Project: HBase Issue Type: Bug Components: HFile, wal Affects Versions: 0.94.0, 0.94.1 Reporter: LiuLei Priority: Blocker Fix For: 0.94.3, 0.96.0 The HFile contains checksums for decrease the iops, so when Hbase read HFile , that dont't need to read the checksum from meta file of HDFS. But HLog file of Hbase don't contain the checksum, so when HBase read the HLog, that must read checksum from meta file of HDFS. We could add setSkipChecksum per file to hdfs or we could write checksums into WAL if this skip checksum facility is enabled -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6868) Skip checksum is broke; are we double-checksumming by default?
[ https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6868: - Attachment: (was: 6868-0.96-idea.txt) Skip checksum is broke; are we double-checksumming by default? -- Key: HBASE-6868 URL: https://issues.apache.org/jira/browse/HBASE-6868 Project: HBase Issue Type: Bug Components: HFile, wal Affects Versions: 0.94.0, 0.94.1 Reporter: LiuLei Priority: Blocker Fix For: 0.94.3, 0.96.0 The HFile contains checksums for decrease the iops, so when Hbase read HFile , that dont't need to read the checksum from meta file of HDFS. But HLog file of Hbase don't contain the checksum, so when HBase read the HLog, that must read checksum from meta file of HDFS. We could add setSkipChecksum per file to hdfs or we could write checksums into WAL if this skip checksum facility is enabled -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6868) Skip checksum is broke; are we double-checksumming by default?
[ https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6868: - Attachment: 6868-0.96-idea.txt The right patch, this time. Skip checksum is broke; are we double-checksumming by default? -- Key: HBASE-6868 URL: https://issues.apache.org/jira/browse/HBASE-6868 Project: HBase Issue Type: Bug Components: HFile, wal Affects Versions: 0.94.0, 0.94.1 Reporter: LiuLei Priority: Blocker Fix For: 0.94.3, 0.96.0 Attachments: 6868-0.96-idea.txt The HFile contains checksums for decrease the iops, so when Hbase read HFile , that dont't need to read the checksum from meta file of HDFS. But HLog file of Hbase don't contain the checksum, so when HBase read the HLog, that must read checksum from meta file of HDFS. We could add setSkipChecksum per file to hdfs or we could write checksums into WAL if this skip checksum facility is enabled -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6868) Skip checksum is broke; are we double-checksumming by default?
[ https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6868: - Summary: Skip checksum is broke; are we double-checksumming by default? (was: Skip checksum is broke; are e double-checksumming by default?) Skip checksum is broke; are we double-checksumming by default? -- Key: HBASE-6868 URL: https://issues.apache.org/jira/browse/HBASE-6868 Project: HBase Issue Type: Bug Components: HFile, wal Affects Versions: 0.94.0, 0.94.1 Reporter: LiuLei Priority: Blocker Fix For: 0.94.3, 0.96.0 The HFile contains checksums for decrease the iops, so when Hbase read HFile , that dont't need to read the checksum from meta file of HDFS. But HLog file of Hbase don't contain the checksum, so when HBase read the HLog, that must read checksum from meta file of HDFS. We could add setSkipChecksum per file to hdfs or we could write checksums into WAL if this skip checksum facility is enabled -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
