[jira] [Updated] (KUDU-2071) disk size is much large than actually data size
[ https://issues.apache.org/jira/browse/KUDU-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Daniel Cryans updated KUDU-2071: - Resolution: Duplicate Fix Version/s: n/a Status: Resolved (was: In Review) I moved this jira from being "In Review" which is a status that we only use when patches are actually up for review, to "Duplicate" since this is really KUDU-1943. > disk size is much large than actually data size > --- > > Key: KUDU-2071 > URL: https://issues.apache.org/jira/browse/KUDU-2071 > Project: Kudu > Issue Type: Improvement > Components: tserver >Affects Versions: 1.3.0 > Environment: system version > 4.9.20-11.31.amzn1.x86_64 #1 SMP Thu Apr 13 01:53:57 UTC 2017 x86_64 x86_64 > x86_64 GNU/Linux > software version: > kudu 1.3.0-cdh5.11.0 > revision 4dcf4a9d516865d249f4cb9b07f93c67e84614ae > build type RELEASE > built by jenkins at 12 Apr 2017 14:02:51 PST on > kudu-centos66-046c.vpc.cloudera.com > build id 2017-04-12_13-25-42 >Reporter: KingLee > Labels: patch > Fix For: n/a > > Original Estimate: 168h > Remaining Estimate: 168h > > I ran m -rf on all the data dirs before reinstalling the cluster, and insert > 100 records to the cluster using yscb, data's size is about 5GB,but it > cost disk size 260GB, one of node 's disk as follows: > before write data: > [root@ip-10-1-42-124 ~]# du -sh /data1/server/kudu/tserver_wal/wals/ > /data2/server/kudu/tserver_data/ /data3/server/kudu/tserver_data/data/ > /data4/server/kudu/tserver_data/data/ > 4.0K/data1/server/kudu/tserver_wal/wals/ > 24K /data2/server/kudu/tserver_data/ > 8.0K/data3/server/kudu/tserver_data/data/ > 8.0K/data4/server/kudu/tserver_data/data/ > after write data: > [root@ip-10-1-42-124 ~]# du -sh /data1/server/kudu/tserver_wal/wals/ > /data2/server/kudu/tserver_data/ /data3/server/kudu/tserver_data/data/ > /data4/server/kudu/tserver_data/data/ > 2.7G/data1/server/kudu/tserver_wal/wals/ > 29G /data2/server/kudu/tserver_data/ > 29G /data3/server/kudu/tserver_data/data/ > 27G /data4/server/kudu/tserver_data/data/ > actually data size : > 9b137115cfaa427a9106c87086f41957 5041*3 MBytes > kudu tserver configure: > --fs_wal_dir=/var/lib/kudu/tserver > --fs_data_dirs=/var/lib/kudu/tserver > --default_num_replicas=3 > --tserver_master_addrs=192.168.1.22:7051,1192.168.1.23:7051,192.168.1.24:7051,192.168.1.25:7051,192.168.1.26:7051 > --maintenance_manager_num_threads=4 > --block_cache_capacity_mb=10240 > --memory_limit_hard_bytes=600 > --fs_wal_dir=/data1/server/kudu/tserver_wal > --fs_data_dirs=/data2/server/kudu/tserver_data,/data3/server/kudu/tserver_data,/data4/server/kudu/tserver_data > --fs_data_dirs_reserved_bytes=100 > --log_segment_size_mb=8 > and our production environment 's data is 25TB, but cost 45TB, where do these > disks go? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KUDU-2071) disk size is much large than actually data size
[ https://issues.apache.org/jira/browse/KUDU-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KingLee updated KUDU-2071: -- Status: In Review (was: Open) > disk size is much large than actually data size > --- > > Key: KUDU-2071 > URL: https://issues.apache.org/jira/browse/KUDU-2071 > Project: Kudu > Issue Type: Improvement > Components: tserver >Affects Versions: 1.3.0 > Environment: system version > 4.9.20-11.31.amzn1.x86_64 #1 SMP Thu Apr 13 01:53:57 UTC 2017 x86_64 x86_64 > x86_64 GNU/Linux > software version: > kudu 1.3.0-cdh5.11.0 > revision 4dcf4a9d516865d249f4cb9b07f93c67e84614ae > build type RELEASE > built by jenkins at 12 Apr 2017 14:02:51 PST on > kudu-centos66-046c.vpc.cloudera.com > build id 2017-04-12_13-25-42 >Reporter: KingLee > Labels: patch > Original Estimate: 168h > Remaining Estimate: 168h > > I ran m -rf on all the data dirs before reinstalling the cluster, and insert > 100 records to the cluster using yscb, data's size is about 5GB,but it > cost disk size 260GB, one of node 's disk as follows: > before write data: > [root@ip-10-1-42-124 ~]# du -sh /data1/server/kudu/tserver_wal/wals/ > /data2/server/kudu/tserver_data/ /data3/server/kudu/tserver_data/data/ > /data4/server/kudu/tserver_data/data/ > 4.0K/data1/server/kudu/tserver_wal/wals/ > 24K /data2/server/kudu/tserver_data/ > 8.0K/data3/server/kudu/tserver_data/data/ > 8.0K/data4/server/kudu/tserver_data/data/ > after write data: > [root@ip-10-1-42-124 ~]# du -sh /data1/server/kudu/tserver_wal/wals/ > /data2/server/kudu/tserver_data/ /data3/server/kudu/tserver_data/data/ > /data4/server/kudu/tserver_data/data/ > 2.7G/data1/server/kudu/tserver_wal/wals/ > 29G /data2/server/kudu/tserver_data/ > 29G /data3/server/kudu/tserver_data/data/ > 27G /data4/server/kudu/tserver_data/data/ > actually data size : > 9b137115cfaa427a9106c87086f41957 5041*3 MBytes > kudu tserver configure: > --fs_wal_dir=/var/lib/kudu/tserver > --fs_data_dirs=/var/lib/kudu/tserver > --default_num_replicas=3 > --tserver_master_addrs=192.168.1.22:7051,1192.168.1.23:7051,192.168.1.24:7051,192.168.1.25:7051,192.168.1.26:7051 > --maintenance_manager_num_threads=4 > --block_cache_capacity_mb=10240 > --memory_limit_hard_bytes=600 > --fs_wal_dir=/data1/server/kudu/tserver_wal > --fs_data_dirs=/data2/server/kudu/tserver_data,/data3/server/kudu/tserver_data,/data4/server/kudu/tserver_data > --fs_data_dirs_reserved_bytes=100 > --log_segment_size_mb=8 > and our production environment 's data is 25TB, but cost 45TB, where do these > disks go? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KUDU-2071) disk size is much large than actually data size
[ https://issues.apache.org/jira/browse/KUDU-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KingLee updated KUDU-2071: -- Description: I ran m -rf on all the data dirs before reinstalling the cluster, and insert 100 records to the cluster using yscb, data's size is about 5GB,but it cost disk size 260GB, one of node 's disk as follows: before write data: [root@ip-10-1-42-124 ~]# du -sh /data1/server/kudu/tserver_wal/wals/ /data2/server/kudu/tserver_data/ /data3/server/kudu/tserver_data/data/ /data4/server/kudu/tserver_data/data/ 4.0K/data1/server/kudu/tserver_wal/wals/ 24K /data2/server/kudu/tserver_data/ 8.0K/data3/server/kudu/tserver_data/data/ 8.0K/data4/server/kudu/tserver_data/data/ after write data: [root@ip-10-1-42-124 ~]# du -sh /data1/server/kudu/tserver_wal/wals/ /data2/server/kudu/tserver_data/ /data3/server/kudu/tserver_data/data/ /data4/server/kudu/tserver_data/data/ 2.7G/data1/server/kudu/tserver_wal/wals/ 29G /data2/server/kudu/tserver_data/ 29G /data3/server/kudu/tserver_data/data/ 27G /data4/server/kudu/tserver_data/data/ actually data size : 9b137115cfaa427a9106c87086f41957 5041*3 MBytes kudu tserver configure: --fs_wal_dir=/var/lib/kudu/tserver --fs_data_dirs=/var/lib/kudu/tserver --default_num_replicas=3 --tserver_master_addrs=192.168.1.22:7051,1192.168.1.23:7051,192.168.1.24:7051,192.168.1.25:7051,192.168.1.26:7051 --maintenance_manager_num_threads=4 --block_cache_capacity_mb=10240 --memory_limit_hard_bytes=600 --fs_wal_dir=/data1/server/kudu/tserver_wal --fs_data_dirs=/data2/server/kudu/tserver_data,/data3/server/kudu/tserver_data,/data4/server/kudu/tserver_data --fs_data_dirs_reserved_bytes=100 --log_segment_size_mb=8 and our production environment 's data is 25TB, but cost 45TB, where do these disks go? was: I ran m -rf on all the data dirs before reinstalling the cluster, and insert 100 records to the cluster using yscb, data's size is about 5GB,but it cost disk size 260GB, one of node 's disk as follows: before write data: [root@ip-10-1-42-124 ~]# du -sh /data1/server/kudu/tserver_wal/wals/ /data2/server/kudu/tserver_data/ /data3/server/kudu/tserver_data/data/ /data4/server/kudu/tserver_data/data/ 4.0K/data1/server/kudu/tserver_wal/wals/ 24K /data2/server/kudu/tserver_data/ 8.0K/data3/server/kudu/tserver_data/data/ 8.0K/data4/server/kudu/tserver_data/data/ after write data: [root@ip-10-1-42-124 ~]# du -sh /data1/server/kudu/tserver_wal/wals/ /data2/server/kudu/tserver_data/ /data3/server/kudu/tserver_data/data/ /data4/server/kudu/tserver_data/data/ 2.7G/data1/server/kudu/tserver_wal/wals/ 29G /data2/server/kudu/tserver_data/ 29G /data3/server/kudu/tserver_data/data/ 27G /data4/server/kudu/tserver_data/data/ actually data size : 9b137115cfaa427a9106c87086f41957 5041MBytes kudu tserver configure: --fs_wal_dir=/var/lib/kudu/tserver --fs_data_dirs=/var/lib/kudu/tserver --default_num_replicas=3 --tserver_master_addrs=192.168.1.22:7051,1192.168.1.23:7051,192.168.1.24:7051,192.168.1.25:7051,192.168.1.26:7051 --maintenance_manager_num_threads=4 --block_cache_capacity_mb=10240 --memory_limit_hard_bytes=600 --fs_wal_dir=/data1/server/kudu/tserver_wal --fs_data_dirs=/data2/server/kudu/tserver_data,/data3/server/kudu/tserver_data,/data4/server/kudu/tserver_data --fs_data_dirs_reserved_bytes=100 --log_segment_size_mb=8 and our production environment 's data is 25TB, but cost 45TB, where do these disks go? > disk size is much large than actually data size > --- > > Key: KUDU-2071 > URL: https://issues.apache.org/jira/browse/KUDU-2071 > Project: Kudu > Issue Type: Improvement > Components: tserver >Affects Versions: 1.3.0 > Environment: system version > 4.9.20-11.31.amzn1.x86_64 #1 SMP Thu Apr 13 01:53:57 UTC 2017 x86_64 x86_64 > x86_64 GNU/Linux > software version: > kudu 1.3.0-cdh5.11.0 > revision 4dcf4a9d516865d249f4cb9b07f93c67e84614ae > build type RELEASE > built by jenkins at 12 Apr 2017 14:02:51 PST on > kudu-centos66-046c.vpc.cloudera.com > build id 2017-04-12_13-25-42 >Reporter: KingLee > Labels: patch > Original Estimate: 168h > Remaining Estimate: 168h > > I ran m -rf on all the data dirs before reinstalling the cluster, and insert > 100 records to the cluster using yscb, data's size is about 5GB,but it > cost disk size 260GB, one of node 's disk as follows: > before write data: > [root@ip-10-1-42-124 ~]# du -sh /data1/server/kudu/tserver_wal/wals/ > /data2/server/kudu/tserver_data/ /data3/server/kudu/tserver_data/data/ > /data4/server/kudu/tserver_data/data/ > 4.0K/data1/server/kudu/tserver_wal/wals/ > 24K /data2/server/kudu/tserver_data/ > 8.0K/data3/server/kudu/tserver_data/data/ > 8.0K
