Hello!
Did you add `ignite-compres` module to your classpath?
Have you tried WAL compression instead? Please check
https://apacheignite.readme.io/docs/write-ahead-log#section-wal-records-compression
<https://apacheignite.readme.io/docs/write-ahead-log#section-wal-records-compression>
Regards,
--
Ilya Kasnacheev
пт, 28 авг. 2020 г. в 06:52, 38797715 <[email protected]
<mailto:[email protected]>>:
Hi,
create table statement are as follows:
CREATETABLEPI_COM_DAY
(COM_ID VARCHAR(30) NOTNULL,
ITEM_ID VARCHAR(30) NOTNULL,
DATE1 VARCHAR(8) NOTNULL,
KIND VARCHAR(1),
QTY_IOD DECIMAL(18, 6) ,
AMT_IOD DECIMAL(18, 6) ,
QTY_PURCH DECIMAL(18, 6) ,
AMT_PURCH DECIMAL(18,6) ,
QTY_SOLD DECIMAL(18,6) ,
AMT_SOLD DECIMAL(18, 6) ,
AMT_SOLD_NO_TAX DECIMAL(18, 6) ,
QTY_PROFIT DECIMAL(18, 6) ,
AMT_PROFIT DECIMAL(18, 6) ,
QTY_LOSS DECIMAL(18,6) ,
AMT_LOSS DECIMAL(18, 6) ,
QTY_EOD DECIMAL(18, 6) ,
AMT_EOD DECIMAL(18,6) ,
UNIT_COST DECIMAL(18,8) ,
SUMCOST_SOLD DECIMAL(18,6) ,
GROSS_PROFIT DECIMAL(18, 6) ,
QTY_ALLOCATION DECIMAL(18,6) ,
AMT_ALLOCATION DECIMAL(18,2) ,
AMT_ALLOCATION_NO_TAX DECIMAL(18, 2) ,
GROSS_PROFIT_ALLOCATION DECIMAL(18,6) ,
SUMCOST_SOLD_ALLOCATION DECIMAL(18,6) ,
PRIMARYKEY(COM_ID,ITEM_ID,DATE1))
WITH"template=cache-partitioned,CACHE_NAME=PI_COM_DAY";
CREATEINDEX IDX_PI_COM_DAY_ITEM_DATE ONPI_COM_DAY(ITEM_ID,DATE1);
I don't think there's anything special about it.
Then we imported 10 million data using the COPY command.Data
is basically the actual production data, I think the
dispersion is OK, not artificial data with high similarity.
I would like to know if there are test results for the
function of disk compression? Most of the other memory
databases also have the function of data compression, but it
doesn't look like it is now, or what's wrong with me?
在 2020/8/28 上午12:39, Michael Cherkasov 写道:
Could you please share your benchmark code? I believe
compression might depend on data you write, if it full
random, it's difficult to compress the data.
On Wed, Aug 26, 2020, 8:26 PM 38797715 <[email protected]
<mailto:[email protected]>> wrote:
Hi,
We turn on disk compression to see the trend of
execution time and disk space.
Our expectation is that after disk compression is turned
on, although more CPU is used, the disk space is less
occupied. Because more data is written per unit time,
the overall execution time will be shortened in the case
of insufficient memory.
However, it is found that the execution time and disk
consumption do not change significantly. We tested the
diskPageCompressionLevel values as 0, 10 and 17
respectively.
Our test method is as follows:
The ignite-compress module has been introduced.
The configuration of ignite is as follows:
<?xmlversion="1.0"encoding="UTF-8"?>
<beansxmlns="http://www.springframework.org/schema/beans";
<http://www.springframework.org/schema/beans>
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
<http://www.w3.org/2001/XMLSchema-instance>xsi:schemaLocation="http://www.springframework.org/schema/beans
<http://www.springframework.org/schema/beans>
http://www.springframework.org/schema/beans/spring-beans.xsd
<http://www.springframework.org/schema/beans/spring-beans.xsd>">
<beanid="ignite.cfg"class="org.apache.ignite.configuration.IgniteConfiguration">
<propertyname="peerClassLoadingEnabled"value="true"/>
<propertyname="consistentId"value="b"/>
<propertyname="igniteInstanceName"value="ClusterName1"/>
<propertyname="workDirectory"value="/home/ignite"/>
<propertyname="gridLogger">
<beanclass="org.apache.ignite.logger.log4j2.Log4J2Logger">
<constructor-argtype="java.lang.String"value="config/ignite-log4j2.xml"/>
</bean>
</property>
<propertyname="cacheConfiguration">
<list>
<beanid="partitioned-cache-template"abstract="true"class="org.apache.ignite.configuration.CacheConfiguration">
<propertyname="name"value="cache-partitioned*"/>
<propertyname="cacheMode"value="PARTITIONED"/>
<propertyname="queryParallelism"value="2"/>
<propertyname="diskPageCompression"value="LZ4"/>
<propertyname="diskPageCompressionLevel"value="17"/>
</bean>
</list>
</property>
<!-- Enabling Apache Ignite Persistent Store. -->
<propertyname="dataStorageConfiguration">
<beanclass="org.apache.ignite.configuration.DataStorageConfiguration">
<propertyname="pageSize"value="#{4096 * 2}"/>
<propertyname="defaultDataRegionConfiguration">
<beanclass="org.apache.ignite.configuration.DataRegionConfiguration">
<propertyname="persistenceEnabled"value="true"/>
<propertyname="maxSize"value="#{1L * 1024 * 1024 * 1024}"/>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
