Hello!
Did you add `ignite-compres` module to your classpath?
Have you tried WAL compression instead? Please check
https://apacheignite.readme.io/docs/write-ahead-log#section-wal-records-compression
<https://apacheignite.readme.io/docs/write-ahead-log#section-wal-records-compression>
Regards,
--
Ilya Kasnacheev
пт, 28 авг. 2020 г. в 06:52, 38797715 <[email protected]
<mailto:[email protected]>>:
Hi,
create table statement are as follows:
CREATETABLEPI_COM_DAY
(COM_ID VARCHAR(30) NOTNULL,
ITEM_ID VARCHAR(30) NOTNULL,
DATE1 VARCHAR(8) NOTNULL,
KIND VARCHAR(1),
QTY_IOD DECIMAL(18, 6) ,
AMT_IOD DECIMAL(18, 6) ,
QTY_PURCH DECIMAL(18, 6) ,
AMT_PURCH DECIMAL(18,6) ,
QTY_SOLD DECIMAL(18,6) ,
AMT_SOLD DECIMAL(18, 6) ,
AMT_SOLD_NO_TAX DECIMAL(18, 6) ,
QTY_PROFIT DECIMAL(18, 6) ,
AMT_PROFIT DECIMAL(18, 6) ,
QTY_LOSS DECIMAL(18,6) ,
AMT_LOSS DECIMAL(18, 6) ,
QTY_EOD DECIMAL(18, 6) ,
AMT_EOD DECIMAL(18,6) ,
UNIT_COST DECIMAL(18,8) ,
SUMCOST_SOLD DECIMAL(18,6) ,
GROSS_PROFIT DECIMAL(18, 6) ,
QTY_ALLOCATION DECIMAL(18,6) ,
AMT_ALLOCATION DECIMAL(18,2) ,
AMT_ALLOCATION_NO_TAX DECIMAL(18, 2) ,
GROSS_PROFIT_ALLOCATION DECIMAL(18,6) ,
SUMCOST_SOLD_ALLOCATION DECIMAL(18,6) ,
PRIMARYKEY(COM_ID,ITEM_ID,DATE1))
WITH"template=cache-partitioned,CACHE_NAME=PI_COM_DAY";
CREATEINDEX IDX_PI_COM_DAY_ITEM_DATE ONPI_COM_DAY(ITEM_ID,DATE1);
I don't think there's anything special about it.
Then we imported 10 million data using the COPY command.Data is
basically the actual production data, I think the dispersion is
OK, not artificial data with high similarity.
I would like to know if there are test results for the function of
disk compression? Most of the other memory databases also have the
function of data compression, but it doesn't look like it is now,
or what's wrong with me?
在 2020/8/28 上午12:39, Michael Cherkasov 写道:
Could you please share your benchmark code? I believe compression
might depend on data you write, if it full random, it's difficult
to compress the data.
On Wed, Aug 26, 2020, 8:26 PM 38797715 <[email protected]
<mailto:[email protected]>> wrote:
Hi,
We turn on disk compression to see the trend of execution
time and disk space.
Our expectation is that after disk compression is turned on,
although more CPU is used, the disk space is less occupied.
Because more data is written per unit time, the overall
execution time will be shortened in the case of insufficient
memory.
However, it is found that the execution time and disk
consumption do not change significantly. We tested the
diskPageCompressionLevel values as 0, 10 and 17 respectively.
Our test method is as follows:
The ignite-compress module has been introduced.
The configuration of ignite is as follows:
<?xmlversion="1.0"encoding="UTF-8"?>
<beansxmlns="http://www.springframework.org/schema/beans";
<http://www.springframework.org/schema/beans>
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
<http://www.w3.org/2001/XMLSchema-instance>xsi:schemaLocation="http://www.springframework.org/schema/beans
<http://www.springframework.org/schema/beans>
http://www.springframework.org/schema/beans/spring-beans.xsd
<http://www.springframework.org/schema/beans/spring-beans.xsd>">
<beanid="ignite.cfg"class="org.apache.ignite.configuration.IgniteConfiguration">
<propertyname="peerClassLoadingEnabled"value="true"/>
<propertyname="consistentId"value="b"/>
<propertyname="igniteInstanceName"value="ClusterName1"/>
<propertyname="workDirectory"value="/home/ignite"/>
<propertyname="gridLogger">
<beanclass="org.apache.ignite.logger.log4j2.Log4J2Logger">
<constructor-argtype="java.lang.String"value="config/ignite-log4j2.xml"/>
</bean>
</property>
<propertyname="cacheConfiguration">
<list>
<beanid="partitioned-cache-template"abstract="true"class="org.apache.ignite.configuration.CacheConfiguration">
<propertyname="name"value="cache-partitioned*"/>
<propertyname="cacheMode"value="PARTITIONED"/>
<propertyname="queryParallelism"value="2"/>
<propertyname="diskPageCompression"value="LZ4"/>
<propertyname="diskPageCompressionLevel"value="17"/>
</bean>
</list>
</property>
<!-- Enabling Apache Ignite Persistent Store. -->
<propertyname="dataStorageConfiguration">
<beanclass="org.apache.ignite.configuration.DataStorageConfiguration">
<propertyname="pageSize"value="#{4096 * 2}"/>
<propertyname="defaultDataRegionConfiguration">
<beanclass="org.apache.ignite.configuration.DataRegionConfiguration">
<propertyname="persistenceEnabled"value="true"/>
<propertyname="maxSize"value="#{1L * 1024 * 1024 * 1024}"/>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
