Hi Rahul, Possibly, mostly a new data is loaded to Ignite. I meant, Ignite allocate new pages, rather than update ones.
In that case, you may not get benefit from increasing checkpoint region size. It will just deffer a checkpoint. Also, you can try to move WAL and ignite store to different disks and to set region initial size to reduce or avoid region extents allocation . On Mon, Apr 2, 2018 at 9:59 AM, rahul aneja <[email protected]> wrote: > Hi Andrey, > > Yes we are using SSD. Earlier we were using default checkpoint buffer 256 > MB , in order to reduce the frequency, we increased the buffer size , but > it didn’t have any impact on performance > > On Fri, 30 Mar 2018 at 10:49 PM, Andrey Mashenkov < > [email protected]> wrote: > >> Hi, >> >> Possibly, storage is a bottleneck or checkpoint buffer is too large. >> Do you use Provissioned IOPS SSD? >> >> >> On Fri, Mar 30, 2018 at 3:32 PM, rahul aneja <[email protected]> >> wrote: >> >>> Hi , >>> >>> We are trying to load orc data (around 50 GB) on s3 from spark using >>> dataframe API. It starts fast with good write throughput and then after >>> sometime throughput just drops and it gets stuck. >>> >>> We also tried changing multiple configurations , but no luck >>> 1. enabling checkpoint write throttling >>> 2. disabling throttling and increasing checkpoint buffer >>> >>> >>> Please find below configuration and properties of the cluster >>> >>> >>> 1. 10 node cluster r4.4xl (EMR aws) and shared with spark >>> 2. ignite is started with -Xms20g -Xmx30g >>> 3. Cache mode is partitioned >>> >>> 4. persistence is enabled >>> 5. DirectIO is enabled >>> 6. No backup >>> >>> <property name=“dataStorageConfiguration”> >>> <bean class=“org.apache.ignite.configuration. >>> DataStorageConfiguration”> >>> <!-- Enable write throttling. --> >>> <property name=“writeThrottlingEnabled” value=“false”/> >>> <property name=“defaultDataRegionConfiguration”> >>> <bean class=“org.apache.ignite.configuration. >>> DataRegionConfiguration”> >>> <property name=“persistenceEnabled” value=“true”/> >>> <property name=“checkpointPageBufferSize” >>> value=“#{20L * 1024 * 1024 * 1024}“/> >>> <property name=“name” value=“Default_Region”/> >>> <property name=“maxSize” value=“#{60L * 1024 * >>> 1024 * 1024}“/> >>> </bean> >>> </property> >>> <property name=“walMode” value=“NONE”/> >>> </bean> >>> </property> >>> >>> >>> Thanks in advance, >>> >>> Rahul Aneja >>> >>> >>> >> >> >> -- >> Best regards, >> Andrey V. Mashenkov >> > -- Best regards, Andrey V. Mashenkov
