Hi,
Can someone please help me with configuration property on how to increase
memory setting for Unifier?
I am using the below way, to increase the memory for Application Master and
opertors of a DAG:
<property>
<name>dt.attr.MASTER_MEMORY_MB</name>
<value>10240</value>
</property>
<property>
<name>dt.operator.*.attr.MEMORY_MB</name>
<value>10240</value>
</property>
UseCase/Scenario:
I am having an apex application that reads from Kafka in 3 partitions with the
setting “ONE_TO_MANY”, and sends the messages to following operators. But, the
problem is, even though the messages are being read thru multiple parititons,
but I see a unifier operation happening after the messages read in multiple
kafka partitions. Which is becoming a bottle nect with the number of increase
in the number of messages being floated.
I have configured the memory all the operators to 10GB, with above given
setting, but, the unifier which has been cropped up, was taking only 2GB by
default, and failing with heap space problems.
Hence I wish to increase the memory for the unifier.
Thanks a lot in advance.
Regards,
Raja.
