kaushik srinivas created FLUME-3213:
---------------------------------------
Summary: TransactionCapacity for spillableMemoryChannel with
overflowCapacity set to '0' is limited to 1000000.
Key: FLUME-3213
URL: https://issues.apache.org/jira/browse/FLUME-3213
Project: Flume
Issue Type: Bug
Affects Versions: 1.7.0
Environment: Below is the setup info,
Channel :SPILLABLEMEMORY
byteCapacityBufferPercentage = 10
transactionCapacity = 2000000
memoryCapacity = 17000000
overflowCapacity = 0
Kafka source and kite data set sink is being used.
Reporter: kaushik srinivas
Intent is to make this spillabeMemory channel work as pure memory channel.
So the parameter,
overflowCapacity is set with value 0.
But then observing exceptions and by default capacity is being set to 1000000.
Error 1:
2018-01-17 05:57:26,783 (lifecycleSupervisor-1-0-EventThread) [WARN -
org.apache.flume.channel.file.FileChannel.configure(FileChannel.java:167)]
Invalid capacity specified, initializing channel to default capacity of 1000000
2018-01-17 05:57:26,787 (lifecycleSupervisor-1-0-EventThread) [ERROR -
org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:205)]
Channel channel_store_sales has been removed due to an error during
configuration
Impact of this is on the transaction capacity.
Error1 sets capacity to 10,00,000 and thus transactionCapacity is not allowed
to be set more than the capacity value of 1000000 limit.
Error2:
java.lang.IllegalStateException: File Channel transaction capacity cannot be
greater than the capacity of the channel.
at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at org.apache.flume.channel.file.FileChannel.configure(FileChannel.java:186)
at
org.apache.flume.channel.SpillableMemoryChannel.configure(SpillableMemoryChannel.java:745)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at
org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:199)
at
org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:100)
at
org.apache.flume.node.PollingZooKeeperConfigurationProvider.refreshConfiguration(PollingZooKeeperConfigurationProvider.java:104)
at
org.apache.flume.node.PollingZooKeeperConfigurationProvider.access$000(PollingZooKeeperConfigurationProvider.java:36)
at
org.apache.flume.node.PollingZooKeeperConfigurationProvider$1.nodeChanged(PollingZooKeeperConfigurationProvider.java:78)
at
org.apache.curator.framework.recipes.cache.NodeCache$4.apply(NodeCache.java:293)
at
org.apache.curator.framework.recipes.cache.NodeCache$4.apply(NodeCache.java:287)
at
org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
at
org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83)
at
org.apache.curator.framework.recipes.cache.NodeCache.setNewData(NodeCache.java:284)
at
org.apache.curator.framework.recipes.cache.NodeCache.processBackgroundResult(NodeCache.java:252)
at
org.apache.curator.framework.recipes.cache.NodeCache.access$300(NodeCache.java:53)
at
org.apache.curator.framework.recipes.cache.NodeCache$3.processResult(NodeCache.java:111)
at
org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:715)
at
org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:502)
at
org.apache.curator.framework.imps.GetDataBuilderImpl$3.processResult(GetDataBuilderImpl.java:254)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:561)
So due to this, transactionCapacity for spillableMemoryChannel with
overflowCapacity set to 0 is limited to 1000000.
So overflowCapacity has to be set with a high number in order to increase the
transactionCapacity beyond1000000, which should not be the ideal scenario.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
