Found the issue, and thank goodness it was a configuration issue on my end:
I was setting the yarn.scheduler.maximum-allocation-vcores too low and
artificially constraining the cluster. This is (as the name implies) the
maximum allocation for the entire cluster; I had interpreted the
description
Interestingly, I just tried setting
yarn.scheduler.minimum-allocation-vcores=2 and restarting everything. On
startup, the RM now displays a Minimum Allocation of , but my application container still shows "Resource:4096 Memory,
1 VCores". The statistics page for the "default" queue shows "Used
Hi Prateek,
I'm not getting an error now, just an unyielding vcore allotment of 1.
I just verified that we're setting
yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
and
rmatharu commented on issue #981: SAMZA-2152 : Bugfix: Making KafkaSytemAdmin's
metadataConsumer accesses thread-safe, enabling StreamRegexMonitors only when
required
URL: https://github.com/apache/samza/pull/981#issuecomment-479104652
addressed your comments.
adminClient accesses seem
And just to double check, you also changed the
yarn.resourcemanager.scheduler.class to CapacityScheduler?
On Tue, Apr 2, 2019 at 9:49 AM Prateek Maheshwari
wrote:
> Is it still the same message from the AM? The one that says: "Got AM
> register response. The YARN RM supports container requests
Is it still the same message from the AM? The one that says: "Got AM
register response. The YARN RM supports container requests with max-mem:
14336, max-cpu: 1"
On Tue, Apr 2, 2019 at 12:09 AM Malcolm McFarland
wrote:
> Hey Prateek,
>
> The upgrade to Hadoop 2.7.6 went fine; everything seems to
prateekm commented on a change in pull request #981: SAMZA-2152 : Bugfix:
Making KafkaSytemAdmin's metadataConsumer accesses thread-safe, enabling
StreamRegexMonitors only when required
URL: https://github.com/apache/samza/pull/981#discussion_r271393999
##
File path:
prateekm commented on a change in pull request #981: SAMZA-2152 : Bugfix:
Making KafkaSytemAdmin's metadataConsumer accesses thread-safe, enabling
StreamRegexMonitors only when required
URL: https://github.com/apache/samza/pull/981#discussion_r271392441
##
File path:
mynameborat commented on issue #905: SAMZA-2055: Async high level api
URL: https://github.com/apache/samza/pull/905#issuecomment-479058100
`testSinkEndToEndWithKeyWithNullRecords` seems to be flaky. I have a 60%
pass rate locally.
Hey Prateek,
The upgrade to Hadoop 2.7.6 went fine; everything seems to be working, and
access to S3 via an access key/secret pair is working as well. However, my
requested tasks are still only getting allocated 1 core, despite requesting
more than that. Once again, I have a 3-node cluster that
10 matches
Mail list logo