yarn-site.xml setup

2020-12-28 Thread Malcolm McFarland 
Hey all,

Sorry for the daft question, but when loading Samza 1.5 into a remote YARN
cluster, where/how is the yarn-site.xml file supposed to be located? I've
set the following environment variables and run-app.sh isn't finding it (or
at least isn't reading my resource manager address) correctly:

HADOOP_YARN_HOME
YARN_HOME
HADOOP_COMMON_HOME
HADOOP_HOME
HADOOP_PREFIX
HADOOP_CONF_DIR

My yarn-site.xml is at $YARN_HOME/etc/hadoop/yarn-site.xml, and contains
the following configuration:


  yarn.resourcemanager.address
  ${RM_IP_ADDRESS}


This worked fine in 0.14.1.

Cheers,
Malcolm McFarland
Cavulus


This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
unauthorized or improper disclosure, copying, distribution, or use of the
contents of this message is prohibited. The information contained in this
message is intended only for the personal and confidential use of the
recipient(s) named above. If you have received this message in error,
please notify the sender immediately and delete the original message.

Re: SAMZA-2612: Kafka topic naming not supported fully

2020-12-28 Thread Yi Pan 
Hey, Stuart,

Sounds great that you have found the way around it! Thanks!

-Yi

On Sat, Dec 19, 2020 at 12:28 PM Stuart Perks 
wrote:

> This can be done using withPhysicalName
>
> Closed the JIRA
>
> On 2020/12/17 12:19:27, Stuart Perks  wrote:
> > https://issues.apache.org/jira/browse/SAMZA-2612>
> >
> > Raised a bug JIRA but wanted to check with the community. Any thoughts
> would be great.>
> >
> >
> > The StreamDescriptor class cannot accept all acceptable formats for
> Kafka Topic names.>
> > StreamDescriptor>
> >   private static final Pattern STREAM_ID_PATTERN =
> Pattern.compile("[\\d\\w-_]+");>
> > Kafka Topic Validation>
> > public static final String LEGAL_CHARS = "[a-zA-Z0-9._-]";>
> > Taking the example this is valid>
> >  KafkaInputDescriptor pageViewStreamDescriptor =
> kafkaSystemDescriptor.getInputDescriptor("page-view-topic", new
> JsonSerdeV2<>(PageView.class));>
> > but this is not if we use the name page.view.topic as . Is not valid in
> the StreamDescriptor.>
> >  KafkaInputDescriptor pageViewStreamDescriptor =
> kafkaSystemDescriptor.getInputDescriptor("page.view.topic", new
> JsonSerdeV2<>(PageView.class));>
> > Stream Descriptor Validation <
> https://github.com/apache/samza/blob/master/samza-api/src/main/java/org/apache/samza/system/descriptors/StreamDescriptor.java#L48>>
>
> > Kafka Topic Validation <
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/internals/Topic.java#L29>>
>