[
https://issues.apache.org/jira/browse/BEAM-5867?focusedWorklogId=173795&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-173795
]
ASF GitHub Bot logged work on BEAM-5867:
----------------------------------------
Author: ASF GitHub Bot
Created on: 10/Dec/18 22:25
Start Date: 10/Dec/18 22:25
Worklog Time Spent: 10m
Work Description: dmvk closed pull request #6846: [BEAM-5867] Operator
translation based on operator name
URL: https://github.com/apache/beam/pull/6846
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/base/Operator.java
b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/base/Operator.java
index 601a0f728982..4f30359e2252 100644
---
a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/base/Operator.java
+++
b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/base/Operator.java
@@ -44,7 +44,7 @@ protected Operator(@Nullable String name, @Nullable
TypeDescriptor<OutputT> outp
*
* @return maybe name
*/
- public final Optional<String> getName() {
+ public Optional<String> getName() {
return Optional.ofNullable(name);
}
@@ -52,4 +52,9 @@ protected Operator(@Nullable String name, @Nullable
TypeDescriptor<OutputT> outp
public Optional<TypeDescriptor<OutputT>> getOutputType() {
return Optional.ofNullable(outputType);
}
+
+ @Override
+ public String toString() {
+ return this.getClass().getSimpleName() + " operator{" + "name='" + name +
'\'' + '}';
+ }
}
diff --git
a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/EuphoriaOptions.java
b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/EuphoriaOptions.java
index f06212501830..5342cf2fc439 100644
---
a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/EuphoriaOptions.java
+++
b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/EuphoriaOptions.java
@@ -19,10 +19,6 @@
import com.fasterxml.jackson.annotation.JsonIgnore;
import
org.apache.beam.sdk.extensions.euphoria.core.client.accumulators.AccumulatorProvider;
-import org.apache.beam.sdk.extensions.euphoria.core.client.operator.FlatMap;
-import org.apache.beam.sdk.extensions.euphoria.core.client.operator.Join;
-import
org.apache.beam.sdk.extensions.euphoria.core.client.operator.ReduceByKey;
-import org.apache.beam.sdk.extensions.euphoria.core.client.operator.Union;
import org.apache.beam.sdk.options.Default;
import org.apache.beam.sdk.options.DefaultValueFactory;
import org.apache.beam.sdk.options.Description;
@@ -36,12 +32,7 @@
@Override
public TranslatorProvider create(PipelineOptions options) {
- return SimpleTranslatorProvider.newBuilder()
- .registerTranslator(FlatMap.class, new FlatMapTranslator<>())
- .registerTranslator(Union.class, new UnionTranslator<>())
- .registerTranslator(ReduceByKey.class, new ReduceByKeyTranslator<>())
- .registerTranslator(Join.class, new JoinTranslator<>())
- .build();
+ return SimpleTranslatorProvider.create();
}
}
diff --git
a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/NameBasedTranslatorProvider.java
b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/NameBasedTranslatorProvider.java
new file mode 100644
index 000000000000..b54625194f2a
--- /dev/null
+++
b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/NameBasedTranslatorProvider.java
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.euphoria.core.translate;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import java.util.Collection;
+import java.util.Objects;
+import java.util.Optional;
+import org.apache.beam.sdk.annotations.Experimental;
+import
org.apache.beam.sdk.extensions.euphoria.core.client.operator.base.Operator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * An implementation of {@link TranslatorProvider}, which allows for custom
translations of hand
+ * picked operators based on operator names.
+ *
+ * <p>It either selects specific translation or defaults to wrapped {@link
TranslatorProvider}.
+ * Selection of specific translation is based on operator's name and {@link
+ * OperatorTranslator#canTranslate(Operator)} method.
+ *
+ * <p>Specific translation could be added through {@link
Builder#addNameBasedTranslation(Class,
+ * OperatorTranslator, String)} method during build step. Names are considered
match when an
+ * operator's name starts with given prefix.
+ */
+@Experimental
+public class NameBasedTranslatorProvider implements TranslatorProvider {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(NameBasedTranslatorProvider.class);
+
+ private final Multimap<Class<? extends Operator>, TranslationCandidate>
translators;
+ private final TranslatorProvider defaultTranslationProvider;
+
+ private NameBasedTranslatorProvider(
+ Multimap<Class<? extends Operator>, TranslationCandidate> translators,
+ TranslatorProvider defaultTranslationProvider) {
+ this.translators = translators;
+ this.defaultTranslationProvider = defaultTranslationProvider;
+ }
+
+ /**
+ * Filters to all translator that can translate operator. Then choose first
match if translator
+ * starts with shortName in shortTranslatorNameMap or if it operator
contains whole name of
+ * translator. If no match in operator's name it choose first translator it
can translate.
+ */
+ @Override
+ public <InputT, OutputT, OperatorT extends Operator<OutputT>>
+ Optional<OperatorTranslator<InputT, OutputT, OperatorT>>
findTranslator(OperatorT operator) {
+
+ Collection<TranslationCandidate> candidates =
translators.get(operator.getClass());
+
+ Optional<OperatorTranslator<InputT, OutputT, OperatorT>> chosenTranslator =
+ candidates
+ .stream()
+ .filter(candidate ->
candidate.getTranslator().canTranslate(operator))
+ .filter(candidate -> isOperatorNameInTranslator(candidate,
operator))
+ .map(TranslationCandidate::<InputT, OutputT,
OperatorT>getTranslator)
+ .findFirst();
+
+ if (chosenTranslator.isPresent()) {
+ LOG.info("For operator {} was chosen translator {}", operator,
chosenTranslator.get());
+ return chosenTranslator;
+ }
+
+ return defaultTranslationProvider.findTranslator(operator);
+ }
+
+ /**
+ * Create a new builder for provider.
+ *
+ * @return builder
+ */
+ public static NameBasedTranslatorProvider.Builder newBuilder() {
+ return new NameBasedTranslatorProvider.Builder();
+ }
+
+ public <OutputT, OperatorT extends Operator<OutputT>> boolean
isOperatorNameInTranslator(
+ TranslationCandidate translationCandidate, OperatorT operator) {
+ if (!operator.getName().isPresent()) {
+ return false;
+ }
+
+ String translationName =
translationCandidate.getTranslationName().toLowerCase();
+ return operator.getName().get().toLowerCase().startsWith(translationName);
+ }
+
+ /** {@link NameBasedTranslatorProvider} builder. */
+ public static class Builder {
+
+ private final Multimap<Class<? extends Operator>, TranslationCandidate>
translators =
+ ArrayListMultimap.create();
+ private TranslatorProvider defaultTranslationProvider;
+
+ private Builder() {}
+
+ public NameBasedTranslatorProvider.Builder setDefaultTranslationProvider(
+ TranslatorProvider defaultTranslationProvider) {
+ this.defaultTranslationProvider = defaultTranslationProvider;
+ return this;
+ }
+
+ /**
+ * Choose translator for operator, which match:
operator.getName().startsWith(translationName)
+ * (case insensitive). If no match use defaultTranslationProvider.
+ *
+ * @param operatorClass operator class
+ * @param translator instance of OperatorTranslator
+ * @param translationName name to match translator
+ * @return builder
+ */
+ public NameBasedTranslatorProvider.Builder addNameBasedTranslation(
+ Class<? extends Operator> operatorClass,
+ OperatorTranslator<?, ?, ?> translator,
+ String translationName) {
+ translators.put(
+ Objects.requireNonNull(operatorClass),
+ new TranslationCandidate(
+ Objects.requireNonNull(translationName),
Objects.requireNonNull(translator)));
+ return this;
+ }
+
+ public NameBasedTranslatorProvider build() {
+ return new NameBasedTranslatorProvider(
+ translators, Objects.requireNonNull(defaultTranslationProvider));
+ }
+ }
+
+ static class TranslationCandidate {
+
+ final String translationName;
+ final OperatorTranslator<?, ?, ?> translator;
+
+ TranslationCandidate(String translationName, OperatorTranslator<?, ?, ?>
operatorTranslator) {
+ this.translationName = translationName;
+ this.translator = operatorTranslator;
+ }
+
+ String getTranslationName() {
+ return translationName;
+ }
+
+ @SuppressWarnings("unchecked")
+ <InputT, OutputT, OperatorT extends Operator>
+ OperatorTranslator<InputT, OutputT, OperatorT> getTranslator() {
+ return (OperatorTranslator<InputT, OutputT, OperatorT>) translator;
+ }
+ }
+}
diff --git
a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/SimpleTranslatorProvider.java
b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/SimpleTranslatorProvider.java
index 7723054ce97c..46b27a1b8573 100644
---
a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/SimpleTranslatorProvider.java
+++
b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/SimpleTranslatorProvider.java
@@ -21,6 +21,10 @@
import com.google.common.collect.Multimap;
import java.util.Collection;
import java.util.Optional;
+import org.apache.beam.sdk.extensions.euphoria.core.client.operator.FlatMap;
+import org.apache.beam.sdk.extensions.euphoria.core.client.operator.Join;
+import
org.apache.beam.sdk.extensions.euphoria.core.client.operator.ReduceByKey;
+import org.apache.beam.sdk.extensions.euphoria.core.client.operator.Union;
import
org.apache.beam.sdk.extensions.euphoria.core.client.operator.base.Operator;
/**
@@ -28,6 +32,16 @@
*/
public class SimpleTranslatorProvider implements TranslatorProvider {
+ public static SimpleTranslatorProvider create() {
+ return SimpleTranslatorProvider.newBuilder()
+ .registerTranslator(FlatMap.class, new FlatMapTranslator<>())
+ .registerTranslator(Union.class, new UnionTranslator<>())
+ .registerTranslator(ReduceByKey.class, new ReduceByKeyTranslator<>())
+ .registerTranslator(Join.class, new JoinTranslator<>())
+ .registerTranslator(Join.class, new BroadcastHashJoinTranslator<>())
+ .build();
+ }
+
/**
* Create a new builder for provider.
*
diff --git
a/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/testkit/BroadcastHashJoinTest.java
b/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/testkit/BroadcastHashJoinTest.java
index aee805f035ea..f480d99c5b5c 100644
---
a/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/testkit/BroadcastHashJoinTest.java
+++
b/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/testkit/BroadcastHashJoinTest.java
@@ -23,14 +23,13 @@
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.extensions.euphoria.core.client.dataset.Dataset;
import org.apache.beam.sdk.extensions.euphoria.core.client.io.Collector;
-import org.apache.beam.sdk.extensions.euphoria.core.client.operator.FlatMap;
import org.apache.beam.sdk.extensions.euphoria.core.client.operator.Join;
import org.apache.beam.sdk.extensions.euphoria.core.client.operator.LeftJoin;
import
org.apache.beam.sdk.extensions.euphoria.core.client.operator.MapElements;
import org.apache.beam.sdk.extensions.euphoria.core.client.operator.RightJoin;
import
org.apache.beam.sdk.extensions.euphoria.core.translate.BroadcastHashJoinTranslator;
import org.apache.beam.sdk.extensions.euphoria.core.translate.EuphoriaOptions;
-import
org.apache.beam.sdk.extensions.euphoria.core.translate.FlatMapTranslator;
+import
org.apache.beam.sdk.extensions.euphoria.core.translate.NameBasedTranslatorProvider;
import
org.apache.beam.sdk.extensions.euphoria.core.translate.SimpleTranslatorProvider;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TypeDescriptor;
@@ -49,9 +48,10 @@
.getOptions()
.as(EuphoriaOptions.class)
.setTranslatorProvider(
- SimpleTranslatorProvider.newBuilder()
- .registerTranslator(FlatMap.class, new FlatMapTranslator<>())
- .registerTranslator(Join.class, new
BroadcastHashJoinTranslator<>())
+ NameBasedTranslatorProvider.newBuilder()
+
.setDefaultTranslationProvider(SimpleTranslatorProvider.create())
+ .addNameBasedTranslation(
+ Join.class, new BroadcastHashJoinTranslator<>(),
"broadcast")
.build());
return super.getOutput(pipeline);
}
@@ -65,7 +65,8 @@ public void leftBroadcastHashJoin() {
@Override
protected Dataset<KV<Integer, String>> getOutput(
Dataset<Integer> left, Dataset<Long> right) {
- return LeftJoin.of(left, MapElements.of(right).using(i ->
i).output())
+ return LeftJoin.named("broadcast-leftJoin")
+ .of(left, MapElements.of(right).using(i -> i).output())
.by(e -> e, e -> (int) (e % 10))
.using(
(Integer l, Optional<Long> r, Collector<String> c) ->
@@ -118,7 +119,8 @@ public void rightBroadcastHashJoin() {
@Override
protected Dataset<KV<Integer, String>> getOutput(
Dataset<Integer> left, Dataset<Long> right) {
- return RightJoin.of(MapElements.of(left).using(i -> i).output(),
right)
+ return RightJoin.named("BroadcastRightJoin")
+ .of(MapElements.of(left).using(i -> i).output(), right)
.by(e -> e, e -> (int) (e % 10))
.using(
(Optional<Integer> l, Long r, Collector<String> c) ->
@@ -171,7 +173,8 @@ public void keyHashCollisionBroadcastHashJoin() {
@Override
protected Dataset<KV<String, String>> getOutput(
Dataset<String> left, Dataset<Integer> right) {
- return LeftJoin.of(left, MapElements.of(right).using(i ->
i).output())
+ return LeftJoin.named("Broadcast-leftJoin")
+ .of(left, MapElements.of(right).using(i -> i).output())
.by(e -> e, e -> e % 2 == 0 ? sameHashCodeKey2 :
sameHashCodeKey1)
.using(
(String l, Optional<Integer> r, Collector<String> c) ->
diff --git
a/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/translate/NameBaseTranslationProviderTest.java
b/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/translate/NameBaseTranslationProviderTest.java
new file mode 100644
index 000000000000..dd28cf7a7f9f
--- /dev/null
+++
b/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/translate/NameBaseTranslationProviderTest.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.euphoria.core.translate;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Optional;
+import org.apache.beam.sdk.extensions.euphoria.core.client.operator.Distinct;
+import org.apache.beam.sdk.extensions.euphoria.core.client.operator.Join;
+import
org.apache.beam.sdk.extensions.euphoria.core.client.operator.base.Operator;
+import org.junit.Test;
+
+/** NameBaseTranslationProvider testing scenarios. */
+public class NameBaseTranslationProviderTest {
+
+ private Join joinMock = mock(Join.class);
+
+ private final TranslatorProvider translatorProvider =
+ NameBasedTranslatorProvider.newBuilder()
+ .setDefaultTranslationProvider(
+ SimpleTranslatorProvider.newBuilder()
+ .registerTranslator(joinMock.getClass(), new
JoinTranslator<>())
+ .build())
+ .addNameBasedTranslation(
+ joinMock.getClass(), new BroadcastHashJoinTranslator<>(),
"broadcast")
+ .build();
+
+ @Test
+ public void testSimpleFindTranslator() {
+ when(joinMock.getName()).thenReturn(Optional.empty());
+ assertEquals(getTranslatorClass(joinMock), JoinTranslator.class);
+ }
+
+ @Test
+ public void testNameBasedFindTranslator() {
+ when(joinMock.getName()).thenReturn(Optional.of("broadcast"));
+ assertEquals(getTranslatorClass(joinMock),
BroadcastHashJoinTranslator.class);
+ }
+
+ @Test
+ public void testNonMatchingName() {
+ when(joinMock.getName()).thenReturn(Optional.of("hello"));
+ assertEquals(getTranslatorClass(joinMock), JoinTranslator.class);
+ }
+
+ @Test
+ public void testNoAvailableTranslator() {
+ Distinct distinctMock = mock(Distinct.class);
+ assertEquals(getTranslator(distinctMock), Optional.empty());
+ }
+
+ <OperatorT extends Operator<Object>>
+ Optional<OperatorTranslator<Object, Object, OperatorT>>
getTranslator(OperatorT operatorT) {
+ return translatorProvider.findTranslator(operatorT);
+ }
+
+ <OperatorT extends Operator<Object>> Class<?> getTranslatorClass(OperatorT
operatorT) {
+ return getTranslator(operatorT).get().getClass();
+ }
+}
diff --git a/website/src/documentation/sdks/euphoria.md
b/website/src/documentation/sdks/euphoria.md
index 6696ce71ae64..8c38ed319f47 100644
--- a/website/src/documentation/sdks/euphoria.md
+++ b/website/src/documentation/sdks/euphoria.md
@@ -343,7 +343,7 @@ Represents left join of two (left and right) datasets on
given key producing sin
// KV(3, "3+rat"), KV(0, "0+null"), KV(4, "4+duck"), KV(3, "3+cat"),
// KV(3, "3+rat"), KV(1, "1+X")]
```
-Euphoria support performance optimization called 'BroadcastHashJoin' for the
`LeftJoin`. User can indicate through previous operator's output hint
`.output(SizeHint.FITS_IN_MEMORY)` that output `Dataset` of that operator fits
in executors memory. And when the `Dataset` is used as right input, Euphoria
will automatically translated `LeftJoin` as 'BroadcastHashJoin'. Broadcast join
can be very efficient when joining between skewed datasets.
+Euphoria support performance optimization called 'BroadcastHashJoin' for the
`LeftJoin`. Broadcast join can be very efficient when joining two dataset where
one fits in memory (in `LeftJoin` right dataset has to fit in memory). How to
use 'Broadcast Hash Join' check [TranslationProviders](#translationproviders).
### `RightJoin`
Represents right join of two (left and right) datasets on given key producing
single new dataset. Key is extracted from both datasets by separate extractors
so elements in left and right can have different types denoted as `LeftT` and
`RightT`. The join itself is performed by user-supplied `BinaryFunctor` which
consumes one element from both dataset, where left is present optionally,
sharing the same key. And outputs result of the join (`OutputT`). The operator
emits output dataset of `KV<K, OutputT>` type.
@@ -362,7 +362,8 @@ Dataset<KV<Integer, String>> joined =
// KV(4, "4+duck"), KV(3, "3+cat"), KV(3, "3+rat"), KV(1, "1+X"),
// KV(8, "null+elephant"), KV(5, "null+mouse")]
```
-Euphoria support performance optimization called 'Broadcast Hash Join' for the
`RightJoin`. User can indicate through previous operator's output hint
`.output(SizeHint.FITS_IN_MEMORY)` that output `Dataset` of that operator fits
in executors memory. And when the `Dataset` is used as left input, Euphoria
will automatically translated `RightJoin` as 'Broadcast Hash Join'. Broadcast
join can be very efficient when joining between skewed datasets.
+Euphoria support performance optimization called 'Broadcast Hash Join' for the
`RightJoin`. Broadcast join can be very efficient when joining two dataset
where one fits in memory (in `RightJoin` left dataset has to fit in memory).
How to use 'Broadcast Hash Join' check
[TranslationProviders](#translationproviders) section.
+
### `FullJoin`
Represents full outer join of two (left and right) datasets on given key
producing single new dataset. Key is extracted from both datasets by separate
extractors so elements in left and right can have different types denoted as
`LeftT` and `RightT`. The join itself is performed by user-supplied
`BinaryFunctor` which consumes one element from both dataset, where both are
present only optionally, sharing the same key. And outputs result of the join
(`OutputT`). The operator emits output dataset of `KV<K, OutputT>` type.
@@ -567,6 +568,24 @@ Dataset<SomeEventObject> timeStampedEvents =
//Euphoria will now know event time for each event
```
+## TranslationProviders
+The fact that Euphoria API is translated to Beam Java SDK give us option to
fine tune the translation itself. Euphoria uses `TranslationProvider` to decide
which `OperatorTranslator` should be used. User of Euphoria API can supply it
ts own `TranslationProvider` by extending `EuphoriaOptions`. Euphoria already
contains some implementations.
+
+### `SimpleTranslatorProvider`
+Default implementation of `TranslationProvider`. It is able to give default
translators for all Euphoria operators.
+
+### `NameBasedTranslatorProvider`
+ TranslationProvider that selects name matching translation (operator name
starts same as added short name translation). This is useful when we want use
more optimized translation
+ for our operator. E.g. We want to use Broadcast Hash Join translation for
[Join](#join), when one side of join fits in memory.
+
+```java
+NameBasedTranslatorProvider.newBuilder()
+ .setDefaultTranslationProvider(SimpleTranslatorProvider.create())
+ .addNameBasedTranslation(Join.class, new
BroadcastHashJoinTranslator<>(), "broadcast")
+ .build();
+```
+Then `BroadcastHashJoinTranslator` will be used for every `LeftJoin` or
`RightJoin` operator, which will have set name -
`LeftJoin.named("broadcast.....")`.
+
## Euphoria To Beam Translation (advanced user section)
Euphoria API is build on top of Beam Java SDK. The API is transparently
translated into Beam's `PTransforms` in background. Most of the translation
happens in `org.apache.beam.sdk.extensions.euphoria.core.translate` package.
Where the most interesting classes are:
* `OperatorTranslator` - Interface which defining inner API of Euphoria to
Beam translation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 173795)
Time Spent: 3h 20m (was: 3h 10m)
> Operator translation based on operator name
> -------------------------------------------
>
> Key: BEAM-5867
> URL: https://issues.apache.org/jira/browse/BEAM-5867
> Project: Beam
> Issue Type: Sub-task
> Components: dsl-euphoria
> Reporter: Marek Simunek
> Assignee: Marek Simunek
> Priority: Major
> Fix For: Not applicable
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Smarter why to choose operator translation based on operators name by
> introducing {{NameBaseTranslationProviderTest}}. Useful e.g. when we want
> Join operator which fits in memory translate with
> `BroadcastHashJoinTranslator`
> Translation provider that selects name matching translation ( operator name
> starts same as added translation by {{Builder#addShortNameTranslation(Class,
> String)}} or first matching translation for the registered operator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
