Github user roshannaik commented on a diff in the pull request:
https://github.com/apache/storm/pull/1914#discussion_r99724571
--- Diff: docs/Joins.md ---
@@ -0,0 +1,102 @@
+---
+title: Joining Streams in Storm Core
+layout: documentation
+documentation: true
+---
+
+Storm core supports joining multiple data streams into one with the help
of `JoinBolt`.
+`JoinBolt` is a Windowed bolt, i.e. it waits for the configured window
duration to match up the
+tuples among the streams being joined. This helps align the streams within
the Window boundary.
+
+Each of `JoinBolt`'s incoming data streams must be Fields Grouped on a
single field. A stream
+should only be joined with the other streams using the field on which it
has been FieldsGrouped.
+Knowing this will help understand the join syntax described below.
+
+## Performing Joins
+Consider the following SQL join involving 4 tables called: stream1,
stream2, stream3 & stream4:
+
+```sql
+select userId, key4, key2, key3
+from stream1
+join stream2 on stream2.userId = stream1.key1
+join stream3 on stream3.key3 = stream2.userId
+left join stream4 on stream4.key4 = stream3.key3
+```
+
+This could be expressed using `JoinBolt` over 4 similarly named streams
as:
+
+```java
+new JoinBolt(JoinBolt.Selector.STREAM, "stream1", "key1") //
from stream1
+ .join ("stream2", "userId", "stream1") //
join stream2 on stream2.userId = stream1.key1
--- End diff --
Yes good question. As of now, grouping on single key and consequently
joining on 1 key only is supported ... noted in the doc,
To get the effect of joining on multiple fields you need to combine
two/more fields into one, somewhere upstream. Then you can join on that merged
field. Perhaps worth mentioning this in docs.
To be more inline with SQL abilities, I started supporting streams grouped
on multiple fields for joins initially, but later scaled it back to single key.
Saw a few issues with opening up grouping and joins on multiple fields. Let me
try to summarize...
It easily gives users the wrong impression that they can do many SQL-type
things .. like
Assuming S1 is grouped on both f1 & f2:
```
from s1
join s2 s2.f1 == s1.f1 && s2.f2 == s1.f2 // uses both f1 & f2 form s1
join s3 s3.f1 == s1.f1 && s3.f2 == s2.f2 // one field from s1 and
another from s2
```
which cannot work correctly due to FG.
Another example is grouping S1 on f1 and f2 and then trying to do this...
```
from s1
join s2 s2.f1 == s1.f1 // use s1.f1
join s3 s3.f1 == s1.f2 // but s1.f2 here
```
**In short:** The extent to which multi field joins can be correctly
supported is limited to whatever can be achieved by merging those fields into
one before joining. Going beyond it complicated the interface and the internal
checking... in addition to easily setting up wrong user expectations.
The downside of current approach is an additional step to merge the fields
in such cases. Thats the trade off I settled for.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
