reuvenlax commented on a change in pull request #10767: Document Beam Schemas
URL: https://github.com/apache/beam/pull/10767#discussion_r399610840
##########
File path: website/src/documentation/programming-guide.md
##########
@@ -1970,7 +1976,1078 @@ records.apply("WriteToText",
See the [Beam-provided I/O Transforms]({{site.baseurl
}}/documentation/io/built-in/)
page for a list of the currently available I/O transforms.
-## 6. Data encoding and type safety {#data-encoding-and-type-safety}
+## 6. Schemas {#schemas}
+Often, the type of records being processed have an obvious structure. Common
Beam sources produce
+JSON, Avro, Protocol Buffer, or database row objects; all of these types have
well defined structures,
+structures that can often be determined by examining the type. Even within a
SDK pipeline, Simple Java POJOs
+(or equivalent structures in other languages) are often used as intermediate
types, and these also have a
+ clear structure that can be inferred by inspecting the class. By
understanding the structure of a pipeline’s
+ records, we can provide much more concise APIs for data processing.
+
+### 6.1. What is a schema {#what-is-a-schema}
+Most structured records share some common characteristics:
+* They can be subdivided into separate named fields. Fields usually have
string names, but sometimes - as in the case of indexed
+ tuples - have numerical indices instead.
+* There is a confined list of primitive types that a field can have. These
often match primitive types in most programming
+ languages: int, long, string, etc.
+* Often a field type can be marked as optional (sometimes referred to as
nullable) or required.
+
+In addition, often records have a nested structure. A nested structure occurs
when a field itself has subfields so the
+type of the field itself has a schema. Fields that are array or map types is
also a common feature of these structured
+records.
+
+For example, consider the following schema, representing actions in a
fictitious e-commerce company:
+
+**Purchase**
+<table>
+ <thead>
+ <tr class="header">
+ <th><b>Field Name</b></th>
+ <th><b>Field Type</b></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>userId</td>
+ <td>STRING</td>
+ </tr>
+ <tr>
+ <td>itemId</td>
+ <td>INT64</td>
+ </tr>
+ <tr>
+ <td>shippingAddress</td>
+ <td>ROW(ShippingAddress)</td>
+ </tr>
+ <tr>
+ <td>cost</td>
+ <td>INT64</td>
+ </tr>
+ <tr>
+ <td>transactions</td>
+ <td>ARRAY[ROW(Transaction)]</td>
+ </tr>
+ </tbody>
+</table>
+<br/>
+
+**ShippingAddress**
+<table>
+ <thead>
+ <tr class="header">
+ <th><b>Field Name</b></th>
+ <th><b>Field Type</b></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>streetAddress</td>
+ <td>STRING</td>
+ </tr>
+ <tr>
+ <td>city</td>
+ <td>STRING</td>
+ </tr>
+ <tr>
+ <td>state</td>
+ <td>nullable STRING</td>
+ </tr>
+ <tr>
+ <td>country</td>
+ <td>STRING</td>
+ </tr>
+ <tr>
+ <td>postCode</td>
+ <td>STRING</td>
+ </tr>
+ </tbody>
+</table>
+<br/>
+
+**Transaction**
+<table>
+ <thead>
+ <tr class="header">
+ <th><b>Field Name</b></th>
+ <th><b>Field Type</b></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>bank</td>
+ <td>STRING</td>
+ </tr>
+ <tr>
+ <td>purchaseAmount</td>
+ <td>DOUBLE</td>
+ </tr>
+ </tbody>
+</table>
+<br/>
+
+Purchase event records are represented by the above purchase schema. Each
purchase event contains a shipping address, which
+is a nested row containing its own schema. Each purchase also contains a list
of credit-card transactions
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
