[
https://issues.apache.org/jira/browse/BEAM-5146?focusedWorklogId=134604&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-134604
]
ASF GitHub Bot logged work on BEAM-5146:
----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Aug/18 17:19
Start Date: 14/Aug/18 17:19
Worklog Time Spent: 10m
Work Description: reuvenlax closed pull request #6218: [BEAM-5146] Fix
inconsistent schemas.
URL: https://github.com/apache/beam/pull/6218
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/GetterBasedSchemaProvider.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/GetterBasedSchemaProvider.java
index 94eba0a5b02..7171878789b 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/GetterBasedSchemaProvider.java
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/GetterBasedSchemaProvider.java
@@ -50,10 +50,14 @@
@Override
public <T> SerializableFunction<T, Row> toRowFunction(TypeDescriptor<T>
typeDescriptor) {
- return o ->
- Row.withSchema(schemaFor(typeDescriptor))
- .withFieldValueGetters(fieldValueGetterFactory(), o)
- .build();
+ // schemaFor is non deterministic - it might return fields in an arbitrary
order. The reason
+ // why is that Java reflection does not guarantee the order in which it
returns fields and
+ // methods, and these schemas are often based on reflective analysis of
classes. Therefore it's
+ // important to capture the schema once here, so all invocations of the
toRowFunction see the
+ // same version of the schema. If schemaFor were to be called inside the
function, different
+ // workers would see different versions of the schema.
+ Schema schema = schemaFor(typeDescriptor);
+ return o ->
Row.withSchema(schema).withFieldValueGetters(fieldValueGetterFactory(),
o).build();
}
@Override
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 134604)
Time Spent: 1.5h (was: 1h 20m)
> GetterBasedSchemaProvider might create inconsistent views of the same schema
> ----------------------------------------------------------------------------
>
> Key: BEAM-5146
> URL: https://issues.apache.org/jira/browse/BEAM-5146
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Reporter: Reuven Lax
> Assignee: Reuven Lax
> Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> schemaFor is called inside the toRowFunction, and schemaFor is not
> deterministic (since Java reflection order is not deterministic). Since the
> toRowFunction is invoked on each worker separately, this creates inconsistent
> views of the schema.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
