hequn8128 commented on a change in pull request #9370: [FLINK-13594][python]
Improve the 'from_element' method of flink python api to apply to blink planner
URL: https://github.com/apache/flink/pull/9370#discussion_r311340002
##########
File path:
flink-python/src/main/java/org/apache/flink/api/common/python/PythonBridgeUtils.java
##########
@@ -44,33 +36,41 @@
*/
public final class PythonBridgeUtils {
- /**
- * Creates a DataStream from a file which contains serialized python
objects.
- */
- public static DataStream<Object[]> createDataStreamFromFile(
- final StreamExecutionEnvironment streamExecutionEnvironment,
- final String fileName,
- final boolean batched) throws IOException {
- return streamExecutionEnvironment
- .fromCollection(readPythonObjects(fileName))
- .flatMap(new PythonFlatMapFunction(batched))
- .returns(Types.GENERIC(Object[].class));
+ private static Object[] getObjectArrayFromUnpickledData(Object input) {
+ if (input.getClass().isArray()) {
+ return (Object[]) input;
+ } else {
+ return ((ArrayList<Object>) input).toArray(new
Object[0]);
+ }
}
- /**
- * Creates a DataSet from a file which contains serialized python
objects.
- */
- public static DataSet<Object[]> createDataSetFromFile(
- final ExecutionEnvironment executionEnvironment,
- final String fileName,
- final boolean batched) throws IOException {
- return executionEnvironment
- .fromCollection(readPythonObjects(fileName))
- .flatMap(new PythonFlatMapFunction(batched))
- .returns(Types.GENERIC(Object[].class));
+ public static List<Object[]> readPythonObjects(String fileName, boolean
batched)
Review comment:
Are there any tests for these methods in this class? It would be good to add
some tests.
Furthermore, maybe better to add some descriptions about these methods.
What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
