[
https://issues.apache.org/jira/browse/BEAM-8664?focusedWorklogId=353640&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353640
]
ASF GitHub Bot logged work on BEAM-8664:
----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Dec/19 18:38
Start Date: 04/Dec/19 18:38
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on pull request #10095:
[BEAM-8664] [SQL] MongoDb project push down
URL: https://github.com/apache/beam/pull/10095#discussion_r353914423
##########
File path:
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbReadWriteIT.java
##########
@@ -187,7 +199,68 @@ public void testWriteAndRead() {
PCollection<Row> output =
BeamSqlRelUtils.toPCollection(readPipeline, sqlEnv.parseQuery("select
* from TEST"));
- assertEquals(output.getSchema(), SOURCE_SCHEMA);
+ assertEquals(SOURCE_SCHEMA, output.getSchema());
+
+ PAssert.that(output).containsInAnyOrder(testRow);
+
+ readPipeline.run().waitUntilFinish();
+ }
+
+ @Test
+ public void testProjectPushDown() {
+ final Schema expectedSchema =
+ Schema.builder()
+ .addNullableField("c_varchar", STRING)
+ .addNullableField("c_boolean", BOOLEAN)
+ .addNullableField("c_integer", INT32)
+ .build();
+ Row testRow = row(expectedSchema, "varchar", true, 2147483647);
+
+ String createTableStatement =
+ "CREATE EXTERNAL TABLE TEST( \n"
+ + " c_bigint BIGINT, \n "
+ + " c_tinyint TINYINT, \n"
+ + " c_smallint SMALLINT, \n"
+ + " c_integer INTEGER, \n"
+ + " c_float FLOAT, \n"
+ + " c_double DOUBLE, \n"
+ + " c_boolean BOOLEAN, \n"
+ + " c_varchar VARCHAR, \n "
+ + " c_arr ARRAY<VARCHAR> \n"
+ + ") \n"
+ + "TYPE 'mongodb' \n"
+ + "LOCATION '"
+ + mongoSqlUrl
+ + "'";
+ sqlEnv.executeDdl(createTableStatement);
+
+ String insertStatement =
+ "INSERT INTO TEST VALUES ("
+ + "9223372036854775807, "
+ + "127, "
+ + "32767, "
+ + "2147483647, "
+ + "1.0, "
+ + "1.0, "
+ + "TRUE, "
+ + "'varchar', "
+ + "ARRAY['123', '456']"
+ + ")";
+
+ BeamRelNode insertRelNode = sqlEnv.parseQuery(insertStatement);
+ BeamSqlRelUtils.toPCollection(writePipeline, insertRelNode);
+ writePipeline.run().waitUntilFinish();
+
+ BeamRelNode node = sqlEnv.parseQuery("select c_varchar, c_boolean,
c_integer from TEST");
+ // Calc should be dropped, since MongoDb supports project push-down and
field reordering.
+ assertThat(node, instanceOf(BeamPushDownIOSourceRel.class));
Review comment:
> I also looked into retrieving some sort of log (from the embedded
database) to check what query actually got executed, but I am not sure that we
can easily retrieve that data.
This is the sort of thing I had in mind. Making an assertion about the query
fn is a little better, but I still think that should be an implementation
detail from the perspective of this test.
We're in sort of a weird position, something like a mock mongodb client
would make this really easy, but that won't work for us since we need to be
able to reference a hostname and port for the mongo instance.
Your link about setting the profile level seems promising - it looks like if
we do that we could get a list of operations/queries from the [`system.profile`
collection](https://docs.mongodb.com/manual/tutorial/manage-the-database-profiler/#view-profiler-data).
We could query that and assert there's just one operation with the expected
projection. That seems like the best bet to me. It's kind of a pain though, so
if you want to punt on it I'd be fine with just filing a jira describing this
approach and referencing it here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 353640)
Time Spent: 4h 10m (was: 4h)
> [SQL] MongoDb should use project push-down
> ------------------------------------------
>
> Key: BEAM-8664
> URL: https://issues.apache.org/jira/browse/BEAM-8664
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
> Reporter: Kirill Kozlov
> Assignee: Kirill Kozlov
> Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> MongoDbTable should implement the following methods:
> {code:java}
> public PCollection<Row> buildIOReader(
> PBegin begin, BeamSqlTableFilter filters, List<String> fieldNames);
> public ProjectSupport supportsProjects();
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
