joshelser commented on code in PR #80:
URL: https://github.com/apache/phoenix-connectors/pull/80#discussion_r880977239
##########
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixInputPartitionReader.java:
##########
@@ -94,6 +99,10 @@ private QueryPlan getQueryPlan() throws SQLException {
}
try (Connection conn = DriverManager.getConnection(
JDBC_PROTOCOL + JDBC_PROTOCOL_SEPARATOR + zkUrl,
overridingProps)) {
+ PTable pTable = PTable.parseFrom(options.getTableBytes());
+ org.apache.phoenix.schema.PTable table =
PTableImpl.createFromProto(pTable);
+ PhoenixConnection phoenixConnection =
conn.unwrap(PhoenixConnection.class);
+ phoenixConnection.addTable(table, System.currentTimeMillis());
Review Comment:
> Can you think of a case of when the jobs are delayed enough for this to
matter, but enough of them start up synchrounously for the generated load to be
a problem (I don't know enough about Spark to tell) ?
This used to be somethign that would happen in MapReduce with lots of
mappers (where the state of the world might change from job submissions until
the last mappers actually got scheduled and ran). Honestly, I don't think it's
something we can really optimize anyways (and it's not the "common" path that
people would be running `alter table` commands multiple times a day).
> the jobs will not be started immediately, and the syscat load is not
really a problem.
That's a good point too. I didn't think about that side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
