OpenOpened commented on a change in pull request #1200: [HUDI-514] A schema
provider to get metadata through Jdbc
URL: https://github.com/apache/incubator-hudi/pull/1200#discussion_r366309913
##########
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java
##########
@@ -236,4 +250,57 @@ public static TypedProperties readConfig(InputStream in)
throws IOException {
defaults.load(in);
return defaults;
}
+
+ /***
+ * call spark function get the schema through jdbc.
+ * @param options
+ * @return
+ * @throws Exception
+ */
+ public static Schema getSchema(Map<String, String> options) throws Exception
{
+ scala.collection.immutable.Map<String, String> ioptions =
toScalaImmutableMap(options);
+ JDBCOptions jdbcOptions = new JDBCOptions(ioptions);
+ Connection conn = JdbcUtils.createConnectionFactory(jdbcOptions).apply();
+ String url = jdbcOptions.url();
+ String table = jdbcOptions.tableOrQuery();
+ JdbcOptionsInWrite jdbcOptionsInWrite = new JdbcOptionsInWrite(ioptions);
+ boolean tableExists = JdbcUtils.tableExists(conn, jdbcOptionsInWrite);
+ if (tableExists) {
+ JdbcDialect dialect = JdbcDialects.get(url);
+ try {
+ PreparedStatement statement =
conn.prepareStatement(dialect.getSchemaQuery(table));
+ try {
+ statement.setQueryTimeout(Integer.parseInt(options.get("timeout")));
+ ResultSet rs = statement.executeQuery();
+ try {
+ StructType structType;
+ if (Boolean.parseBoolean(ioptions.get("nullable").get())) {
+ structType = JdbcUtils.getSchema(rs, dialect, true);
+ } else {
+ structType = JdbcUtils.getSchema(rs, dialect, false);
+ }
+ return
AvroConversionUtils.convertStructTypeToAvroSchema(structType, table, "hoodie."
+ table);
+ } finally {
+ rs.close();
+ }
+ } finally {
+ statement.close();
+ }
+ } finally {
+ conn.close();
+ }
+ } else {
+ throw new HoodieException(String.format("%s table not exists!", table));
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <K, V> scala.collection.immutable.Map<K, V>
toScalaImmutableMap(java.util.Map<K, V> javaMap) {
Review comment:
Because the underlying spark function only accepts parameters of type
scala.collection.immutable.Map, I provide a private conversion function from
java map to scala immutable Map.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
