Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 merged PR #4815: URL: https://github.com/apache/gravitino/pull/4815 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 commented on PR #4791: URL: https://github.com/apache/gravitino/pull/4791#issuecomment-2320660204 @coolderli , thanks for your work! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 closed pull request #4815: [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector URL: https://github.com/apache/gravitino/pull/4815 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 merged PR #4791: URL: https://github.com/apache/gravitino/pull/4791 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 commented on code in PR #4791:
URL: https://github.com/apache/gravitino/pull/4791#discussion_r1738299874
##
flink-connector/flink/build.gradle.kts:
##
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+repositories {
+ mavenCentral()
+}
+
+val flinkVersion: String = libs.versions.flink.get()
+val flinkMajorVersion: String = flinkVersion.substringBeforeLast(".")
+
+
+// The Flink only support scala 2.12, and all scala api will be removed in a
future version.
+// You can find more detail at the following issues:
+// https://issues.apache.org/jira/browse/FLINK-23986,
+// https://issues.apache.org/jira/browse/FLINK-20845,
+// https://issues.apache.org/jira/browse/FLINK-13414.
+val scalaVersion: String = "2.12"
+val artifactName =
"${rootProject.name}-flink-${flinkMajorVersion}_$scalaVersion"
+
+dependencies {
+ implementation(project(":api"))
+ implementation(project(":catalogs:catalog-common"))
+ implementation(project(":common"))
+ implementation(project(":core"))
Review Comment:
I'm afraid there maybe something unexpected, I will merge this PR to kick
off 0.6.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
mchades commented on code in PR #4791:
URL: https://github.com/apache/gravitino/pull/4791#discussion_r1738254480
##
flink-connector/flink/build.gradle.kts:
##
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+repositories {
+ mavenCentral()
+}
+
+val flinkVersion: String = libs.versions.flink.get()
+val flinkMajorVersion: String = flinkVersion.substringBeforeLast(".")
+
+
+// The Flink only support scala 2.12, and all scala api will be removed in a
future version.
+// You can find more detail at the following issues:
+// https://issues.apache.org/jira/browse/FLINK-23986,
+// https://issues.apache.org/jira/browse/FLINK-20845,
+// https://issues.apache.org/jira/browse/FLINK-13414.
+val scalaVersion: String = "2.12"
+val artifactName =
"${rootProject.name}-flink-${flinkMajorVersion}_$scalaVersion"
+
+dependencies {
+ implementation(project(":api"))
+ implementation(project(":catalogs:catalog-common"))
+ implementation(project(":common"))
+ implementation(project(":core"))
Review Comment:
> @mchades could you create a separate issue to track this?
If this is just a minor fix, it is recommended to fix it in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 commented on code in PR #4791:
URL: https://github.com/apache/gravitino/pull/4791#discussion_r1738239258
##
flink-connector/flink/build.gradle.kts:
##
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+repositories {
+ mavenCentral()
+}
+
+val flinkVersion: String = libs.versions.flink.get()
+val flinkMajorVersion: String = flinkVersion.substringBeforeLast(".")
+
+
+// The Flink only support scala 2.12, and all scala api will be removed in a
future version.
+// You can find more detail at the following issues:
+// https://issues.apache.org/jira/browse/FLINK-23986,
+// https://issues.apache.org/jira/browse/FLINK-20845,
+// https://issues.apache.org/jira/browse/FLINK-13414.
+val scalaVersion: String = "2.12"
+val artifactName =
"${rootProject.name}-flink-${flinkMajorVersion}_$scalaVersion"
+
+dependencies {
+ implementation(project(":api"))
+ implementation(project(":catalogs:catalog-common"))
+ implementation(project(":common"))
+ implementation(project(":core"))
Review Comment:
@mchades could you create a separate issue to track this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 commented on code in PR #4791:
URL: https://github.com/apache/gravitino/pull/4791#discussion_r1738239258
##
flink-connector/flink/build.gradle.kts:
##
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+repositories {
+ mavenCentral()
+}
+
+val flinkVersion: String = libs.versions.flink.get()
+val flinkMajorVersion: String = flinkVersion.substringBeforeLast(".")
+
+
+// The Flink only support scala 2.12, and all scala api will be removed in a
future version.
+// You can find more detail at the following issues:
+// https://issues.apache.org/jira/browse/FLINK-23986,
+// https://issues.apache.org/jira/browse/FLINK-20845,
+// https://issues.apache.org/jira/browse/FLINK-13414.
+val scalaVersion: String = "2.12"
+val artifactName =
"${rootProject.name}-flink-${flinkMajorVersion}_$scalaVersion"
+
+dependencies {
+ implementation(project(":api"))
+ implementation(project(":catalogs:catalog-common"))
+ implementation(project(":common"))
+ implementation(project(":core"))
Review Comment:
@mchades could you create a separate PR to track this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
mchades commented on code in PR #4791:
URL: https://github.com/apache/gravitino/pull/4791#discussion_r1738219194
##
flink-connector/flink/build.gradle.kts:
##
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+repositories {
+ mavenCentral()
+}
+
+val flinkVersion: String = libs.versions.flink.get()
+val flinkMajorVersion: String = flinkVersion.substringBeforeLast(".")
+
+
+// The Flink only support scala 2.12, and all scala api will be removed in a
future version.
+// You can find more detail at the following issues:
+// https://issues.apache.org/jira/browse/FLINK-23986,
+// https://issues.apache.org/jira/browse/FLINK-20845,
+// https://issues.apache.org/jira/browse/FLINK-13414.
+val scalaVersion: String = "2.12"
+val artifactName =
"${rootProject.name}-flink-${flinkMajorVersion}_$scalaVersion"
+
+dependencies {
+ implementation(project(":api"))
+ implementation(project(":catalogs:catalog-common"))
+ implementation(project(":common"))
+ implementation(project(":core"))
Review Comment:
I am confused why you need to depend on `core` module?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
mchades commented on code in PR #4791:
URL: https://github.com/apache/gravitino/pull/4791#discussion_r1738193021
##
flink-connector/flink/build.gradle.kts:
##
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+repositories {
+ mavenCentral()
+}
+
+val flinkVersion: String = libs.versions.flink.get()
+val flinkMajorVersion: String = flinkVersion.substringBeforeLast(".")
+
+
+// The Flink only support scala 2.12, and all scala api will be removed in a
future version.
+// You can find more detail at the following issues:
+// https://issues.apache.org/jira/browse/FLINK-23986,
+// https://issues.apache.org/jira/browse/FLINK-20845,
+// https://issues.apache.org/jira/browse/FLINK-13414.
+val scalaVersion: String = "2.12"
+val artifactName =
"${rootProject.name}-flink-${flinkMajorVersion}_$scalaVersion"
+
+dependencies {
+ implementation(project(":api"))
+ implementation(project(":catalogs:catalog-common"))
+ implementation(project(":common"))
+ implementation(project(":core"))
Review Comment:
you can add ` {
exclude("*")
} ` to shrink the size of lib
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 commented on PR #4791: URL: https://github.com/apache/gravitino/pull/4791#issuecomment-2320473871 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 commented on code in PR #4791: URL: https://github.com/apache/gravitino/pull/4791#discussion_r1738078996 ## docs/flink-connector/flink-connector.md: ## @@ -23,7 +23,7 @@ This capability allows users to perform federation queries, accessing data from ## How to use it -1. [Build](../how-to-build.md) or [download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector) the Gravitino flink connector jar, and place it to the classpath of Flink. +1. [Build](../how-to-build.md) or [download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector) the Gravitino flink connector runtime jar, and place it to the classpath of Flink. Review Comment: I mean the runtime jar download link doesn't contain scala version ## docs/flink-connector/flink-connector.md: ## @@ -23,7 +23,7 @@ This capability allows users to perform federation queries, accessing data from ## How to use it -1. [Build](../how-to-build.md) or [download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector) the Gravitino flink connector jar, and place it to the classpath of Flink. +1. [Build](../how-to-build.md) or [download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector) the Gravitino flink connector runtime jar, and place it to the classpath of Flink. Review Comment: https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-spark-connector-runtime-3.4 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
coolderli commented on code in PR #4791: URL: https://github.com/apache/gravitino/pull/4791#discussion_r1738054080 ## docs/flink-connector/flink-connector.md: ## @@ -23,7 +23,7 @@ This capability allows users to perform federation queries, accessing data from ## How to use it -1. [Build](../how-to-build.md) or [download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector) the Gravitino flink connector jar, and place it to the classpath of Flink. +1. [Build](../how-to-build.md) or [download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector) the Gravitino flink connector runtime jar, and place it to the classpath of Flink. Review Comment: @FANNG1 But I saw the spark has the scala version(_2.12).  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 commented on code in PR #4791:
URL: https://github.com/apache/gravitino/pull/4791#discussion_r1738036722
##
docs/flink-connector/flink-connector.md:
##
@@ -23,7 +23,7 @@ This capability allows users to perform federation queries,
accessing data from
## How to use it
-1. [Build](../how-to-build.md) or
[download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector)
the Gravitino flink connector jar, and place it to the classpath of Flink.
+1. [Build](../how-to-build.md) or
[download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector)
the Gravitino flink connector runtime jar, and place it to the classpath of
Flink.
Review Comment:
we should use `1.18` not `1.18.0` here? and link should be
`https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector-runtime-1.18`
like spark not include scala versions.
```
val flinkMajorVersion: String = flink.substringBeforeLast(".")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 commented on code in PR #4791:
URL: https://github.com/apache/gravitino/pull/4791#discussion_r1738033371
##
settings.gradle.kts:
##
@@ -66,3 +67,5 @@ project(":spark-connector:spark-runtime-3.5").projectDir =
file("spark-connector
include("web")
include("docs")
include("integration-test-common")
+include("flink-connector:flink")
Review Comment:
seems these two lines are not necessary? it will include flink module if
scalaVersion is 2.13
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 commented on code in PR #4791:
URL: https://github.com/apache/gravitino/pull/4791#discussion_r1738032214
##
flink-connector/flink/build.gradle.kts:
##
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+repositories {
+ mavenCentral()
+}
+
+val flinkVersion: String = libs.versions.flink.get()
+
+// The Flink only support scala 2.12, and all scala api will be removed in a
future version.
+// You can find more detail at the following issues:
+// https://issues.apache.org/jira/browse/FLINK-23986,
+// https://issues.apache.org/jira/browse/FLINK-20845,
+// https://issues.apache.org/jira/browse/FLINK-13414.
+val scalaVersion: String = "2.12"
+val artifactName = "gravitino-${project.name}_$scalaVersion"
+
+dependencies {
+ implementation(project(":api"))
+ implementation(project(":catalogs:catalog-common"))
+ implementation(project(":common"))
+ implementation(project(":core"))
+
+ compileOnly(libs.bundles.log4j)
+ implementation(libs.commons.lang3)
+ implementation(libs.guava)
+ implementation(libs.httpclient5)
+ implementation(libs.jackson.databind)
+ implementation(libs.jackson.annotations)
+ implementation(libs.jackson.datatype.jdk8)
+ implementation(libs.jackson.datatype.jsr310)
+
+
compileOnly("org.apache.flink:flink-connector-hive_$scalaVersion:$flinkVersion")
+ compileOnly("org.apache.flink:flink-table-common:$flinkVersion")
+ compileOnly("org.apache.flink:flink-table-api-java:$flinkVersion")
+
+ compileOnly(libs.hive2.exec) {
Review Comment:
add testImplement(libs.hive2.exec) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 commented on PR #4791: URL: https://github.com/apache/gravitino/pull/4791#issuecomment-2319976104 LGTM, except minor comments, @coolderli could you check it again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 commented on code in PR #4791: URL: https://github.com/apache/gravitino/pull/4791#discussion_r1737881730 ## docs/flink-connector/flink-connector.md: ## @@ -23,7 +23,7 @@ This capability allows users to perform federation queries, accessing data from ## How to use it -1. [Build](../how-to-build.md) or [download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector) the Gravitino flink connector jar, and place it to the classpath of Flink. +1. [Build](../how-to-build.md) or [download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector) the Gravitino flink connector runtime jar, and place it to the classpath of Flink. Review Comment: so the download link should be `https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector-runtime-1.18.0_2.12`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 commented on code in PR #4791:
URL: https://github.com/apache/gravitino/pull/4791#discussion_r1737871984
##
flink-connector/flink/build.gradle.kts:
##
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+repositories {
+ mavenCentral()
+}
+
+val flinkVersion: String = libs.versions.flink.get()
+
+// The Flink only support scala 2.12, and all scala api will be removed in a
future version.
+// You can find more detail at the following issues:
+// https://issues.apache.org/jira/browse/FLINK-23986,
+// https://issues.apache.org/jira/browse/FLINK-20845,
+// https://issues.apache.org/jira/browse/FLINK-13414.
+val scalaVersion: String = "2.12"
+val artifactName = "gravitino-${project.name}_$scalaVersion"
+
+dependencies {
+ implementation(project(":api"))
+ implementation(project(":catalogs:catalog-common"))
+ implementation(project(":common"))
+ implementation(project(":core"))
+
+ implementation(libs.bundles.log4j)
+ implementation(libs.commons.lang3)
+ implementation(libs.guava)
+ implementation(libs.httpclient5)
+ implementation(libs.jackson.databind)
+ implementation(libs.jackson.annotations)
+ implementation(libs.jackson.datatype.jdk8)
+ implementation(libs.jackson.datatype.jsr310)
+
+
implementation("org.apache.flink:flink-connector-hive_$scalaVersion:$flinkVersion")
Review Comment:
I perfer not to add the jar because most users using have should download
the jar and may conflict with the `flink-connector-hive`, same for
`flink-table-common` `flink-table-api-java`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
yuqi1129 commented on code in PR #4791: URL: https://github.com/apache/gravitino/pull/4791#discussion_r1737647094 ## docs/flink-connector/flink-connector.md: ## @@ -23,7 +23,7 @@ This capability allows users to perform federation queries, accessing data from ## How to use it -1. [Build](../how-to-build.md) or [download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector) the Gravitino flink connector jar, and place it to the classpath of Flink. +1. [Build](../how-to-build.md) or [download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector) the Gravitino flink connector runtime jar, and place it to the classpath of Flink. Review Comment: There is no artifact named `gravitino-flink-connector-runtime`, only `gravitino-flink-connector-runtime-1.18.0_2.12` and `gravitino-flink_2.12`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
coolderli commented on code in PR #4791:
URL: https://github.com/apache/gravitino/pull/4791#discussion_r1737683579
##
flink-connector/flink/build.gradle.kts:
##
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+repositories {
+ mavenCentral()
+}
+
+val flinkVersion: String = libs.versions.flink.get()
+
+// The Flink only support scala 2.12, and all scala api will be removed in a
future version.
+// You can find more detail at the following issues:
+// https://issues.apache.org/jira/browse/FLINK-23986,
+// https://issues.apache.org/jira/browse/FLINK-20845,
+// https://issues.apache.org/jira/browse/FLINK-13414.
+val scalaVersion: String = "2.12"
+val artifactName = "gravitino-${project.name}_$scalaVersion"
+
+dependencies {
+ implementation(project(":api"))
+ implementation(project(":catalogs:catalog-common"))
+ implementation(project(":common"))
+ implementation(project(":core"))
+
+ implementation(libs.bundles.log4j)
+ implementation(libs.commons.lang3)
+ implementation(libs.guava)
+ implementation(libs.httpclient5)
+ implementation(libs.jackson.databind)
+ implementation(libs.jackson.annotations)
+ implementation(libs.jackson.datatype.jdk8)
+ implementation(libs.jackson.datatype.jsr310)
+
+
implementation("org.apache.flink:flink-connector-hive_$scalaVersion:$flinkVersion")
Review Comment:
@FANNG1 I'm not sure if we should package the flink-connector-hive. Should
we assume that the engine side already has it？Flink does not add this
dependency by default and users need to download it themselves.
https://nightlies.apache.org/flink/flink-docs-release-1.17/zh/docs/connectors/table/hive/overview/#%e4%bd%bf%e7%94%a8-flink-%e6%8f%90%e4%be%9b%e7%9a%84-hive-jar
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 commented on code in PR #4791:
URL: https://github.com/apache/gravitino/pull/4791#discussion_r1737639046
##
flink-connector/flink/build.gradle.kts:
##
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+repositories {
+ mavenCentral()
+}
+
+val flinkVersion: String = libs.versions.flink.get()
+
+// The Flink only support scala 2.12, and all scala api will be removed in a
future version.
+// You can find more detail at the following issues:
+// https://issues.apache.org/jira/browse/FLINK-23986,
+// https://issues.apache.org/jira/browse/FLINK-20845,
+// https://issues.apache.org/jira/browse/FLINK-13414.
+val scalaVersion: String = "2.12"
+val artifactName = "gravitino-${project.name}_$scalaVersion"
+
+dependencies {
+ implementation(project(":api"))
+ implementation(project(":catalogs:catalog-common"))
+ implementation(project(":common"))
+ implementation(project(":core"))
+
+ implementation(libs.bundles.log4j)
Review Comment:
From the experience of the spark connector, there may be class conflict with
log4j, I wonder whether to exclude it or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 commented on code in PR #4791:
URL: https://github.com/apache/gravitino/pull/4791#discussion_r1737636502
##
settings.gradle.kts:
##
@@ -66,3 +67,7 @@ project(":spark-connector:spark-runtime-3.5").projectDir =
file("spark-connector
include("web")
include("docs")
include("integration-test-common")
+include("flink-connector:flink")
+findProject(":flink-connector:flink")?.name = "flink"
+include("flink-connector:flink-runtime")
+findProject(":flink-connector:flink-runtime")?.name = "flink-runtime"
Review Comment:
what's the purpose of setting the name? I failed to run `./gradlew
flink-runtime:build` is it expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
yuqi1129 commented on code in PR #4791: URL: https://github.com/apache/gravitino/pull/4791#discussion_r1737633310 ## docs/flink-connector/flink-connector.md: ## @@ -23,7 +23,7 @@ This capability allows users to perform federation queries, accessing data from ## How to use it -1. [Build](../how-to-build.md) or [download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector) the Gravitino flink connector jar, and place it to the classpath of Flink. +1. [Build](../how-to-build.md) or [download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector) the Gravitino flink connector runtime jar, and place it to the classpath of Flink. Review Comment: Let me check it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 commented on code in PR #4791:
URL: https://github.com/apache/gravitino/pull/4791#discussion_r1737632838
##
flink-connector/flink/build.gradle.kts:
##
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+repositories {
+ mavenCentral()
+}
+
+val flinkVersion: String = libs.versions.flink.get()
+
+// The Flink only support scala 2.12, and all scala api will be removed in a
future version.
+// You can find more detail at the following issues:
+// https://issues.apache.org/jira/browse/FLINK-23986,
+// https://issues.apache.org/jira/browse/FLINK-20845,
+// https://issues.apache.org/jira/browse/FLINK-13414.
+val scalaVersion: String = "2.12"
+val artifactName = "gravitino-${project.name}_$scalaVersion"
+
+dependencies {
+ implementation(project(":api"))
+ implementation(project(":catalogs:catalog-common"))
+ implementation(project(":common"))
+ implementation(project(":core"))
+
+ implementation(libs.bundles.log4j)
+ implementation(libs.commons.lang3)
+ implementation(libs.guava)
+ implementation(libs.httpclient5)
+ implementation(libs.jackson.databind)
+ implementation(libs.jackson.annotations)
+ implementation(libs.jackson.datatype.jdk8)
+ implementation(libs.jackson.datatype.jsr310)
+
+
implementation("org.apache.flink:flink-connector-hive_$scalaVersion:$flinkVersion")
Review Comment:
should we package flink jar to the flink connector? as the flink connector
is run under the flink enviroment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 commented on code in PR #4791: URL: https://github.com/apache/gravitino/pull/4791#discussion_r1737631605 ## docs/flink-connector/flink-connector.md: ## @@ -23,7 +23,7 @@ This capability allows users to perform federation queries, accessing data from ## How to use it -1. [Build](../how-to-build.md) or [download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector) the Gravitino flink connector jar, and place it to the classpath of Flink. +1. [Build](../how-to-build.md) or [download](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector) the Gravitino flink connector runtime jar, and place it to the classpath of Flink. Review Comment: @yuqi please help to verify that the flink runtime jar link will be `https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-flink-connector-runtime` ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
coolderli commented on PR #4791: URL: https://github.com/apache/gravitino/pull/4791#issuecomment-2319610635 > Hi @coolderli without this runtime jar, how do we run Flink Gravitino connector previously? @jerryshao Just use the flink connector jar. But there are some conflicts here. Therefore, a shaded operation was performed in the runtime. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
FANNG1 commented on code in PR #4791:
URL: https://github.com/apache/gravitino/pull/4791#discussion_r1737539414
##
flink-connector/flink/build.gradle.kts:
##
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+repositories {
+ mavenCentral()
+}
+
+val flinkVersion: String = libs.versions.flink.get()
+
+// The Flink only support scala 2.12, and all scala api will be removed in a
future version.
+// You can find more detail at the following issues:
+// https://issues.apache.org/jira/browse/FLINK-23986,
+// https://issues.apache.org/jira/browse/FLINK-20845,
+// https://issues.apache.org/jira/browse/FLINK-13414.
+val scalaVersion: String = "2.12"
+val artifactName = "gravitino-${project.name}_$scalaVersion"
+
+dependencies {
+ implementation(project(":api"))
+ implementation(project(":catalogs:catalog-common"))
+ implementation(project(":common"))
+ implementation(project(":core"))
+ implementation(project(":clients:client-java"))
Review Comment:
compileOnly(":clients:client-java") ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
jerryshao commented on PR #4791: URL: https://github.com/apache/gravitino/pull/4791#issuecomment-2318274603 Hi @coolderli without this runtime jar, how do we run Flink Gravitino connector previously? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#3514] improvement(flink-connector): add flink-connector-runtime to build flink connector [gravitino]
coolderli commented on PR #4791: URL: https://github.com/apache/gravitino/pull/4791#issuecomment-2318259438 @FANNG1 PTAL. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
