[
https://issues.apache.org/jira/browse/HIVE-30067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramit Gupta updated HIVE-30067:
-------------------------------
Description:
Geospatial UDFs that use JTS GeoJSON ( like
[GeoJsonReader.java|[https://github.com/locationtech/jts/blob/master/modules/io/common/src/main/java/org/locationtech/jts/io/geojson/GeoJsonReader.java])]
via jts-io-common e.g. in
[[GeometryUtils.java|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/esri/GeometryUtils.java]|
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/esri/GeometryUtils.java]]
depend on {{json-simple}} for parsing
It can be reproduced locally via
{code:java}
mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver
-Dqfile=geospatial_udfs.q -Dtest.output.overwrite{code}
running into
{code:java}
ClassNotFoundException: org.json.simple.parser.ParseException{code}
due to absance of json-simple in test classpath
Why it is not picked in CI?
CI uses
[https://github.com/apache/hive/blob/4cf07f490af49d0e0e019d07c45b629ed8802136/Jenkinsfile#L408]
, whose similar locally working eg. :
{code:java}
mvn -Pitests org.apache.maven.plugins:maven-surefire-plugin:test
Dtest=TestMiniLlapLocalCliDriver.java -Dqfile=geospatial_udfs.q
{code}
since it is ran from project root, due to which maven resolves the entire
dependency chain (including transitive ones) and get pulled into test
classpath due to single reactor for all modules as -pl is not specified
Similarly at runtime all the transitive deps are part of classpath so it
doesn't get picked up there as well.
{*}Note{*}: This change only focus on Deployments and tests that rely on a
single {{hive-exec-*.jar}} like the case I mentioned of MiniLlap driver above.
was:
Geospatial UDFs that use JTS GeoJSON ( like
[GeoJsonReader.java|[https://github.com/locationtech/jts/blob/master/modules/io/common/src/main/java/org/locationtech/jts/io/geojson/GeoJsonReader.java])]
via jts-io-common e.g. in
[[GeometryUtils.java|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/esri/GeometryUtils.java]|
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/esri/GeometryUtils.java]]
depend on {{json-simple}} for parsing
It can be reproduced locally via
{code:java}
mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver
-Dqfile=geospatial_udfs.q -Dtest.output.overwrite{code}
running into
{code:java}
ClassNotFoundException: org.json.simple.parser.ParseException{code}
due to absance of json-simple in test classpath
Why it is not picked in CI?
CI uses
[https://github.com/apache/hive/blob/4cf07f490af49d0e0e019d07c45b629ed8802136/Jenkinsfile#L408]
, whose similar locally working eg. :
{code:java}
mvn -Pitests org.apache.maven.plugins:maven-surefire-plugin:test
Dtest=TestMiniLlapLocalCliDriver.java -Dqfile=geospatial_udfs.q
{code}
since it is ran from project root, due to which maven resolves the entire
dependency chain (including transitive ones) and get pulled into test
classpath due to single reactor for all modules as -pl is not specified
Similarly at runtime all the transitive deps are part of classpath so it
doesn't get picked up there as well.
{*}Note{*}: This change only focus on Deployments and tests that rely on a
single {{hive-exec-*.jar}} like the case I mentioned of MiniLlap driver above.
> Include json-simple in hive-exec shaded jar for JTS GeoJSON
> -----------------------------------------------------------
>
> Key: HIVE-30067
> URL: https://issues.apache.org/jira/browse/HIVE-30067
> Project: Hive
> Issue Type: Bug
> Reporter: Ramit Gupta
> Assignee: Ramit Gupta
> Priority: Minor
>
> Geospatial UDFs that use JTS GeoJSON ( like
> [GeoJsonReader.java|[https://github.com/locationtech/jts/blob/master/modules/io/common/src/main/java/org/locationtech/jts/io/geojson/GeoJsonReader.java])]
> via jts-io-common e.g. in
> [[GeometryUtils.java|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/esri/GeometryUtils.java]|
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/esri/GeometryUtils.java]]
> depend on {{json-simple}} for parsing
> It can be reproduced locally via
> {code:java}
> mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=geospatial_udfs.q -Dtest.output.overwrite{code}
> running into
> {code:java}
> ClassNotFoundException: org.json.simple.parser.ParseException{code}
> due to absance of json-simple in test classpath
> Why it is not picked in CI?
> CI uses
> [https://github.com/apache/hive/blob/4cf07f490af49d0e0e019d07c45b629ed8802136/Jenkinsfile#L408]
> , whose similar locally working eg. :
> {code:java}
> mvn -Pitests org.apache.maven.plugins:maven-surefire-plugin:test
> Dtest=TestMiniLlapLocalCliDriver.java -Dqfile=geospatial_udfs.q
> {code}
> since it is ran from project root, due to which maven resolves the entire
> dependency chain (including transitive ones) and get pulled into test
> classpath due to single reactor for all modules as -pl is not specified
> Similarly at runtime all the transitive deps are part of classpath so it
> doesn't get picked up there as well.
> {*}Note{*}: This change only focus on Deployments and tests that rely on a
> single {{hive-exec-*.jar}} like the case I mentioned of MiniLlap driver above.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
