Build failed in Jenkins: hudi-snapshot-deployment-0.5 #183

2020-02-08 Thread Apache Jenkins Server 
See 


Changes:


--
[...truncated 2.24 KB...]
plexus-classworlds-2.5.2.jar

/home/jenkins/tools/maven/apache-maven-3.5.4/conf:
logging
settings.xml
toolchains.xml

/home/jenkins/tools/maven/apache-maven-3.5.4/conf/logging:
simplelogger.properties

/home/jenkins/tools/maven/apache-maven-3.5.4/lib:
aopalliance-1.0.jar
cdi-api-1.0.jar
cdi-api.license
commons-cli-1.4.jar
commons-cli.license
commons-io-2.5.jar
commons-io.license
commons-lang3-3.5.jar
commons-lang3.license
ext
guava-20.0.jar
guice-4.2.0-no_aop.jar
jansi-1.17.1.jar
jansi-native
javax.inject-1.jar
jcl-over-slf4j-1.7.25.jar
jcl-over-slf4j.license
jsr250-api-1.0.jar
jsr250-api.license
maven-artifact-3.5.4.jar
maven-artifact.license
maven-builder-support-3.5.4.jar
maven-builder-support.license
maven-compat-3.5.4.jar
maven-compat.license
maven-core-3.5.4.jar
maven-core.license
maven-embedder-3.5.4.jar
maven-embedder.license
maven-model-3.5.4.jar
maven-model-builder-3.5.4.jar
maven-model-builder.license
maven-model.license
maven-plugin-api-3.5.4.jar
maven-plugin-api.license
maven-repository-metadata-3.5.4.jar
maven-repository-metadata.license
maven-resolver-api-1.1.1.jar
maven-resolver-api.license
maven-resolver-connector-basic-1.1.1.jar
maven-resolver-connector-basic.license
maven-resolver-impl-1.1.1.jar
maven-resolver-impl.license
maven-resolver-provider-3.5.4.jar
maven-resolver-provider.license
maven-resolver-spi-1.1.1.jar
maven-resolver-spi.license
maven-resolver-transport-wagon-1.1.1.jar
maven-resolver-transport-wagon.license
maven-resolver-util-1.1.1.jar
maven-resolver-util.license
maven-settings-3.5.4.jar
maven-settings-builder-3.5.4.jar
maven-settings-builder.license
maven-settings.license
maven-shared-utils-3.2.1.jar
maven-shared-utils.license
maven-slf4j-provider-3.5.4.jar
maven-slf4j-provider.license
org.eclipse.sisu.inject-0.3.3.jar
org.eclipse.sisu.inject.license
org.eclipse.sisu.plexus-0.3.3.jar
org.eclipse.sisu.plexus.license
plexus-cipher-1.7.jar
plexus-cipher.license
plexus-component-annotations-1.7.1.jar
plexus-component-annotations.license
plexus-interpolation-1.24.jar
plexus-interpolation.license
plexus-sec-dispatcher-1.4.jar
plexus-sec-dispatcher.license
plexus-utils-3.1.0.jar
plexus-utils.license
slf4j-api-1.7.25.jar
slf4j-api.license
wagon-file-3.1.0.jar
wagon-file.license
wagon-http-3.1.0-shaded.jar
wagon-http.license
wagon-provider-api-3.1.0.jar
wagon-provider-api.license

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/ext:
README.txt

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native:
freebsd32
freebsd64
linux32
linux64
osx
README.txt
windows32
windows64

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/freebsd32:
libjansi.so

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/freebsd64:
libjansi.so

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/linux32:
libjansi.so

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/linux64:
libjansi.so

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/osx:
libjansi.jnilib

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/windows32:
jansi.dll

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/windows64:
jansi.dll
Finished /home/jenkins/tools/maven/apache-maven-3.5.4 Directory Listing :
Detected current version as: 
'HUDI_home=
0.5.2-SNAPSHOT'
Process leaked file descriptors. See 
https://jenkins.io/redirect/troubleshooting/process-leaked-file-descriptors for 
more information[INFO] Scanning for projects...
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for 
org.apache.hudi:hudi-spark_2.11:jar:0.5.2-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ 
org.apache.hudi:hudi-spark_${scala.binary.version}:[unknown-version], 

 line 26, column 15
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for 
org.apache.hudi:hudi-utilities_2.11:jar:0.5.2-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ 
org.apache.hudi:hudi-utilities_${scala.binary.version}:[unknown-version], 

 line 26, column 15
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for 
org.apache.hudi:hudi-spark-bundle_2.11:jar:0.5.2-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ 
org.apache.hudi:hudi-spark-bundle_${scala.binary.version}:[unknown-version], 

 line 26, column 15
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for

[jira] [Comment Edited] (HUDI-85) Improve bloom index speed using interval trees for range pruning

2020-02-08 Thread sivabalan narayanan (Jira) 


[ 
https://issues.apache.org/jira/browse/HUDI-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033017#comment-17033017
 ] 

sivabalan narayanan edited comment on HUDI-85 at 2/8/20 7:54 PM:
-

[~vinoth]: I remember you had some benchmark w/ interval trees somewhere.Would 
be nice if you can attach to this ticket. 


was (Author: shivnarayan):
[~vinoth]: I remember you had some benchmark w/ interval trees somewhere. If 
you can find it, can you update this ticket w/ the same. We can close it then. 

> Improve bloom index speed using interval trees for range pruning
> 
>
> Key: HUDI-85
> URL: https://issues.apache.org/jira/browse/HUDI-85
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Index, Performance
>Reporter: Vinoth Chandar
>Assignee: Vinoth Chandar
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> https://github.com/apache/incubator-hudi/pull/513



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HUDI-85) Improve bloom index speed using interval trees for range pruning

2020-02-08 Thread sivabalan narayanan (Jira) 


[ 
https://issues.apache.org/jira/browse/HUDI-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033017#comment-17033017
 ] 

sivabalan narayanan commented on HUDI-85:
-

[~vinoth]: I remember you had some benchmark w/ interval trees somewhere. If 
you can find it, can you update this ticket w/ the same. We can close it then. 

> Improve bloom index speed using interval trees for range pruning
> 
>
> Key: HUDI-85
> URL: https://issues.apache.org/jira/browse/HUDI-85
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Index, Performance
>Reporter: Vinoth Chandar
>Assignee: Vinoth Chandar
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> https://github.com/apache/incubator-hudi/pull/513



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-hudi] nsivabalan opened a new pull request #1315: [Hudi-108] Removing 2GB spark partition limitations in HoodieBloomIndex with spark 2.4.4

2020-02-08 Thread GitBox 
nsivabalan opened a new pull request #1315: [Hudi-108] Removing 2GB spark 
partition limitations in HoodieBloomIndex with spark 2.4.4
URL: https://github.com/apache/incubator-hudi/pull/1315
 
 
   ## What is the purpose of the pull request
   - Removing 2GB spark partition limitations in HoodieBloomIndex with spark 
2.4.4
   - Prior to spark 2.4.0, each spark partition has a limit of 2GB. In Hudi 
0.5.1, after we upgraded to spark 2.4.4, we don't have the limitation anymore. 
Hence removing the safe parallelism constraint we had in HoodieBloomIndex. 
   More info can be found here : 
https://issues.apache.org/jira/browse/SPARK-24296
   
   ## Brief change log
   
 - *Removing 2GB spark partition limitations in HoodieBloomIndex with spark 
2.4.4*
   
   ## Verify this pull request
   
   *(Please pick either of the following options)*
   
   This pull request is a trivial rework / code cleanup without any test 
coverage. Only the parallelism is adjusted w/o any business logic change. 
   
   ## Committer checklist
   
- [ ] Has a corresponding JIRA in PR title & commit

- [ ] Commit message is descriptive of the change

- [ ] CI is green
   
- [ ] Necessary doc changes done or have another open PR
  
- [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (HUDI-108) Simplify HoodieBloomIndex without the need for 2GB limit handling

2020-02-08 Thread sivabalan narayanan (Jira) 


 [ 
https://issues.apache.org/jira/browse/HUDI-108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sivabalan narayanan reassigned HUDI-108:


Assignee: sivabalan narayanan  (was: Vinoth Chandar)

> Simplify HoodieBloomIndex without the need for 2GB limit handling
> -
>
> Key: HUDI-108
> URL: https://issues.apache.org/jira/browse/HUDI-108
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Index, Performance
>Reporter: Vinoth Chandar
>Assignee: sivabalan narayanan
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-hudi] OpenOpened commented on issue #1200: [HUDI-514] A schema provider to get metadata through Jdbc

2020-02-08 Thread GitBox 
OpenOpened commented on issue #1200: [HUDI-514] A schema provider to get 
metadata through Jdbc
URL: https://github.com/apache/incubator-hudi/pull/1200#issuecomment-583741643
 
 
   @vinothchandar @leesf please review again


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] OpenOpened commented on a change in pull request #1200: [HUDI-514] A schema provider to get metadata through Jdbc

2020-02-08 Thread GitBox 
OpenOpened commented on a change in pull request #1200: [HUDI-514] A schema 
provider to get metadata through Jdbc
URL: https://github.com/apache/incubator-hudi/pull/1200#discussion_r376708108
 
 

 ##
 File path: 
hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java
 ##
 @@ -235,4 +248,57 @@ public static TypedProperties readConfig(InputStream in) 
throws IOException {
 defaults.load(in);
 return defaults;
   }
+
+  /***
+   * call spark function get the schema through jdbc.
+   * @param options
+   * @return
+   * @throws Exception
+   */
+  public static Schema getJDBCSchema(Map options) throws 
Exception {
+scala.collection.immutable.Map ioptions = 
toScalaImmutableMap(options);
+JDBCOptions jdbcOptions = new JDBCOptions(ioptions);
+Connection conn = JdbcUtils.createConnectionFactory(jdbcOptions).apply();
+String url = jdbcOptions.url();
+String table = jdbcOptions.tableOrQuery();
+JdbcOptionsInWrite jdbcOptionsInWrite = new JdbcOptionsInWrite(ioptions);
+boolean tableExists = JdbcUtils.tableExists(conn, jdbcOptionsInWrite);
+if (tableExists) {
+  JdbcDialect dialect = JdbcDialects.get(url);
+  try {
+PreparedStatement statement = 
conn.prepareStatement(dialect.getSchemaQuery(table));
+try {
+  statement.setQueryTimeout(Integer.parseInt(options.get("timeout")));
+  ResultSet rs = statement.executeQuery();
+  try {
+StructType structType;
+if (Boolean.parseBoolean(ioptions.get("nullable").get())) {
+  structType = JdbcUtils.getSchema(rs, dialect, true);
+} else {
+  structType = JdbcUtils.getSchema(rs, dialect, false);
+}
+return 
AvroConversionUtils.convertStructTypeToAvroSchema(structType, table, "hoodie." 
+ table);
+  } finally {
+rs.close();
+  }
+} finally {
+  statement.close();
+}
+  } finally {
+conn.close();
+  }
 
 Review comment:
   thank you for your advice. i wll do it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] OpenOpened closed pull request #1200: [HUDI-514] A schema provider to get metadata through Jdbc

2020-02-08 Thread GitBox 
OpenOpened closed pull request #1200: [HUDI-514] A schema provider to get 
metadata through Jdbc
URL: https://github.com/apache/incubator-hudi/pull/1200
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] OpenOpened opened a new pull request #1200: [HUDI-514] A schema provider to get metadata through Jdbc

2020-02-08 Thread GitBox 
OpenOpened opened a new pull request #1200: [HUDI-514] A schema provider to get 
metadata through Jdbc
URL: https://github.com/apache/incubator-hudi/pull/1200
 
 
   ## What is the purpose of the pull request
   
   In our production environment, we usually need to synchronize data from 
mysql, and at the same time, we need to get the schema from the database. So I 
submitted this PR. A schema provider that obtains metadata through Jdbc calls 
the Spark JDBC related methods by design. And ensure the uniformity of the 
schema, such as reading historical data from spark jdbc, and Use delta streamer 
to synchronize data.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] leesf commented on a change in pull request #1200: [HUDI-514] A schema provider to get metadata through Jdbc

2020-02-08 Thread GitBox 
leesf commented on a change in pull request #1200: [HUDI-514] A schema provider 
to get metadata through Jdbc
URL: https://github.com/apache/incubator-hudi/pull/1200#discussion_r376705690
 
 

 ##
 File path: 
hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java
 ##
 @@ -235,4 +248,57 @@ public static TypedProperties readConfig(InputStream in) 
throws IOException {
 defaults.load(in);
 return defaults;
   }
+
+  /***
+   * call spark function get the schema through jdbc.
+   * @param options
+   * @return
+   * @throws Exception
+   */
+  public static Schema getJDBCSchema(Map options) throws 
Exception {
+scala.collection.immutable.Map ioptions = 
toScalaImmutableMap(options);
+JDBCOptions jdbcOptions = new JDBCOptions(ioptions);
+Connection conn = JdbcUtils.createConnectionFactory(jdbcOptions).apply();
+String url = jdbcOptions.url();
+String table = jdbcOptions.tableOrQuery();
+JdbcOptionsInWrite jdbcOptionsInWrite = new JdbcOptionsInWrite(ioptions);
+boolean tableExists = JdbcUtils.tableExists(conn, jdbcOptionsInWrite);
+if (tableExists) {
+  JdbcDialect dialect = JdbcDialects.get(url);
+  try {
+PreparedStatement statement = 
conn.prepareStatement(dialect.getSchemaQuery(table));
+try {
+  statement.setQueryTimeout(Integer.parseInt(options.get("timeout")));
+  ResultSet rs = statement.executeQuery();
+  try {
+StructType structType;
+if (Boolean.parseBoolean(ioptions.get("nullable").get())) {
+  structType = JdbcUtils.getSchema(rs, dialect, true);
+} else {
+  structType = JdbcUtils.getSchema(rs, dialect, false);
+}
+return 
AvroConversionUtils.convertStructTypeToAvroSchema(structType, table, "hoodie." 
+ table);
+  } finally {
+rs.close();
+  }
+} finally {
+  statement.close();
+}
+  } finally {
+conn.close();
+  }
 
 Review comment:
   Would be changed to `try-with-resources`? since `try-catch-finnaly` has been 
optimized to `try-with-resources` in the project now. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua edited a comment on issue #1309: [HUDI-592] Remove duplicated dependencies in the pom file of test suite module

2020-02-08 Thread GitBox 
yanghua edited a comment on issue #1309: [HUDI-592] Remove duplicated 
dependencies in the pom file of test suite module
URL: https://github.com/apache/incubator-hudi/pull/1309#issuecomment-583722209
 
 
   > > @n3nash It's very strange. The travis always fails caused by job log 
exceed or timeout.
   > 
   > Seems like you need cherry-pick some avro related commits from master 
branch?
   
   Which commits? It seems I have fixed the avro specific issues. See pr #1297 
and after that pr, there are no big changes. However, Travis starts to fail.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua edited a comment on issue #1309: [HUDI-592] Remove duplicated dependencies in the pom file of test suite module

2020-02-08 Thread GitBox 
yanghua edited a comment on issue #1309: [HUDI-592] Remove duplicated 
dependencies in the pom file of test suite module
URL: https://github.com/apache/incubator-hudi/pull/1309#issuecomment-583722209
 
 
   > > @n3nash It's very strange. The travis always fails caused by job log 
exceed or timeout.
   > 
   > Seems like you need cherry-pick some avro related commits from master 
branch?
   
   Which commits? It seems I have fixed the avro specific issues. See pr #1297 
and after that pr, there is no big changes. However, Travis starts to fail.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on issue #1309: [HUDI-592] Remove duplicated dependencies in the pom file of test suite module

2020-02-08 Thread GitBox 
yanghua commented on issue #1309: [HUDI-592] Remove duplicated dependencies in 
the pom file of test suite module
URL: https://github.com/apache/incubator-hudi/pull/1309#issuecomment-583722209
 
 
   > > @n3nash It's very strange. The travis always fails caused by job log 
exceed or timeout.
   > 
   > Seems like you need cherry-pick some avro related commits from master 
branch?
   
   Which commits? It seems I have fixed the avro specific issues. See pr #1297 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] leesf commented on issue #1309: [HUDI-592] Remove duplicated dependencies in the pom file of test suite module

2020-02-08 Thread GitBox 
leesf commented on issue #1309: [HUDI-592] Remove duplicated dependencies in 
the pom file of test suite module
URL: https://github.com/apache/incubator-hudi/pull/1309#issuecomment-583720166
 
 
   > @n3nash It's very strange. The travis always fails caused by job log 
exceed or timeout.
   
   Seems like you need cherry-pick some avro related commits from master branch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] leesf commented on issue #1313: [SUPPORT]used in test inveroment

2020-02-08 Thread GitBox 
leesf commented on issue #1313: [SUPPORT]used in test inveroment
URL: https://github.com/apache/incubator-hudi/issues/1313#issuecomment-583719769
 
 
   @c-f-cooper would you please provide more info?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (HUDI-542) Introduce a new pom module named hudi-writer-common

2020-02-08 Thread vinoyang (Jira) 


 [ 
https://issues.apache.org/jira/browse/HUDI-542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

vinoyang resolved HUDI-542.
---
Resolution: Done

Done via restructure-hudi-client branch: 
a1c2296d4e02baccd47155e341029df70604325c

> Introduce a new pom module named hudi-writer-common
> ---
>
> Key: HUDI-542
> URL: https://issues.apache.org/jira/browse/HUDI-542
> Project: Apache Hudi (incubating)
>  Issue Type: Sub-task
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We try to restructure the {{hudi-client}} module. Some classes would be put 
> into a new module named {{hudi-writer-common}}. So we should introduce a new 
> module.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-hudi] yanghua merged pull request #1314: [HUDI-542] Introduce a new pom module named hudi-writer-common

2020-02-08 Thread GitBox 
yanghua merged pull request #1314: [HUDI-542] Introduce a new pom module named 
hudi-writer-common
URL: https://github.com/apache/incubator-hudi/pull/1314
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on issue #1314: [HUDI-542] Introduce a new pom module named hudi-writer-common

2020-02-08 Thread GitBox 
yanghua commented on issue #1314: [HUDI-542] Introduce a new pom module named 
hudi-writer-common
URL: https://github.com/apache/incubator-hudi/pull/1314#issuecomment-583715237
 
 
   @vinothchandar @leesf FYI: new module `hudi-writer-common` on 
`restructure-hudi-client`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-hudi] branch restructure-hudi-client updated: [HUDI-542] Introduce a new pom module named hudi-writer-common (#1314)

2020-02-08 Thread vinoyang 
This is an automated email from the ASF dual-hosted git repository.

vinoyang pushed a commit to branch restructure-hudi-client
in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git


The following commit(s) were added to refs/heads/restructure-hudi-client by 
this push:
 new e7b1961  [HUDI-542] Introduce a new pom module named 
hudi-writer-common (#1314)
e7b1961 is described below

commit e7b1961de082b626fdb27890eb4ff701f11dffb4
Author: vinoyang 
AuthorDate: Sat Feb 8 16:20:33 2020 +0800

[HUDI-542] Introduce a new pom module named hudi-writer-common (#1314)
---
 hudi-writer-common/pom.xml | 15 +
 .../org/apache/hudi/writer/common/Placeholder.java | 26 ++
 .../apache/hudi/writer/common/PlaceholderTest.java | 26 ++
 pom.xml|  1 +
 4 files changed, 68 insertions(+)

diff --git a/hudi-writer-common/pom.xml b/hudi-writer-common/pom.xml
new file mode 100644
index 000..eed8c5a
--- /dev/null
+++ b/hudi-writer-common/pom.xml
@@ -0,0 +1,15 @@
+
+http://maven.apache.org/POM/4.0.0;
+  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+  
+hudi
+org.apache.hudi
+0.5.2-SNAPSHOT
+  
+  4.0.0
+
+  hudi-writer-common
+
+
+
\ No newline at end of file
diff --git 
a/hudi-writer-common/src/main/java/org/apache/hudi/writer/common/Placeholder.java
 
b/hudi-writer-common/src/main/java/org/apache/hudi/writer/common/Placeholder.java
new file mode 100644
index 000..93d8a35
--- /dev/null
+++ 
b/hudi-writer-common/src/main/java/org/apache/hudi/writer/common/Placeholder.java
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.writer.common;
+
+/**
+ * Used for placeholder purpose.
+ */
+public class Placeholder {
+
+}
diff --git 
a/hudi-writer-common/src/test/java/org/apache/hudi/writer/common/PlaceholderTest.java
 
b/hudi-writer-common/src/test/java/org/apache/hudi/writer/common/PlaceholderTest.java
new file mode 100644
index 000..2a10a58
--- /dev/null
+++ 
b/hudi-writer-common/src/test/java/org/apache/hudi/writer/common/PlaceholderTest.java
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.writer.common;
+
+/**
+ * Used for placeholder purpose.
+ */
+public class PlaceholderTest {
+
+}
diff --git a/pom.xml b/pom.xml
index 8bdb4a6..b6782ad 100644
--- a/pom.xml
+++ b/pom.xml
@@ -51,6 +51,7 @@
 packaging/hudi-timeline-server-bundle
 docker/hoodie/hadoop
 hudi-integ-test
+hudi-writer-common

[jira] [Commented] (HUDI-554) Restructure code/packages to move more code back into hudi-writer-common

2020-02-08 Thread vinoyang (Jira) 


[ 
https://issues.apache.org/jira/browse/HUDI-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032843#comment-17032843
 ] 

vinoyang commented on HUDI-554:
---

[~vinoth] Do you want to move and refactor all the classes via this ticket?

> Restructure code/packages  to move more code back into hudi-writer-common
> -
>
> Key: HUDI-554
> URL: https://issues.apache.org/jira/browse/HUDI-554
> Project: Apache Hudi (incubating)
>  Issue Type: Sub-task
>  Components: Code Cleanup
>Reporter: Vinoth Chandar
>Assignee: Vinoth Chandar
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HUDI-538) Restructuring hudi client module for multi engine support

2020-02-08 Thread vinoyang (Jira) 


 [ 
https://issues.apache.org/jira/browse/HUDI-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

vinoyang updated HUDI-538:
--
Status: In Progress  (was: Open)

> Restructuring hudi client module for multi engine support
> -
>
> Key: HUDI-538
> URL: https://issues.apache.org/jira/browse/HUDI-538
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Code Cleanup
>Reporter: vinoyang
>Priority: Major
>
> Hudi is currently tightly coupled with the Spark framework. It caused the 
> integration with other computing engine more difficult. We plan to decouple 
> it with Spark. This umbrella issue used to track this work.
> Some thoughts wrote here: 
> https://docs.google.com/document/d/1Q9w_4K6xzGbUrtTS0gAlzNYOmRXjzNUdbbe0q59PX9w/edit?usp=sharing
> The feature branch is {{restructure-hudi-client}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HUDI-538) Restructuring hudi client module for multi engine support

2020-02-08 Thread vinoyang (Jira) 


 [ 
https://issues.apache.org/jira/browse/HUDI-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

vinoyang updated HUDI-538:
--
Issue Type: Improvement  (was: Wish)

> Restructuring hudi client module for multi engine support
> -
>
> Key: HUDI-538
> URL: https://issues.apache.org/jira/browse/HUDI-538
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Code Cleanup
>Reporter: vinoyang
>Priority: Major
>
> Hudi is currently tightly coupled with the Spark framework. It caused the 
> integration with other computing engine more difficult. We plan to decouple 
> it with Spark. This umbrella issue used to track this work.
> Some thoughts wrote here: 
> https://docs.google.com/document/d/1Q9w_4K6xzGbUrtTS0gAlzNYOmRXjzNUdbbe0q59PX9w/edit?usp=sharing
> The feature branch is {{restructure-hudi-client}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HUDI-538) Restructuring hudi client module for multi engine support

2020-02-08 Thread vinoyang (Jira) 


 [ 
https://issues.apache.org/jira/browse/HUDI-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

vinoyang updated HUDI-538:
--
Status: Open  (was: New)

> Restructuring hudi client module for multi engine support
> -
>
> Key: HUDI-538
> URL: https://issues.apache.org/jira/browse/HUDI-538
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Code Cleanup
>Reporter: vinoyang
>Priority: Major
>
> Hudi is currently tightly coupled with the Spark framework. It caused the 
> integration with other computing engine more difficult. We plan to decouple 
> it with Spark. This umbrella issue used to track this work.
> Some thoughts wrote here: 
> https://docs.google.com/document/d/1Q9w_4K6xzGbUrtTS0gAlzNYOmRXjzNUdbbe0q59PX9w/edit?usp=sharing
> The feature branch is {{restructure-hudi-client}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)