[GitHub] [incubator-hudi] adamjoneill edited a comment on issue #1325: presto - querying nested object in parquet file created by hudi

adamjoneill edited a comment on issue #1325: presto - querying nested object in parquet file created by hudi URL: https://github.com/apache/incubator-hudi/issues/1325#issuecomment-585338502 i've managed to narrow down the issue to the data that is coming off the kinesis stream.

[GitHub] [incubator-hudi] yanghua commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end

yanghua commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end URL: https://github.com/apache/incubator-hudi/pull/1100#issuecomment-585549211 > sg. In that case, I would like to re-ask my intiial questions again.. How

Build failed in Jenkins: hudi-snapshot-deployment-0.5 #187

See Changes: -- [...truncated 2.28 KB...] plexus-classworlds-2.5.2.jar /home/jenkins/tools/maven/apache-maven-3.5.4/conf: logging settings.xml toolchains.xml

[GitHub] [incubator-hudi] vinothchandar commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end

vinothchandar commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end URL: https://github.com/apache/incubator-hudi/pull/1100#issuecomment-585528405 sg. In that case, I would like to re-ask my intiial questions again..

[GitHub] [incubator-hudi] yanghua commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end

yanghua commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end URL: https://github.com/apache/incubator-hudi/pull/1100#issuecomment-585522237 > Or is it this PR? Yes

[GitHub] [incubator-hudi] yanghua commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end

yanghua commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end URL: https://github.com/apache/incubator-hudi/pull/1100#issuecomment-585519006 > @yanghua merged the pom PR, can you rebase this so we can merge this PR ?

[jira] [Updated] (HUDI-607) Hive sync fails to register tables partitioned by Date Type column

[ https://issues.apache.org/jira/browse/HUDI-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-607: Labels: pull-request-available (was: ) > Hive sync fails to register tables partitioned by Date

[GitHub] [incubator-hudi] umehrot2 opened a new pull request #1330: [HUDI-607] Fix to allow creation/syncing of Hive tables partitioned by Date type columns

umehrot2 opened a new pull request #1330: [HUDI-607] Fix to allow creation/syncing of Hive tables partitioned by Date type columns URL: https://github.com/apache/incubator-hudi/pull/1330 ## What is the purpose of the pull request The issue is well described in

[GitHub] [incubator-hudi] bwu2 edited a comment on issue #1328: Hudi upsert hangs

bwu2 edited a comment on issue #1328: Hudi upsert hangs URL: https://github.com/apache/incubator-hudi/issues/1328#issuecomment-585512613 @vinothchandar Thanks for taking the time to reply! Let me describe the simplest example of this problem on a tiny COW data set: Create a data

[GitHub] [incubator-hudi] bwu2 edited a comment on issue #1328: Hudi upsert hangs

bwu2 edited a comment on issue #1328: Hudi upsert hangs URL: https://github.com/apache/incubator-hudi/issues/1328#issuecomment-585512613 @vinothchandar Thanks for taking the time to reply! Let me describe the simplest example of this problem on a tiny COW data set: Create a data

[GitHub] [incubator-hudi] bwu2 edited a comment on issue #1328: Hudi upsert hangs

bwu2 edited a comment on issue #1328: Hudi upsert hangs URL: https://github.com/apache/incubator-hudi/issues/1328#issuecomment-585512613 @vinothchandar Thanks for taking the time to reply! Let me describe the simplest example of this problem on a tiny COW data set: Create a data

[GitHub] [incubator-hudi] bwu2 edited a comment on issue #1328: Hudi upsert hangs

bwu2 edited a comment on issue #1328: Hudi upsert hangs URL: https://github.com/apache/incubator-hudi/issues/1328#issuecomment-585512613 @vinothchandar Thanks for taking the time to reply! Let me describe the simplest example of this problem on a tiny COW data set: Create a data

[GitHub] [incubator-hudi] bwu2 edited a comment on issue #1328: Hudi upsert hangs

bwu2 edited a comment on issue #1328: Hudi upsert hangs URL: https://github.com/apache/incubator-hudi/issues/1328#issuecomment-585512613 @vinothchandar Thanks for taking the time to reply! Let me describe the simplest example of this problem on a tiny COW data set: Create a data

[jira] [Assigned] (HUDI-607) Hive sync fails to register tables partitioned by Date Type column

[ https://issues.apache.org/jira/browse/HUDI-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udit Mehrotra reassigned HUDI-607: -- Assignee: Udit Mehrotra > Hive sync fails to register tables partitioned by Date Type column >

[jira] [Created] (HUDI-607) Hive sync fails to register tables partitioned by Date Type column

Udit Mehrotra created HUDI-607: -- Summary: Hive sync fails to register tables partitioned by Date Type column Key: HUDI-607 URL: https://issues.apache.org/jira/browse/HUDI-607 Project: Apache Hudi

[GitHub] [incubator-hudi] vinothchandar commented on issue #1325: presto - querying nested object in parquet file created by hudi

vinothchandar commented on issue #1325: presto - querying nested object in parquet file created by hudi URL: https://github.com/apache/incubator-hudi/issues/1325#issuecomment-585504515 cc @bhasudha as well This is an

[GitHub] [incubator-hudi] vinothchandar commented on issue #1329: [SUPPORT] Presto cannot query non-partitioned table

vinothchandar commented on issue #1329: [SUPPORT] Presto cannot query non-partitioned table URL: https://github.com/apache/incubator-hudi/issues/1329#issuecomment-585504364 @bhasudha can you please help out here This is an

[GitHub] [incubator-hudi] vinothchandar edited a comment on issue #1328: Hudi upsert hangs

vinothchandar edited a comment on issue #1328: Hudi upsert hangs URL: https://github.com/apache/incubator-hudi/issues/1328#issuecomment-585503340 Reposting my response here.. There seems to be a lot of common concerns here..

[GitHub] [incubator-hudi] vinothchandar commented on issue #1328: Hudi upsert hangs

vinothchandar commented on issue #1328: Hudi upsert hangs URL: https://github.com/apache/incubator-hudi/issues/1328#issuecomment-585503340 Reposting my response here.. There seems to be a lot of common concerns here.. https://cwiki.apache.org/confluence/display/HUDI/Tuning+Guide is

[GitHub] [incubator-hudi] vinothchandar commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end

vinothchandar commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end URL: https://github.com/apache/incubator-hudi/pull/1100#issuecomment-585487447 Or is it this PR?

[GitHub] [incubator-hudi] vinothchandar commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end

vinothchandar commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end URL: https://github.com/apache/incubator-hudi/pull/1100#issuecomment-585487391 @n3nash @yanghua once ready, please open a PR against master.. Love to

[GitHub] [incubator-hudi] lamber-ken commented on issue #1326: org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0

lamber-ken commented on issue #1326: org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0 URL: https://github.com/apache/incubator-hudi/issues/1326#issuecomment-585482228 @matthewLiem welcome :) **Record some notes:** Change

[GitHub] [incubator-hudi] bwu2 edited a comment on issue #1328: Hudi upsert hangs

bwu2 edited a comment on issue #1328: Hudi upsert hangs URL: https://github.com/apache/incubator-hudi/issues/1328#issuecomment-585471970 An update on this: the job did eventually complete. But took 2.4 hours (other similar jobs ran more than 9 hours before we had to kill them). This

[GitHub] [incubator-hudi] bwu2 commented on issue #1328: Hudi upsert hangs

bwu2 commented on issue #1328: Hudi upsert hangs URL: https://github.com/apache/incubator-hudi/issues/1328#issuecomment-585471970 An update on this: the job did eventually complete. But took 2.4 hours (other similar jobs ran more than 9 hours before we had to kill them). This doesn't seem

[GitHub] [incubator-hudi] matthewLiem closed issue #1326: org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0

matthewLiem closed issue #1326: org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0 URL: https://github.com/apache/incubator-hudi/issues/1326 This is an automated message

[GitHub] [incubator-hudi] matthewLiem commented on issue #1326: org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0

matthewLiem commented on issue #1326: org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0 URL: https://github.com/apache/incubator-hudi/issues/1326#issuecomment-585446206 Thanks for the help @lamber-ken. The issue was due to a mismatch in

[GitHub] [incubator-hudi] n3nash commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end

n3nash commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end URL: https://github.com/apache/incubator-hudi/pull/1100#issuecomment-585444365 @yanghua merged the pom PR, can you rebase this so we can merge this PR ? I

[incubator-hudi] branch master updated (d2c872e -> 63b4216)

This is an automated email from the ASF dual-hosted git repository. nagarwal pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git. from d2c872e [HUDI-605] Avoid calculating the size of schema redundantly (#1317) add 63b4216 CLI - add

[GitHub] [incubator-hudi] n3nash merged pull request #1318: [HUDI-571] CLI - add option to print additional commit metadata

n3nash merged pull request #1318: [HUDI-571] CLI - add option to print additional commit metadata URL: https://github.com/apache/incubator-hudi/pull/1318 This is an automated message from the Apache Git Service. To respond

[GitHub] [incubator-hudi] popart opened a new issue #1329: [SUPPORT] Presto cannot query non-partitioned table

popart opened a new issue #1329: [SUPPORT] Presto cannot query non-partitioned table URL: https://github.com/apache/incubator-hudi/issues/1329 **Describe the problem you faced** I made a non-partitioned Hudi table using Spark. I was able to query it with Spark & Presto, but when I

[jira] [Commented] (HUDI-407) Implement a join-based index

[ https://issues.apache.org/jira/browse/HUDI-407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035721#comment-17035721 ] Vinoth Chandar commented on HUDI-407: - [~shivnarayan] are you working on this? if not, would love to

[jira] [Updated] (HUDI-554) Restructure code/packages to move more code back into hudi-writer-common

[ https://issues.apache.org/jira/browse/HUDI-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-554: Fix Version/s: 0.6.0 > Restructure code/packages to move more code back into hudi-writer-common >

[jira] [Commented] (HUDI-554) Restructure code/packages to move more code back into hudi-writer-common

[ https://issues.apache.org/jira/browse/HUDI-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035718#comment-17035718 ] Vinoth Chandar commented on HUDI-554: - [~yanghua] sg.. a branch seems good when we actually break it up

[GitHub] [incubator-hudi] bwu2 opened a new issue #1328: Hudi upsert hangs

bwu2 opened a new issue #1328: Hudi upsert hangs URL: https://github.com/apache/incubator-hudi/issues/1328 **Describe the problem you faced** When we upsert data into Hudi, we're finding that the job just hangs in some cases. Specifically, we have an ETL pipeline where we re-ingest a

[GitHub] [incubator-hudi] satishkotha commented on issue #1320: [HUDI-571] Add min/max headers on archived files

satishkotha commented on issue #1320: [HUDI-571] Add min/max headers on archived files URL: https://github.com/apache/incubator-hudi/pull/1320#issuecomment-585398180 @n3nash thanks. looks like this may need some more discussion. Take a look at my reply and let me know what you think.

[GitHub] [incubator-hudi] satishkotha commented on a change in pull request #1320: [HUDI-571] Add min/max headers on archived files

satishkotha commented on a change in pull request #1320: [HUDI-571] Add min/max headers on archived files URL: https://github.com/apache/incubator-hudi/pull/1320#discussion_r378489085 ## File path:

[GitHub] [incubator-hudi] satishkotha commented on a change in pull request #1320: [HUDI-571] Add min/max headers on archived files

satishkotha commented on a change in pull request #1320: [HUDI-571] Add min/max headers on archived files URL: https://github.com/apache/incubator-hudi/pull/1320#discussion_r378485704 ## File path: hudi-client/src/main/java/org/apache/hudi/io/HoodieCommitArchiveLog.java

[GitHub] [incubator-hudi] satishkotha commented on a change in pull request #1320: [HUDI-571] Add min/max headers on archived files

satishkotha commented on a change in pull request #1320: [HUDI-571] Add min/max headers on archived files URL: https://github.com/apache/incubator-hudi/pull/1320#discussion_r378484755 ## File path:

[GitHub] [incubator-hudi] n3nash commented on issue #1309: [HUDI-592] Remove duplicated dependencies in the pom file of test suite module

n3nash commented on issue #1309: [HUDI-592] Remove duplicated dependencies in the pom file of test suite module URL: https://github.com/apache/incubator-hudi/pull/1309#issuecomment-585392157 Retriggered the build, it passes, i think there is still some flakiness in the build, lets

[incubator-hudi] branch hudi_test_suite_refactor updated (044759a -> 5f22849)

This is an automated email from the ASF dual-hosted git repository. nagarwal pushed a change to branch hudi_test_suite_refactor in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git. from 044759a [HUDI-503] Add hudi test suite documentation into the README file of the test

[GitHub] [incubator-hudi] n3nash merged pull request #1309: [HUDI-592] Remove duplicated dependencies in the pom file of test suite module

n3nash merged pull request #1309: [HUDI-592] Remove duplicated dependencies in the pom file of test suite module URL: https://github.com/apache/incubator-hudi/pull/1309 This is an automated message from the Apache Git

[GitHub] [incubator-hudi] satishkotha commented on a change in pull request #1320: [HUDI-571] Add min/max headers on archived files

satishkotha commented on a change in pull request #1320: [HUDI-571] Add min/max headers on archived files URL: https://github.com/apache/incubator-hudi/pull/1320#discussion_r378480109 ## File path: hudi-client/src/main/java/org/apache/hudi/io/HoodieCommitArchiveLog.java

[GitHub] [incubator-hudi] satishkotha commented on issue #1318: [HUDI-571] CLI - add option to print additional commit metadata

satishkotha commented on issue #1318: [HUDI-571] CLI - add option to print additional commit metadata URL: https://github.com/apache/incubator-hudi/pull/1318#issuecomment-585385524 @n3nash addressed comments. Please take a look.

[GitHub] [incubator-hudi] satishkotha commented on a change in pull request #1312: [HUDI-571] Add "compactions show archived" command to CLI

satishkotha commented on a change in pull request #1312: [HUDI-571] Add "compactions show archived" command to CLI URL: https://github.com/apache/incubator-hudi/pull/1312#discussion_r378475212 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/CompactionCommand.java

[GitHub] [incubator-hudi] satishkotha commented on a change in pull request #1318: [HUDI-571] CLI - add option to print additional commit metadata

satishkotha commented on a change in pull request #1318: [HUDI-571] CLI - add option to print additional commit metadata URL: https://github.com/apache/incubator-hudi/pull/1318#discussion_r378471663 ## File path:

[GitHub] [incubator-hudi] adamjoneill edited a comment on issue #1325: presto - querying nested object in parquet file created by hudi

adamjoneill edited a comment on issue #1325: presto - querying nested object in parquet file created by hudi URL: https://github.com/apache/incubator-hudi/issues/1325#issuecomment-585338502 i've managed to narrow down the issue to the data that is coming off the kinesis stream.

[GitHub] [incubator-hudi] hmatu commented on a change in pull request #1242: [HUDI-544] Adjust the read and write path of archive

hmatu commented on a change in pull request #1242: [HUDI-544] Adjust the read and write path of archive URL: https://github.com/apache/incubator-hudi/pull/1242#discussion_r378446313 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java

[GitHub] [incubator-hudi] n3nash commented on issue #1216: [HUDI-525] lack of insert info in delta_commit inflight

n3nash commented on issue #1216: [HUDI-525] lack of insert info in delta_commit inflight URL: https://github.com/apache/incubator-hudi/pull/1216#issuecomment-585358877 @liujianhuiouc on second thoughts, do you have a definite use case for knowing the number of inserts during a failed

[GitHub] [incubator-hudi] adamjoneill commented on issue #1324: Presto - select * from table does not work

adamjoneill commented on issue #1324: Presto - select * from table does not work URL: https://github.com/apache/incubator-hudi/issues/1324#issuecomment-585357996 i'll close this, i believe to be related to https://github.com/apache/incubator-hudi/issues/1325

[GitHub] [incubator-hudi] adamjoneill closed issue #1324: Presto - select * from table does not work

adamjoneill closed issue #1324: Presto - select * from table does not work URL: https://github.com/apache/incubator-hudi/issues/1324 This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [incubator-hudi] n3nash commented on a change in pull request #1242: [HUDI-544] Adjust the read and write path of archive

n3nash commented on a change in pull request #1242: [HUDI-544] Adjust the read and write path of archive URL: https://github.com/apache/incubator-hudi/pull/1242#discussion_r378441023 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java

[GitHub] [incubator-hudi] n3nash commented on a change in pull request #1318: [HUDI-571] CLI - add option to print additional commit metadata

n3nash commented on a change in pull request #1318: [HUDI-571] CLI - add option to print additional commit metadata URL: https://github.com/apache/incubator-hudi/pull/1318#discussion_r378435736 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/CommitsCommand.java

[GitHub] [incubator-hudi] nbalajee commented on a change in pull request #1312: [HUDI-571] Add "compactions show archived" command to CLI

nbalajee commented on a change in pull request #1312: [HUDI-571] Add "compactions show archived" command to CLI URL: https://github.com/apache/incubator-hudi/pull/1312#discussion_r378433086 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/CompactionCommand.java

[GitHub] [incubator-hudi] n3nash commented on a change in pull request #1320: [HUDI-571] Add min/max headers on archived files

n3nash commented on a change in pull request #1320: [HUDI-571] Add min/max headers on archived files URL: https://github.com/apache/incubator-hudi/pull/1320#discussion_r378431282 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java

[GitHub] [incubator-hudi] n3nash commented on a change in pull request #1320: [HUDI-571] Add min/max headers on archived files

n3nash commented on a change in pull request #1320: [HUDI-571] Add min/max headers on archived files URL: https://github.com/apache/incubator-hudi/pull/1320#discussion_r378427516 ## File path: hudi-client/src/main/java/org/apache/hudi/io/HoodieCommitArchiveLog.java ## @@

[GitHub] [incubator-hudi] n3nash commented on a change in pull request #1320: [HUDI-571] Add min/max headers on archived files

n3nash commented on a change in pull request #1320: [HUDI-571] Add min/max headers on archived files URL: https://github.com/apache/incubator-hudi/pull/1320#discussion_r378425612 ## File path:

[GitHub] [incubator-hudi] n3nash commented on a change in pull request #1320: [HUDI-571] Add min/max headers on archived files

n3nash commented on a change in pull request #1320: [HUDI-571] Add min/max headers on archived files URL: https://github.com/apache/incubator-hudi/pull/1320#discussion_r378424618 ## File path: hudi-client/src/main/java/org/apache/hudi/io/HoodieCommitArchiveLog.java ## @@

[GitHub] [incubator-hudi] adamjoneill edited a comment on issue #1325: presto - querying nested object in parquet file created by hudi

adamjoneill edited a comment on issue #1325: presto - querying nested object in parquet file created by hudi URL: https://github.com/apache/incubator-hudi/issues/1325#issuecomment-585338502 i've managed to narrow down the issue to the data that is coming off the kinesis stream.

[GitHub] [incubator-hudi] adamjoneill edited a comment on issue #1325: presto - querying nested object in parquet file created by hudi

adamjoneill edited a comment on issue #1325: presto - querying nested object in parquet file created by hudi URL: https://github.com/apache/incubator-hudi/issues/1325#issuecomment-585338502 i've managed to narrow down the issue to the data that is coming off the kinesis stream.

[GitHub] [incubator-hudi] adamjoneill commented on issue #1325: presto - querying nested object in parquet file created by hudi

adamjoneill commented on issue #1325: presto - querying nested object in parquet file created by hudi URL: https://github.com/apache/incubator-hudi/issues/1325#issuecomment-585338502 i've managed to narrow down the issue to the data that is coming off the kinesis stream. when i

[GitHub] [incubator-hudi] leesf merged pull request #1317: [HUDI-605] Avoid calculating the size of schema redundantly

leesf merged pull request #1317: [HUDI-605] Avoid calculating the size of schema redundantly URL: https://github.com/apache/incubator-hudi/pull/1317 This is an automated message from the Apache Git Service. To respond to

[incubator-hudi] branch master updated (e5a69ed -> d2c872e)

This is an automated email from the ASF dual-hosted git repository. leesf pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git. from e5a69ed [HUDI-478] Fix too many files with unapproved license when execute build_local_docker_images

[GitHub] [incubator-hudi] lamber-ken commented on issue #1325: presto - querying nested object in parquet file created by hudi

lamber-ken commented on issue #1325: presto - querying nested object in parquet file created by hudi URL: https://github.com/apache/incubator-hudi/issues/1325#issuecomment-585147638 I guess you used `release-0.5.1` or master branch. `release-0.5.0` use

[GitHub] [incubator-hudi] adamjoneill edited a comment on issue #1325: presto - querying nested object in parquet file created by hudi

adamjoneill edited a comment on issue #1325: presto - querying nested object in parquet file created by hudi URL: https://github.com/apache/incubator-hudi/issues/1325#issuecomment-585128793 thanks @lamber-ken I've updated my deploy step to remove all references to

[GitHub] [incubator-hudi] adamjoneill commented on issue #1325: presto - querying nested object in parquet file created by hudi

adamjoneill commented on issue #1325: presto - querying nested object in parquet file created by hudi URL: https://github.com/apache/incubator-hudi/issues/1325#issuecomment-585128793 thanks @lamber-ken I've updated my deploy step to remove all references to

[GitHub] [incubator-hudi] OpenOpened commented on a change in pull request #1200: [HUDI-514] A schema provider to get metadata through Jdbc

OpenOpened commented on a change in pull request #1200: [HUDI-514] A schema provider to get metadata through Jdbc URL: https://github.com/apache/incubator-hudi/pull/1200#discussion_r378134572 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java