[jira] [Created] (HUDI-1393) Add compaction action in archive command
hong dongdong created HUDI-1393: --- Summary: Add compaction action in archive command Key: HUDI-1393 URL: https://issues.apache.org/jira/browse/HUDI-1393 Project: Apache Hudi Issue Type: Bug Components: CLI Reporter: hong dongdong Assignee: hong dongdong show archived commits command cannot recognize compaction action, add the case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-487) Unit tests for hudi-cli
[ https://issues.apache.org/jira/browse/HUDI-487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187369#comment-17187369 ] hong dongdong commented on HUDI-487: [~shivnarayan]: yes, recently, I have been entangled in work. Compaction will be done in the next two days. > Unit tests for hudi-cli > --- > > Key: HUDI-487 > URL: https://issues.apache.org/jira/browse/HUDI-487 > Project: Apache Hudi > Issue Type: Bug > Components: CLI, Testing >Reporter: Vinoth Chandar >Assignee: hong dongdong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1176) Support log4j2 config
hong dongdong created HUDI-1176: --- Summary: Support log4j2 config Key: HUDI-1176 URL: https://issues.apache.org/jira/browse/HUDI-1176 Project: Apache Hudi Issue Type: Bug Components: Testing Reporter: hong dongdong Assignee: hong dongdong Now in some modules(like cli, client) use log4j2, and it cannot correct load config file (ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1085) Support specifies archived path in HoodieSparkSqlWriter
[ https://issues.apache.org/jira/browse/HUDI-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated HUDI-1085: Description: The default archived path is '' before. Now, we can specifies archived path by overwrite conf "hoodie.archivelog.folder"(HoodieTableConfig.HOODIE_ARCHIVELOG_FOLDER_PROP_NAME). > Support specifies archived path in HoodieSparkSqlWriter > --- > > Key: HUDI-1085 > URL: https://issues.apache.org/jira/browse/HUDI-1085 > Project: Apache Hudi > Issue Type: Task > Components: Docs, Release & Administrative >Reporter: hong dongdong >Assignee: hong dongdong >Priority: Major > > The default archived path is '' before. Now, we can specifies archived path > by overwrite conf > "hoodie.archivelog.folder"(HoodieTableConfig.HOODIE_ARCHIVELOG_FOLDER_PROP_NAME). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1085) Support specifies archived path in HoodieSparkSqlWriter
[ https://issues.apache.org/jira/browse/HUDI-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated HUDI-1085: Summary: Support specifies archived path in HoodieSparkSqlWriter (was: Doc for archived commits command code cleanup) > Support specifies archived path in HoodieSparkSqlWriter > --- > > Key: HUDI-1085 > URL: https://issues.apache.org/jira/browse/HUDI-1085 > Project: Apache Hudi > Issue Type: Task > Components: Docs, Release & Administrative >Reporter: hong dongdong >Assignee: hong dongdong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-544) Archived commits command code cleanup
[ https://issues.apache.org/jira/browse/HUDI-544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated HUDI-544: --- Summary: Archived commits command code cleanup (was: Adjust the read and write path of archive) > Archived commits command code cleanup > - > > Key: HUDI-544 > URL: https://issues.apache.org/jira/browse/HUDI-544 > Project: Apache Hudi > Issue Type: Bug > Components: CLI, Writer Core >Reporter: hong dongdong >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Now, archive path have two different default value: "archived" and "". It > cause a bug. > [#HUDI-540] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1085) Doc for archived commits command code cleanup
[ https://issues.apache.org/jira/browse/HUDI-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated HUDI-1085: Component/s: Release & Administrative > Doc for archived commits command code cleanup > - > > Key: HUDI-1085 > URL: https://issues.apache.org/jira/browse/HUDI-1085 > Project: Apache Hudi > Issue Type: Task > Components: Docs, Release & Administrative >Reporter: hong dongdong >Assignee: hong dongdong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1085) Doc for archived commits command code cleanup
hong dongdong created HUDI-1085: --- Summary: Doc for archived commits command code cleanup Key: HUDI-1085 URL: https://issues.apache.org/jira/browse/HUDI-1085 Project: Apache Hudi Issue Type: Task Components: Docs Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1033) Remove redundant CLI tests
[ https://issues.apache.org/jira/browse/HUDI-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17141836#comment-17141836 ] hong dongdong commented on HUDI-1033: - [~vbalaji]: Somethings that I had not explained clearly. There is no duplicated testing, TestXXX for unit test and ITTestXXX for integration testing, they test different command. For example, CleansCommand TestCleansCommand test for commands: 'cleans show' 'clean showpartitions'. ITTestCleansCommand test for command: 'cleans run'. Why need integration testing? Some commands(like cleans run) run on spark, and SparkLauncher needs to load the jars(like hudi-common/hudi-client) under lib(generated during the package phase) when init (not means test for loading the jars). And, as you think, all tests actually perform live actions, not use mocking. > Remove redundant CLI tests > --- > > Key: HUDI-1033 > URL: https://issues.apache.org/jira/browse/HUDI-1033 > Project: Apache Hudi > Issue Type: Task > Components: Testing >Reporter: Balaji Varadarajan >Assignee: vinoyang >Priority: Major > Fix For: 0.6.0 > > > There are some tests like ITTestRepairsCommand vs TestRepairsCommand, > ITTestCleanerCommand vs TestCleanerCommand. Please consolidate if they are > redundant. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1033) Remove redundant CLI tests
[
https://issues.apache.org/jira/browse/HUDI-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17141624#comment-17141624
]
hong dongdong commented on HUDI-1033:
-
Hi [~vbalaji] : as [~yanghua] said there is difference between them. {{ITXXX}}
test cases need run on spark which need load the jars generated during the
package phase. I'm trying to move {{ITXXX to hudi-integ-test module.}}
> Remove redundant CLI tests
> ---
>
> Key: HUDI-1033
> URL: https://issues.apache.org/jira/browse/HUDI-1033
> Project: Apache Hudi
> Issue Type: Task
> Components: Testing
>Reporter: Balaji Varadarajan
>Assignee: vinoyang
>Priority: Major
> Fix For: 0.6.0
>
>
> There are some tests like ITTestRepairsCommand vs TestRepairsCommand,
> ITTestCleanerCommand vs TestCleanerCommand. Please consolidate if they are
> redundant.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HUDI-1019) Clean refresh command in CLI
hong dongdong created HUDI-1019: --- Summary: Clean refresh command in CLI Key: HUDI-1019 URL: https://issues.apache.org/jira/browse/HUDI-1019 Project: Apache Hudi Issue Type: Improvement Components: CLI Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-989) Support long options for prepare_integration_suite
hong dongdong created HUDI-989: -- Summary: Support long options for prepare_integration_suite Key: HUDI-989 URL: https://issues.apache.org/jira/browse/HUDI-989 Project: Apache Hudi Issue Type: Sub-task Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-978) Specify version information for each component separately
[ https://issues.apache.org/jira/browse/HUDI-978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated HUDI-978: --- Fix Version/s: 0.6.0 > Specify version information for each component separately > -- > > Key: HUDI-978 > URL: https://issues.apache.org/jira/browse/HUDI-978 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: hong dongdong >Assignee: hong dongdong >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-978) Specify version information for each component separately
hong dongdong created HUDI-978: -- Summary: Specify version information for each component separately Key: HUDI-978 URL: https://issues.apache.org/jira/browse/HUDI-978 Project: Apache Hudi Issue Type: Sub-task Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-593) Use scala_version placeholder to replace scala version hard code e.g. 2.11
[
https://issues.apache.org/jira/browse/HUDI-593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hong dongdong reassigned HUDI-593:
--
Assignee: hong dongdong
> Use scala_version placeholder to replace scala version hard code e.g. 2.11
> --
>
> Key: HUDI-593
> URL: https://issues.apache.org/jira/browse/HUDI-593
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Testing
>Reporter: vinoyang
>Assignee: hong dongdong
>Priority: Major
>
> After releasing Hudi v0.5.1, now Hudi supports scala 2.11/2.12. While the
> test suite still exists some hard code(2.11) of the Scala version. IMO, we
> can use {{${scala_version}}} to replace them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-859) Improve documentation around key generators
[ https://issues.apache.org/jira/browse/HUDI-859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17115315#comment-17115315 ] hong dongdong commented on HUDI-859: [~Pratyaksh] I have been trapped by other things these past two days, will assign this back to you. > Improve documentation around key generators > --- > > Key: HUDI-859 > URL: https://issues.apache.org/jira/browse/HUDI-859 > Project: Apache Hudi > Issue Type: Improvement > Components: Docs >Reporter: Pratyaksh Sharma >Assignee: Pratyaksh Sharma >Priority: Major > Labels: bug-bash-0.6.0 > Fix For: 0.6.0 > > > Proper documentation is required to help users understand what all key > generators are currently supported, how to use them etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-859) Improve documentation around key generators
[ https://issues.apache.org/jira/browse/HUDI-859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong reassigned HUDI-859: -- Assignee: Pratyaksh Sharma (was: hong dongdong) > Improve documentation around key generators > --- > > Key: HUDI-859 > URL: https://issues.apache.org/jira/browse/HUDI-859 > Project: Apache Hudi > Issue Type: Improvement > Components: Docs >Reporter: Pratyaksh Sharma >Assignee: Pratyaksh Sharma >Priority: Major > Labels: bug-bash-0.6.0 > Fix For: 0.6.0 > > > Proper documentation is required to help users understand what all key > generators are currently supported, how to use them etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-919) Run hudi-cli ITTest in docker.
hong dongdong created HUDI-919: -- Summary: Run hudi-cli ITTest in docker. Key: HUDI-919 URL: https://issues.apache.org/jira/browse/HUDI-919 Project: Apache Hudi (incubating) Issue Type: Improvement Components: CLI Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-861) Add Github and Twitter Widget on Hudi's official website
[ https://issues.apache.org/jira/browse/HUDI-861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated HUDI-861: --- Status: In Progress (was: Open) > Add Github and Twitter Widget on Hudi's official website > > > Key: HUDI-861 > URL: https://issues.apache.org/jira/browse/HUDI-861 > Project: Apache Hudi (incubating) > Issue Type: Improvement >Reporter: vinoyang >Assignee: hong dongdong >Priority: Major > > In order to further strengthen the influence of the Hudi community. I suggest > that we can embed Github and Twitter widgets on Hudi's official website as > Apahce ignite does. [https://ignite.apache.org/] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-861) Add Github and Twitter Widget on Hudi's official website
[ https://issues.apache.org/jira/browse/HUDI-861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated HUDI-861: --- Status: Open (was: New) > Add Github and Twitter Widget on Hudi's official website > > > Key: HUDI-861 > URL: https://issues.apache.org/jira/browse/HUDI-861 > Project: Apache Hudi (incubating) > Issue Type: Improvement >Reporter: vinoyang >Assignee: hong dongdong >Priority: Major > > In order to further strengthen the influence of the Hudi community. I suggest > that we can embed Github and Twitter widgets on Hudi's official website as > Apahce ignite does. [https://ignite.apache.org/] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-706) Add unit test for SavepointsCommand
[ https://issues.apache.org/jira/browse/HUDI-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated HUDI-706: --- Status: Open (was: New) > Add unit test for SavepointsCommand > --- > > Key: HUDI-706 > URL: https://issues.apache.org/jira/browse/HUDI-706 > Project: Apache Hudi (incubating) > Issue Type: Sub-task > Components: CLI, Testing >Reporter: hong dongdong >Assignee: hong dongdong >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-706) Add unit test for SavepointsCommand
[ https://issues.apache.org/jira/browse/HUDI-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong closed HUDI-706. -- Fix Version/s: 0.6.0 Resolution: Fixed > Add unit test for SavepointsCommand > --- > > Key: HUDI-706 > URL: https://issues.apache.org/jira/browse/HUDI-706 > Project: Apache Hudi (incubating) > Issue Type: Sub-task > Components: CLI, Testing >Reporter: hong dongdong >Assignee: hong dongdong >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-859) Improve documentation around key generators
[ https://issues.apache.org/jira/browse/HUDI-859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109353#comment-17109353 ] hong dongdong commented on HUDI-859: [~shivnarayan]: I'll discuss with [~Pratyaksh] for this. > Improve documentation around key generators > --- > > Key: HUDI-859 > URL: https://issues.apache.org/jira/browse/HUDI-859 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Docs >Reporter: Pratyaksh Sharma >Assignee: hong dongdong >Priority: Major > Labels: bug-bash-0.6.0 > Fix For: 0.6.0 > > > Proper documentation is required to help users understand what all key > generators are currently supported, how to use them etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-890) Prepare for 0.5.3 patch release
[ https://issues.apache.org/jira/browse/HUDI-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106804#comment-17106804 ] hong dongdong commented on HUDI-890: Hi, [~bhavanisudha] take a look when free https://jira.apache.org/jira/browse/HUDI-789 > Prepare for 0.5.3 patch release > --- > > Key: HUDI-890 > URL: https://issues.apache.org/jira/browse/HUDI-890 > Project: Apache Hudi (incubating) > Issue Type: Task >Reporter: Bhavani Sudha >Assignee: Bhavani Sudha >Priority: Major > Fix For: 0.5.3 > > > The following commits are included in this release. > * #1372 [HUDI-652] Decouple HoodieReadClient and AbstractHoodieClient to > break the inheritance chain > * #1388 [HUDI-681] Remove embeddedTimelineService from HoodieReadClient > * #1350 [HUDI-629]: Replace Guava's Hashing with an equivalent in > NumericUtils.java > * #1505 [HUDI - 738] Add validation to DeltaStreamer to fail fast when > filterDupes is enabled on UPSERT mode. > * #1517 [HUDI-799] Use appropriate FS when loading configs > * #1406 [HUDI-713] Fix conversion of Spark array of struct type to Avro > schema > * #1394 [HUDI-656][Performance] Return a dummy Spark relation after writing > the DataFrame > * #1576 [HUDI-850] Avoid unnecessary listings in incremental cleaning mode > * #1421 [HUDI-724] Parallelize getSmallFiles for partitions > * #1330 [HUDI-607] Fix to allow creation/syncing of Hive tables partitioned > by Date type columns > * #1413 Add constructor to HoodieROTablePathFilter > * #1415 [HUDI-539] Make ROPathFilter conf member serializable > * #1578 Add changes for presto mor queries > * #1506 [HUDI-782] Add support of Aliyun object storage service. > * #1432 [HUDI-716] Exception: Not an Avro data file when running > HoodieCleanClient.runClean > * #1422 [HUDI-400] Check upgrade from old plan to new plan for compaction > * #1448 [MINOR] Update DOAP with 0.5.2 Release > * #1466 [HUDI-742] Fix Java Math Exception > * #1416 [HUDI-717] Fixed usage of HiveDriver for DDL statements. > * #1427 [HUDI-727]: Copy default values of fields if not present when > rewriting incoming record with new schema > * #1515 [HUDI-795] Handle auto-deleted empty aux folder > * #1547 [MINOR]: Fix cli docs for DeltaStreamer > * #1580 [HUDI-852] adding check for table name for Append Save mode > * #1537 [MINOR] fixed building IndexFileFilter with a wrong condition in > HoodieGlobalBloomIndex class > * #1434 [HUDI-616] Fixed parquet files getting created on local FS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-840) Clean blank file created by HoodieLogFormatWriter
[ https://issues.apache.org/jira/browse/HUDI-840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated HUDI-840: --- Description: When roll over is true, HoodieLogFormatWriter will create next version log file. But it always left a blank file when close. (was: When roll over is true, ) > Clean blank file created by HoodieLogFormatWriter > - > > Key: HUDI-840 > URL: https://issues.apache.org/jira/browse/HUDI-840 > Project: Apache Hudi (incubating) > Issue Type: Bug > Components: Common Core >Reporter: hong dongdong >Assignee: hong dongdong >Priority: Major > > When roll over is true, HoodieLogFormatWriter will create next version log > file. But it always left a blank file when close. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-840) Clean blank file created by HoodieLogFormatWriter
[ https://issues.apache.org/jira/browse/HUDI-840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated HUDI-840: --- Description: When roll over is true, > Clean blank file created by HoodieLogFormatWriter > - > > Key: HUDI-840 > URL: https://issues.apache.org/jira/browse/HUDI-840 > Project: Apache Hudi (incubating) > Issue Type: Bug > Components: Common Core >Reporter: hong dongdong >Assignee: hong dongdong >Priority: Major > > When roll over is true, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-840) Clean blank file created by HoodieLogFormatWriter
hong dongdong created HUDI-840: -- Summary: Clean blank file created by HoodieLogFormatWriter Key: HUDI-840 URL: https://issues.apache.org/jira/browse/HUDI-840 Project: Apache Hudi (incubating) Issue Type: Bug Components: Common Core Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-789) Adjust logic of upsert in HDFSParquetImporter
hong dongdong created HUDI-789: -- Summary: Adjust logic of upsert in HDFSParquetImporter Key: HUDI-789 URL: https://issues.apache.org/jira/browse/HUDI-789 Project: Apache Hudi (incubating) Issue Type: Bug Components: Utilities Reporter: hong dongdong Assignee: hong dongdong In HDFSParquetImporter, upsert is equivalent to insert (remove old metadata, then insert). But upsert means update and insert on old data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-740) Fix can not specify the sparkMaster and code clean for SparkUtil
[ https://issues.apache.org/jira/browse/HUDI-740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated HUDI-740: --- Summary: Fix can not specify the sparkMaster and code clean for SparkUtil (was: [HUDI-740]Fix can not specify the sparkMaster of clean and compact commands) > Fix can not specify the sparkMaster and code clean for SparkUtil > > > Key: HUDI-740 > URL: https://issues.apache.org/jira/browse/HUDI-740 > Project: Apache Hudi (incubating) > Issue Type: Bug > Components: CLI >Reporter: hong dongdong >Assignee: hong dongdong >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Now, We can specify the sparkMaster of cleans run command, but it's not work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-740) [HUDI-740]Fix can not specify the sparkMaster of clean and compact commands
[ https://issues.apache.org/jira/browse/HUDI-740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated HUDI-740: --- Summary: [HUDI-740]Fix can not specify the sparkMaster of clean and compact commands (was: Fix can not specify the sparkMaster of cleans run command) > [HUDI-740]Fix can not specify the sparkMaster of clean and compact commands > > > Key: HUDI-740 > URL: https://issues.apache.org/jira/browse/HUDI-740 > Project: Apache Hudi (incubating) > Issue Type: Bug > Components: CLI >Reporter: hong dongdong >Assignee: hong dongdong >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Now, We can specify the sparkMaster of cleans run command, but it's not work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-752) Make CompactionAdminClient spark-free
[ https://issues.apache.org/jira/browse/HUDI-752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated HUDI-752: --- Description: Now, we always pass jsc, there can only one sparkContext in JVM. So, we can store it in a Factory class, then we can get it everywhere. After that, we make many class spark-free > Make CompactionAdminClient spark-free > - > > Key: HUDI-752 > URL: https://issues.apache.org/jira/browse/HUDI-752 > Project: Apache Hudi (incubating) > Issue Type: Sub-task >Reporter: hong dongdong >Assignee: hong dongdong >Priority: Major > > Now, we always pass jsc, there can only one sparkContext in JVM. So, we can > store it in a Factory class, then we can get it everywhere. After that, we > make many class spark-free -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-752) Make CompactionAdminClient spark-free
hong dongdong created HUDI-752: -- Summary: Make CompactionAdminClient spark-free Key: HUDI-752 URL: https://issues.apache.org/jira/browse/HUDI-752 Project: Apache Hudi (incubating) Issue Type: Sub-task Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-750) Make AbstractHoodieClient spark-free
hong dongdong created HUDI-750: -- Summary: Make AbstractHoodieClient spark-free Key: HUDI-750 URL: https://issues.apache.org/jira/browse/HUDI-750 Project: Apache Hudi (incubating) Issue Type: Sub-task Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-677) Abstract/Refactor all transaction management logic into a set of classes from HoodieWriteClient
[ https://issues.apache.org/jira/browse/HUDI-677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17070217#comment-17070217 ] hong dongdong commented on HUDI-677: [~vinoth] Of course not. > Abstract/Refactor all transaction management logic into a set of classes from > HoodieWriteClient > --- > > Key: HUDI-677 > URL: https://issues.apache.org/jira/browse/HUDI-677 > Project: Apache Hudi (incubating) > Issue Type: Sub-task > Components: Code Cleanup >Reporter: Vinoth Chandar >Assignee: Vinoth Chandar >Priority: Major > Fix For: 0.6.0 > > > Over time a lot of the core transaction management code has been split > across various files in hudi-client.. We want to clean this up and present a > nice interface.. > Some notes and thoughts and suggestions.. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-740) Fix can not specify the sparkMaster of cleans run command
hong dongdong created HUDI-740: -- Summary: Fix can not specify the sparkMaster of cleans run command Key: HUDI-740 URL: https://issues.apache.org/jira/browse/HUDI-740 Project: Apache Hudi (incubating) Issue Type: Bug Components: CLI Reporter: hong dongdong Assignee: hong dongdong Now, We can specify the sparkMaster of cleans run command, but it's not work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-677) Abstract/Refactor all transaction management logic into a set of classes from HoodieWriteClient
[ https://issues.apache.org/jira/browse/HUDI-677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17067538#comment-17067538 ] hong dongdong commented on HUDI-677: [~vinoth] As summary said, all transaction need move out of HoodieWriteClient. A brief description of what I'm going to do: [https://docs.google.com/document/d/1-hXvcpQz42zORDlrDhJ9xEf33cmynw9hwtIb2K5DeL0/edit?usp=sharing] . Do you have any other suggestions or reminders > Abstract/Refactor all transaction management logic into a set of classes from > HoodieWriteClient > --- > > Key: HUDI-677 > URL: https://issues.apache.org/jira/browse/HUDI-677 > Project: Apache Hudi (incubating) > Issue Type: Sub-task > Components: Code Cleanup >Reporter: Vinoth Chandar >Assignee: hong dongdong >Priority: Major > Fix For: 0.6.0 > > > Over time a lot of the core transaction management code has been split > across various files in hudi-client.. We want to clean this up and present a > nice interface.. > Some notes and thoughts and suggestions.. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-715) Fix duplicate name in TableCommand
hong dongdong created HUDI-715: -- Summary: Fix duplicate name in TableCommand Key: HUDI-715 URL: https://issues.apache.org/jira/browse/HUDI-715 Project: Apache Hudi (incubating) Issue Type: Bug Components: CLI Reporter: hong dongdong Assignee: hong dongdong connect command has duplicate key name maxCheckIntervalMs, fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-692) Add delete savepoint for cli
[ https://issues.apache.org/jira/browse/HUDI-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong closed HUDI-692. -- Fix Version/s: 0.6.0 Resolution: Implemented > Add delete savepoint for cli > > > Key: HUDI-692 > URL: https://issues.apache.org/jira/browse/HUDI-692 > Project: Apache Hudi (incubating) > Issue Type: New Feature > Components: CLI >Reporter: hong dongdong >Assignee: hong dongdong >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Now, deleteSavepoint already provided in HoodieWriteClient, but not provide > to user, add it in CLI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-692) Add delete savepoint for cli
[ https://issues.apache.org/jira/browse/HUDI-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated HUDI-692: --- Status: Open (was: New) > Add delete savepoint for cli > > > Key: HUDI-692 > URL: https://issues.apache.org/jira/browse/HUDI-692 > Project: Apache Hudi (incubating) > Issue Type: New Feature > Components: CLI >Reporter: hong dongdong >Assignee: hong dongdong >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Now, deleteSavepoint already provided in HoodieWriteClient, but not provide > to user, add it in CLI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-709) Add unit test for UtilsCommand
hong dongdong created HUDI-709: -- Summary: Add unit test for UtilsCommand Key: HUDI-709 URL: https://issues.apache.org/jira/browse/HUDI-709 Project: Apache Hudi (incubating) Issue Type: Sub-task Components: CLI, Testing Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-708) Add unit test for TempViewCommand
hong dongdong created HUDI-708: -- Summary: Add unit test for TempViewCommand Key: HUDI-708 URL: https://issues.apache.org/jira/browse/HUDI-708 Project: Apache Hudi (incubating) Issue Type: Sub-task Components: CLI, Testing Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-707) Add unit test for StatsCommand
hong dongdong created HUDI-707: -- Summary: Add unit test for StatsCommand Key: HUDI-707 URL: https://issues.apache.org/jira/browse/HUDI-707 Project: Apache Hudi (incubating) Issue Type: Sub-task Components: CLI, Testing Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-706) Add unit test for SavepointsCommand
hong dongdong created HUDI-706: -- Summary: Add unit test for SavepointsCommand Key: HUDI-706 URL: https://issues.apache.org/jira/browse/HUDI-706 Project: Apache Hudi (incubating) Issue Type: Sub-task Components: CLI, Testing Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-705) Add unit test for RollbacksCommand
hong dongdong created HUDI-705: -- Summary: Add unit test for RollbacksCommand Key: HUDI-705 URL: https://issues.apache.org/jira/browse/HUDI-705 Project: Apache Hudi (incubating) Issue Type: Sub-task Components: CLI, Testing Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-703) Add unit test for HoodieSyncCommand
hong dongdong created HUDI-703: -- Summary: Add unit test for HoodieSyncCommand Key: HUDI-703 URL: https://issues.apache.org/jira/browse/HUDI-703 Project: Apache Hudi (incubating) Issue Type: Sub-task Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-704) Add unit test for RepairsCommand
hong dongdong created HUDI-704: -- Summary: Add unit test for RepairsCommand Key: HUDI-704 URL: https://issues.apache.org/jira/browse/HUDI-704 Project: Apache Hudi (incubating) Issue Type: Sub-task Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-702) Add unit test for HoodieLogFileCommand
hong dongdong created HUDI-702: -- Summary: Add unit test for HoodieLogFileCommand Key: HUDI-702 URL: https://issues.apache.org/jira/browse/HUDI-702 Project: Apache Hudi (incubating) Issue Type: Sub-task Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-700) Add unit test for FileSystemViewCommand
hong dongdong created HUDI-700: -- Summary: Add unit test for FileSystemViewCommand Key: HUDI-700 URL: https://issues.apache.org/jira/browse/HUDI-700 Project: Apache Hudi (incubating) Issue Type: Sub-task Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-701) Add unit test for HDFSParquetImportCommand
hong dongdong created HUDI-701: -- Summary: Add unit test for HDFSParquetImportCommand Key: HUDI-701 URL: https://issues.apache.org/jira/browse/HUDI-701 Project: Apache Hudi (incubating) Issue Type: Sub-task Reporter: hong dongdong Assignee: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-699) Add unit test for CompactionCommand
hong dongdong created HUDI-699: -- Summary: Add unit test for CompactionCommand Key: HUDI-699 URL: https://issues.apache.org/jira/browse/HUDI-699 Project: Apache Hudi (incubating) Issue Type: Sub-task Components: CLI, Testing Reporter: hong dongdong Assignee: hong dongdong Add unit test for CompactionCommand in hudi-cli module -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-698) Add unit test for CleansCommand
hong dongdong created HUDI-698: -- Summary: Add unit test for CleansCommand Key: HUDI-698 URL: https://issues.apache.org/jira/browse/HUDI-698 Project: Apache Hudi (incubating) Issue Type: Sub-task Components: CLI, Testing Reporter: hong dongdong Assignee: hong dongdong Add unit test for CleansCommand in hudi-cli module -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-697) Add unit test for ArchivedCommitsCommand
hong dongdong created HUDI-697: -- Summary: Add unit test for ArchivedCommitsCommand Key: HUDI-697 URL: https://issues.apache.org/jira/browse/HUDI-697 Project: Apache Hudi (incubating) Issue Type: Sub-task Reporter: hong dongdong Assignee: hong dongdong Add unit test for ArchivedCommitsCommand in hudi-cli module -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-696) Add unit test for CommitsCommand
hong dongdong created HUDI-696: -- Summary: Add unit test for CommitsCommand Key: HUDI-696 URL: https://issues.apache.org/jira/browse/HUDI-696 Project: Apache Hudi (incubating) Issue Type: Sub-task Components: CLI, Testing Reporter: hong dongdong Assignee: hong dongdong Add unit test for CommitsCommand in hudi-cli module -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-695) Add unit test for TableCommand
[ https://issues.apache.org/jira/browse/HUDI-695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated HUDI-695: --- Parent: HUDI-487 Issue Type: Sub-task (was: Bug) > Add unit test for TableCommand > -- > > Key: HUDI-695 > URL: https://issues.apache.org/jira/browse/HUDI-695 > Project: Apache Hudi (incubating) > Issue Type: Sub-task > Components: CLI, Testing >Reporter: hong dongdong >Assignee: hong dongdong >Priority: Major > > Add unit test for TableCommand in hudi-cli -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-695) Add unit test for TableCommand
hong dongdong created HUDI-695: -- Summary: Add unit test for TableCommand Key: HUDI-695 URL: https://issues.apache.org/jira/browse/HUDI-695 Project: Apache Hudi (incubating) Issue Type: Bug Components: CLI, Testing Reporter: hong dongdong Assignee: hong dongdong Add unit test for TableCommand in hudi-cli -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-694) Add unit test for SparkEnvCommand
hong dongdong created HUDI-694: -- Summary: Add unit test for SparkEnvCommand Key: HUDI-694 URL: https://issues.apache.org/jira/browse/HUDI-694 Project: Apache Hudi (incubating) Issue Type: Sub-task Components: CLI, Testing Reporter: hong dongdong Assignee: hong dongdong Add unit test for SparkEnvCommand in hudi-cli -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-693) Add unit test for hudi-cli module
[ https://issues.apache.org/jira/browse/HUDI-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong reassigned HUDI-693: -- Assignee: hong dongdong > Add unit test for hudi-cli module > - > > Key: HUDI-693 > URL: https://issues.apache.org/jira/browse/HUDI-693 > Project: Apache Hudi (incubating) > Issue Type: New Feature > Components: CLI, Testing >Reporter: hong dongdong >Assignee: hong dongdong >Priority: Major > > There is no unit tests for this module overall, need to add it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-487) Unit tests for hudi-cli
[ https://issues.apache.org/jira/browse/HUDI-487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong reassigned HUDI-487: -- Assignee: hong dongdong (was: Forward Xu) > Unit tests for hudi-cli > --- > > Key: HUDI-487 > URL: https://issues.apache.org/jira/browse/HUDI-487 > Project: Apache Hudi (incubating) > Issue Type: Bug > Components: CLI, Testing >Reporter: Vinoth Chandar >Assignee: hong dongdong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-487) Unit tests for hudi-cli
[ https://issues.apache.org/jira/browse/HUDI-487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17057670#comment-17057670 ] hong dongdong commented on HUDI-487: [~vinoth] I will do it. > Unit tests for hudi-cli > --- > > Key: HUDI-487 > URL: https://issues.apache.org/jira/browse/HUDI-487 > Project: Apache Hudi (incubating) > Issue Type: Bug > Components: CLI, Testing >Reporter: Vinoth Chandar >Assignee: Forward Xu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-693) Add unit test for hudi-cli module
hong dongdong created HUDI-693: -- Summary: Add unit test for hudi-cli module Key: HUDI-693 URL: https://issues.apache.org/jira/browse/HUDI-693 Project: Apache Hudi (incubating) Issue Type: New Feature Components: CLI, Testing Reporter: hong dongdong There is no unit tests for this module overall, need to add it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-692) Add delete savepoint for cli
[ https://issues.apache.org/jira/browse/HUDI-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong reassigned HUDI-692: -- Assignee: hong dongdong > Add delete savepoint for cli > > > Key: HUDI-692 > URL: https://issues.apache.org/jira/browse/HUDI-692 > Project: Apache Hudi (incubating) > Issue Type: New Feature > Components: CLI >Reporter: hong dongdong >Assignee: hong dongdong >Priority: Major > > Now, deleteSavepoint already provided in HoodieWriteClient, but not provide > to user, add it in CLI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-692) Add delete savepoint for cli
hong dongdong created HUDI-692: -- Summary: Add delete savepoint for cli Key: HUDI-692 URL: https://issues.apache.org/jira/browse/HUDI-692 Project: Apache Hudi (incubating) Issue Type: New Feature Components: CLI Reporter: hong dongdong Now, deleteSavepoint already provided in HoodieWriteClient, but not provide to user, add it in CLI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-677) Abstract/Refactor all transaction management logic into a set of classes from HoodieWriteClient
[ https://issues.apache.org/jira/browse/HUDI-677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054391#comment-17054391 ] hong dongdong commented on HUDI-677: yep，I will try with this issue. > Abstract/Refactor all transaction management logic into a set of classes from > HoodieWriteClient > --- > > Key: HUDI-677 > URL: https://issues.apache.org/jira/browse/HUDI-677 > Project: Apache Hudi (incubating) > Issue Type: Sub-task > Components: Code Cleanup >Reporter: Vinoth Chandar >Assignee: Vinoth Chandar >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-544) Adjust the read and write path of archive
[ https://issues.apache.org/jira/browse/HUDI-544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated HUDI-544: --- Description: Now, archive path have two different default value: "archived" and "". It cause a bug. [#HUDI-540] was:Now, archive path have two different default value: "archived" and "". It cause a bug. > Adjust the read and write path of archive > - > > Key: HUDI-544 > URL: https://issues.apache.org/jira/browse/HUDI-544 > Project: Apache Hudi (incubating) > Issue Type: Bug > Components: CLI, Writer Core >Reporter: hong dongdong >Priority: Major > > Now, archive path have two different default value: "archived" and "". It > cause a bug. > [#HUDI-540] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-544) Adjust the read and write path of archive
hong dongdong created HUDI-544: -- Summary: Adjust the read and write path of archive Key: HUDI-544 URL: https://issues.apache.org/jira/browse/HUDI-544 Project: Apache Hudi (incubating) Issue Type: Bug Components: CLI, Writer Core Reporter: hong dongdong Now, archive path have two different default value: "archived" and "". It cause a bug. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-538) Restructuring hudi client module for multi engine support
[ https://issues.apache.org/jira/browse/HUDI-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017648#comment-17017648 ] hong dongdong commented on HUDI-538: [~yanghua] I am interesting in this work, please @ me if any need. And I had want to do this before. > Restructuring hudi client module for multi engine support > - > > Key: HUDI-538 > URL: https://issues.apache.org/jira/browse/HUDI-538 > Project: Apache Hudi (incubating) > Issue Type: Wish >Reporter: vinoyang >Priority: Major > > Hudi is currently tightly coupled with the Spark framework. It caused the > integration with other computing engine more difficult. We plan to decouple > it with Spark. This umbrella issue used to track this work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-539) No FileSystem for scheme: abfss
[
https://issues.apache.org/jira/browse/HUDI-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017644#comment-17017644
]
hong dongdong commented on HUDI-539:
[~ssomuah] As far as I know, ABFS was support after hadoop-3.2.0, may be you
need a higher version of hadoop.
> No FileSystem for scheme: abfss
> ---
>
> Key: HUDI-539
> URL: https://issues.apache.org/jira/browse/HUDI-539
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: Common Core
>Affects Versions: 0.5.1
> Environment: Spark version : 2.4.4
> Hadoop version : 2.7.3
> Databricks Runtime: 6.1
>Reporter: Sam Somuah
>Priority: Major
>
> Hi,
> I'm trying to use hudi to write to one of the Azure storage container file
> systems, ADLS Gen 2 (abfs://). ABFS:// is one of the whitelisted file
> schemes. The issue I'm facing is that in {{HoodieROTablePathFilter}} it tries
> to get a file path passing in a blank hadoop configuration. This manifests as
> {{java.io.IOException: No FileSystem for scheme: abfss}} because it doesn't
> have any of the configuration in the environment.
> The problematic line is
> [https://github.com/apache/incubator-hudi/blob/2bb0c21a3dd29687e49d362ed34f050380ff47ae/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieROTablePathFilter.java#L96]
>
> Stacktrace
> java.io.IOException: No FileSystem for scheme: abfss
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
> at
> org.apache.hudi.hadoop.HoodieROTablePathFilter.accept(HoodieROTablePathFilter.java:96)
> at
> org.apache.spark.sql.execution.datasources.InMemoryFileIndex$$anonfun$16.apply(InMemoryFileIndex.scala:349)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HUDI-540) Incorrect archive directory path in show archived commits cli
[ https://issues.apache.org/jira/browse/HUDI-540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong reassigned HUDI-540: -- Assignee: hong dongdong > Incorrect archive directory path in show archived commits cli > - > > Key: HUDI-540 > URL: https://issues.apache.org/jira/browse/HUDI-540 > Project: Apache Hudi (incubating) > Issue Type: Bug > Components: CLI > Environment: EMR, S3 >Reporter: Venkatesh >Assignee: hong dongdong >Priority: Major > > Archive path is specified as - new Path(basePath + > "/.hoodie/.commits_.archive*"); , but should be -new Path(basePath + > "/.hoodie/archived/.commits_.archive*"); > We are using S3 to store hudi dataset if that matters. > > [https://github.com/apache/incubator-hudi/blob/a733f4ef723865738d8541282c0c7234d64668db/hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java#L143] > > [https://github.com/apache/incubator-hudi/blob/a733f4ef723865738d8541282c0c7234d64668db/hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java#L66] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-540) Incorrect archive directory path in show archived commits cli
[
https://issues.apache.org/jira/browse/HUDI-540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017639#comment-17017639
]
hong dongdong commented on HUDI-540:
[~venkee14] It‘s need
{code:java}
@CliOption(key = {"archiveFolderPattern"}, help = "Archive Folder",
unspecifiedDefaultValue = "") String folder
{code}
Command 'show archived commit stats' always work correctly with this cliOption.
HoodieSparkSqlWriter.scala set archiveLogfolder as 'archived' when create a new
table, otherwise the default is "".
I will try to address this later.
> Incorrect archive directory path in show archived commits cli
> -
>
> Key: HUDI-540
> URL: https://issues.apache.org/jira/browse/HUDI-540
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: CLI
> Environment: EMR, S3
>Reporter: Venkatesh
>Priority: Major
>
> Archive path is specified as - new Path(basePath +
> "/.hoodie/.commits_.archive*"); , but should be -new Path(basePath +
> "/.hoodie/archived/.commits_.archive*");
> We are using S3 to store hudi dataset if that matters.
>
> [https://github.com/apache/incubator-hudi/blob/a733f4ef723865738d8541282c0c7234d64668db/hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java#L143]
>
> [https://github.com/apache/incubator-hudi/blob/a733f4ef723865738d8541282c0c7234d64668db/hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java#L66]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HUDI-523) Upgrade Hudi to Spark DataSource V2
[ https://issues.apache.org/jira/browse/HUDI-523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated HUDI-523: --- Description: May be need spark3 (was: As spark upgrade to 2.4, we can upgrade to datasource api v2 now.) > Upgrade Hudi to Spark DataSource V2 > --- > > Key: HUDI-523 > URL: https://issues.apache.org/jira/browse/HUDI-523 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Writer Core >Reporter: hong dongdong >Priority: Major > > May be need spark3 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-523) Upgrade Hudi to Spark DataSource V2
[ https://issues.apache.org/jira/browse/HUDI-523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong reassigned HUDI-523: -- Assignee: (was: hong dongdong) > Upgrade Hudi to Spark DataSource V2 > --- > > Key: HUDI-523 > URL: https://issues.apache.org/jira/browse/HUDI-523 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Writer Core >Reporter: hong dongdong >Priority: Major > > As spark upgrade to 2.4, we can upgrade to datasource api v2 now. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-523) Upgrade Hudi to Spark DataSource V2
hong dongdong created HUDI-523: -- Summary: Upgrade Hudi to Spark DataSource V2 Key: HUDI-523 URL: https://issues.apache.org/jira/browse/HUDI-523 Project: Apache Hudi (incubating) Issue Type: Improvement Components: Writer Core Reporter: hong dongdong As spark upgrade to 2.4, we can upgrade to datasource api v2 now. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-523) Upgrade Hudi to Spark DataSource V2
[ https://issues.apache.org/jira/browse/HUDI-523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong reassigned HUDI-523: -- Assignee: hong dongdong > Upgrade Hudi to Spark DataSource V2 > --- > > Key: HUDI-523 > URL: https://issues.apache.org/jira/browse/HUDI-523 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Writer Core >Reporter: hong dongdong >Assignee: hong dongdong >Priority: Major > > As spark upgrade to 2.4, we can upgrade to datasource api v2 now. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-492) show env all CLI command can not work in hudi-cli
[
https://issues.apache.org/jira/browse/HUDI-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007211#comment-17007211
]
hong dongdong commented on HUDI-492:
[~yanghua] [~vinoth] [~lamber-ken] please see pr [
#1172|https://github.com/apache/incubator-hudi/pull/1172] .In latest code,
`show env all` is confused with `show env --conf` for spring-shell. We can
delete is or change to `show envs all`.
> show env all CLI command can not work in hudi-cli
> -
>
> Key: HUDI-492
> URL: https://issues.apache.org/jira/browse/HUDI-492
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: CLI
>Reporter: vinoyang
>Priority: Major
> Labels: pull-request-available
> Attachments: Screen Shot 2020-01-02 at 8.33.35 PM.png, hudi-11.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, {{show env all}} CLI command can not work in hudi-cli. Please see
> the picture attachment.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HUDI-492) show env all CLI command can not work in hudi-cli
[
https://issues.apache.org/jira/browse/HUDI-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007188#comment-17007188
]
hong dongdong edited comment on HUDI-492 at 1/3/20 2:59 AM:
[~yanghua] thanks, I found there are problem in `show env all` when I update to
the latest code, it should be due to other code upgrades.
was (Author: hongdongdong):
[~yanghua] thanks, I found there are problem in `show env all` when I update to
the latest code.
> show env all CLI command can not work in hudi-cli
> -
>
> Key: HUDI-492
> URL: https://issues.apache.org/jira/browse/HUDI-492
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: CLI
>Reporter: vinoyang
>Priority: Major
> Attachments: Screen Shot 2020-01-02 at 8.33.35 PM.png, hudi-11.png
>
>
> Currently, {{show env all}} CLI command can not work in hudi-cli. Please see
> the picture attachment.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-492) show env all CLI command can not work in hudi-cli
[
https://issues.apache.org/jira/browse/HUDI-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007188#comment-17007188
]
hong dongdong commented on HUDI-492:
[~yanghua] thanks, I found there are problem in `show env all` when I update to
the latest code.
> show env all CLI command can not work in hudi-cli
> -
>
> Key: HUDI-492
> URL: https://issues.apache.org/jira/browse/HUDI-492
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: CLI
>Reporter: vinoyang
>Priority: Major
> Attachments: Screen Shot 2020-01-02 at 8.33.35 PM.png, hudi-11.png
>
>
> Currently, {{show env all}} CLI command can not work in hudi-cli. Please see
> the picture attachment.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HUDI-492) show env all CLI command can not work in hudi-cli
[
https://issues.apache.org/jira/browse/HUDI-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007175#comment-17007175
]
hong dongdong edited comment on HUDI-492 at 1/3/20 2:27 AM:
!hudi-11.png!
[~yanghua] can you give a picture which with command `help show env all`. It is
working fine on my side, actually `show env` is equivalent to `show env all`
and is same code block. I wonder how this happened.
was (Author: hongdongdong):
[~yanghua] can you give a picture which with command `help show env all`. It is
working fine on my side, actually `show env` is equivalent to `show env all`
and is same code block. I wonder how this happened.
> show env all CLI command can not work in hudi-cli
> -
>
> Key: HUDI-492
> URL: https://issues.apache.org/jira/browse/HUDI-492
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: CLI
>Reporter: vinoyang
>Priority: Major
> Attachments: Screen Shot 2020-01-02 at 8.33.35 PM.png, hudi-11.png
>
>
> Currently, {{show env all}} CLI command can not work in hudi-cli. Please see
> the picture attachment.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HUDI-492) show env all CLI command can not work in hudi-cli
[
https://issues.apache.org/jira/browse/HUDI-492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hong dongdong updated HUDI-492:
---
Attachment: hudi-11.png
> show env all CLI command can not work in hudi-cli
> -
>
> Key: HUDI-492
> URL: https://issues.apache.org/jira/browse/HUDI-492
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: CLI
>Reporter: vinoyang
>Priority: Major
> Attachments: Screen Shot 2020-01-02 at 8.33.35 PM.png, hudi-11.png
>
>
> Currently, {{show env all}} CLI command can not work in hudi-cli. Please see
> the picture attachment.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-492) show env all CLI command can not work in hudi-cli
[
https://issues.apache.org/jira/browse/HUDI-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007175#comment-17007175
]
hong dongdong commented on HUDI-492:
[~yanghua] can you give a picture which with command `help show env all`. It is
working fine on my side, actually `show env` is equivalent to `show env all`
and is same code block. I wonder how this happened.
> show env all CLI command can not work in hudi-cli
> -
>
> Key: HUDI-492
> URL: https://issues.apache.org/jira/browse/HUDI-492
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: CLI
>Reporter: vinoyang
>Priority: Major
> Attachments: Screen Shot 2020-01-02 at 8.33.35 PM.png
>
>
> Currently, {{show env all}} CLI command can not work in hudi-cli. Please see
> the picture attachment.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HUDI-438) Merge duplicated code fragment in HoodieSparkSqlWriter.scala
hong dongdong created HUDI-438: -- Summary: Merge duplicated code fragment in HoodieSparkSqlWriter.scala Key: HUDI-438 URL: https://issues.apache.org/jira/browse/HUDI-438 Project: Apache Hudi (incubating) Issue Type: Improvement Components: code cleanup Reporter: hong dongdong As the summary says. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-332) Add operation type (insert/upsert/bulkinsert/delete) to HoodieCommitMetadata
[ https://issues.apache.org/jira/browse/HUDI-332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999705#comment-16999705 ] hong dongdong commented on HUDI-332: [~vinoth] Thanks, I will take care of your remind. > Add operation type (insert/upsert/bulkinsert/delete) to HoodieCommitMetadata > > > Key: HUDI-332 > URL: https://issues.apache.org/jira/browse/HUDI-332 > Project: Apache Hudi (incubating) > Issue Type: Improvement >Reporter: sivabalan narayanan >Assignee: hong dongdong >Priority: Major > Original Estimate: 1m > Remaining Estimate: 1m > > Would be nice to add info on operation type (insert/upsert/bulkinsert/delete) > to the HoodieCommitMetadata. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-332) Add operation type (insert/upsert/bulkinsert/delete) to HoodieCommitMetadata
[ https://issues.apache.org/jira/browse/HUDI-332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong reassigned HUDI-332: -- Assignee: hong dongdong > Add operation type (insert/upsert/bulkinsert/delete) to HoodieCommitMetadata > > > Key: HUDI-332 > URL: https://issues.apache.org/jira/browse/HUDI-332 > Project: Apache Hudi (incubating) > Issue Type: Improvement >Reporter: sivabalan narayanan >Assignee: hong dongdong >Priority: Major > Original Estimate: 1m > Remaining Estimate: 1m > > Would be nice to add info on operation type (insert/upsert/bulkinsert/delete) > to the HoodieCommitMetadata. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-332) Add operation type (insert/upsert/bulkinsert/delete) to HoodieCommitMetadata
[ https://issues.apache.org/jira/browse/HUDI-332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999686#comment-16999686 ] hong dongdong commented on HUDI-332: [~shivnarayan] [~vinoth] Is any one working on this? I have interest in this, may i work on this? > Add operation type (insert/upsert/bulkinsert/delete) to HoodieCommitMetadata > > > Key: HUDI-332 > URL: https://issues.apache.org/jira/browse/HUDI-332 > Project: Apache Hudi (incubating) > Issue Type: Improvement >Reporter: sivabalan narayanan >Priority: Major > Original Estimate: 1m > Remaining Estimate: 1m > > Would be nice to add info on operation type (insert/upsert/bulkinsert/delete) > to the HoodieCommitMetadata. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-416) Improve hint information for Cli
hong dongdong created HUDI-416:
--
Summary: Improve hint information for Cli
Key: HUDI-416
URL: https://issues.apache.org/jira/browse/HUDI-416
Project: Apache Hudi (incubating)
Issue Type: Improvement
Components: CLI
Reporter: hong dongdong
Right now, cli always give error information:
{code:java}
Command 'desc' was found but is not currently available (type 'help' then ENTER
to learn about this command)
{code}
but it is confused to user. We can give a hint clearly like:
{code:java}
Command failed java.lang.NullPointerException: There is no hudi dataset. Please
use connect command to set dataset first
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-331) Fix java docs for all public apis (HoodieWriteClient)
[ https://issues.apache.org/jira/browse/HUDI-331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997780#comment-16997780 ] hong dongdong commented on HUDI-331: I will working on it. [~xleesf] > Fix java docs for all public apis (HoodieWriteClient) > - > > Key: HUDI-331 > URL: https://issues.apache.org/jira/browse/HUDI-331 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Docs >Reporter: sivabalan narayanan >Priority: Major > Labels: newbie > Fix For: 0.5.1 > > Original Estimate: 24h > Remaining Estimate: 24h > > Some public apis in HoodieWriteClient need to be fixed with sufficient info. > Creating this ticket to get it fixed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-401) Remove unnecessary use of spark in savepoint timeline
hong dongdong created HUDI-401: -- Summary: Remove unnecessary use of spark in savepoint timeline Key: HUDI-401 URL: https://issues.apache.org/jira/browse/HUDI-401 Project: Apache Hudi (incubating) Issue Type: Improvement Components: CLI, Write Client Reporter: hong dongdong Currently, javasparkcontext was inited when savepoint create, but it is not necessary. Javasparkcontext's whole work is provide hadoopconfig, but need time and resources to init it. So we can use hadoop config instead of jsc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-398) Add set env for spark launcher
hong dongdong created HUDI-398: -- Summary: Add set env for spark launcher Key: HUDI-398 URL: https://issues.apache.org/jira/browse/HUDI-398 Project: Apache Hudi (incubating) Issue Type: New Feature Components: CLI Reporter: hong dongdong Attachments: image-2019-12-11-14-44-55-064.png, image-2019-12-11-14-45-27-764.png It always throw exception 'SPAR_HOEM not found' when SPARK_HOME is not set. So we need quit and set it. !image-2019-12-11-14-45-27-764.png! After add this function for cli, we can type SPARK_HOEM and other conf on hudi-CLI. !image-2019-12-11-14-44-55-064.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-374) Unable to generateUpdates in QuickstartUtils
hong dongdong created HUDI-374:
--
Summary: Unable to generateUpdates in QuickstartUtils
Key: HUDI-374
URL: https://issues.apache.org/jira/browse/HUDI-374
Project: Apache Hudi (incubating)
Issue Type: Bug
Components: Spark datasource
Reporter: hong dongdong
{code:java}
scala> convertToStringList(dataGen.generateInserts(1))
res0: java.util.List[String] = [{"ts": 0.0, "uuid":
"78956d3a-c13b-4871-8b14-596b2a7e11d9", "rider": "rider-213", "driver":
"driver-213", "begin_lat": 0.4726905879569653, "begin_lon":
0.46157858450465483, "end_lat": 0.754803407008858, "end_lon":
0.9671159942018241, "fare": 34.158284716382845, "partitionpath":
"americas/brazil/sao_paulo"}]
scala> convertToStringList(dataGen.generateUpdates(1))
java.lang.IllegalArgumentException: bound must be positive
at java.util.Random.nextInt(Random.java:388)
at
org.apache.hudi.QuickstartUtils$DataGenerator.generateUpdates(QuickstartUtils.java:163)
... 73 elided
{code}
When `numExistingKeys = 1`, `rand.nextInt(numExistingKeys - 1 )` is equivalent
to `rand.nextInt(0)` and bound of nextInt() must be positive. On the other
hand, the range of nextInt is [0, numExistingKeys),
rand.nextInt(numExistingKeys) here is right.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HUDI-362) Adds a check for the existence of field
[
https://issues.apache.org/jira/browse/HUDI-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hong dongdong updated HUDI-362:
---
Attachment: image-2019-11-25-15-33-21-610.png
> Adds a check for the existence of field
> ---
>
> Key: HUDI-362
> URL: https://issues.apache.org/jira/browse/HUDI-362
> Project: Apache Hudi (incubating)
> Issue Type: Improvement
> Components: CLI
>Reporter: hong dongdong
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2019-11-25-15-32-14-057.png,
> image-2019-11-25-15-33-21-610.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Use command
> {code:java}
> commits show --sortBy "Total Bytes Written" --desc true --limit 10{code}
> when sortBy field not in columns, it throw
> {code:java}
> Command failed java.lang.ArrayIndexOutOfBoundsException: -1
> -1
> java.lang.ArrayIndexOutOfBoundsException: -1
> at java.util.ArrayList.elementData(ArrayList.java:418)
> at java.util.ArrayList.get(ArrayList.java:431)
> at org.apache.hudi.cli.Table$1.compare(Table.java:125)
> at org.apache.hudi.cli.Table$1.compare(Table.java:122)
> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355){code}
> It is better to give a friendly hint as:
> {code:java}
> Field[Total Bytes Writte] is not in table, given columns[[CommitTime, Total
> Bytes Written, Total Files Added, Total Files Updated, Total Partitions
> Written, Total Records Written, Total Update Records Written, Total Errors]]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HUDI-362) Adds a check for the existence of field
[
https://issues.apache.org/jira/browse/HUDI-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hong dongdong updated HUDI-362:
---
Description:
Use command
{code:java}
commits show --sortBy "Total Bytes Written" --desc true --limit 10{code}
when sortBy field not in columns, it throw
!image-2019-11-25-15-32-14-057.png!
It is better to give a friendly hint as: !image-2019-11-25-15-33-21-610.png!
was:
Use command
{code:java}
commits show --sortBy "Total Bytes Written" --desc true --limit 10{code}
when sortBy field not in columns, it throw
{code:java}
Command failed java.lang.ArrayIndexOutOfBoundsException: -1
-1
java.lang.ArrayIndexOutOfBoundsException: -1
at java.util.ArrayList.elementData(ArrayList.java:418)
at java.util.ArrayList.get(ArrayList.java:431)
at org.apache.hudi.cli.Table$1.compare(Table.java:125)
at org.apache.hudi.cli.Table$1.compare(Table.java:122)
at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355){code}
It is better to give a friendly hint as:
{code:java}
Field[Total Bytes Writte] is not in table, given columns[[CommitTime, Total
Bytes Written, Total Files Added, Total Files Updated, Total Partitions
Written, Total Records Written, Total Update Records Written, Total Errors]]
{code}
> Adds a check for the existence of field
> ---
>
> Key: HUDI-362
> URL: https://issues.apache.org/jira/browse/HUDI-362
> Project: Apache Hudi (incubating)
> Issue Type: Improvement
> Components: CLI
>Reporter: hong dongdong
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2019-11-25-15-32-14-057.png,
> image-2019-11-25-15-33-21-610.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Use command
> {code:java}
> commits show --sortBy "Total Bytes Written" --desc true --limit 10{code}
> when sortBy field not in columns, it throw
> !image-2019-11-25-15-32-14-057.png!
> It is better to give a friendly hint as: !image-2019-11-25-15-33-21-610.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HUDI-362) Adds a check for the existence of field
[
https://issues.apache.org/jira/browse/HUDI-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hong dongdong updated HUDI-362:
---
Attachment: image-2019-11-25-15-32-14-057.png
> Adds a check for the existence of field
> ---
>
> Key: HUDI-362
> URL: https://issues.apache.org/jira/browse/HUDI-362
> Project: Apache Hudi (incubating)
> Issue Type: Improvement
> Components: CLI
>Reporter: hong dongdong
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2019-11-25-15-32-14-057.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Use command
> {code:java}
> commits show --sortBy "Total Bytes Written" --desc true --limit 10{code}
> when sortBy field not in columns, it throw
> {code:java}
> Command failed java.lang.ArrayIndexOutOfBoundsException: -1
> -1
> java.lang.ArrayIndexOutOfBoundsException: -1
> at java.util.ArrayList.elementData(ArrayList.java:418)
> at java.util.ArrayList.get(ArrayList.java:431)
> at org.apache.hudi.cli.Table$1.compare(Table.java:125)
> at org.apache.hudi.cli.Table$1.compare(Table.java:122)
> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355){code}
> It is better to give a friendly hint as:
> {code:java}
> Field[Total Bytes Writte] is not in table, given columns[[CommitTime, Total
> Bytes Written, Total Files Added, Total Files Updated, Total Partitions
> Written, Total Records Written, Total Update Records Written, Total Errors]]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HUDI-362) Adds a check for the existence of field
[
https://issues.apache.org/jira/browse/HUDI-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hong dongdong updated HUDI-362:
---
Summary: Adds a check for the existence of field (was: Add a check if
field in columns.)
> Adds a check for the existence of field
> ---
>
> Key: HUDI-362
> URL: https://issues.apache.org/jira/browse/HUDI-362
> Project: Apache Hudi (incubating)
> Issue Type: Improvement
> Components: CLI
>Reporter: hong dongdong
>Priority: Major
>
> Use command
> {code:java}
> commits show --sortBy "Total Bytes Written" --desc true --limit 10{code}
> when sortBy field not in columns, it throw
> {code:java}
> Command failed java.lang.ArrayIndexOutOfBoundsException: -1
> -1
> java.lang.ArrayIndexOutOfBoundsException: -1
> at java.util.ArrayList.elementData(ArrayList.java:418)
> at java.util.ArrayList.get(ArrayList.java:431)
> at org.apache.hudi.cli.Table$1.compare(Table.java:125)
> at org.apache.hudi.cli.Table$1.compare(Table.java:122)
> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355){code}
> It is better to give a friendly hint as:
> {code:java}
> Field[Total Bytes Writte] is not in table, given columns[[CommitTime, Total
> Bytes Written, Total Files Added, Total Files Updated, Total Partitions
> Written, Total Records Written, Total Update Records Written, Total Errors]]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HUDI-362) Add a check if field in columns.
hong dongdong created HUDI-362:
--
Summary: Add a check if field in columns.
Key: HUDI-362
URL: https://issues.apache.org/jira/browse/HUDI-362
Project: Apache Hudi (incubating)
Issue Type: Improvement
Components: CLI
Reporter: hong dongdong
Use command
{code:java}
commits show --sortBy "Total Bytes Written" --desc true --limit 10{code}
when sortBy field not in columns, it throw
{code:java}
Command failed java.lang.ArrayIndexOutOfBoundsException: -1
-1
java.lang.ArrayIndexOutOfBoundsException: -1
at java.util.ArrayList.elementData(ArrayList.java:418)
at java.util.ArrayList.get(ArrayList.java:431)
at org.apache.hudi.cli.Table$1.compare(Table.java:125)
at org.apache.hudi.cli.Table$1.compare(Table.java:122)
at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355){code}
It is better to give a friendly hint as:
{code:java}
Field[Total Bytes Writte] is not in table, given columns[[CommitTime, Total
Bytes Written, Total Files Added, Total Files Updated, Total Partitions
Written, Total Records Written, Total Update Records Written, Total Errors]]
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HUDI-359) Add hudi-env for hudi-cli module
hong dongdong created HUDI-359: -- Summary: Add hudi-env for hudi-cli module Key: HUDI-359 URL: https://issues.apache.org/jira/browse/HUDI-359 Project: Apache Hudi (incubating) Issue Type: Improvement Components: CLI Reporter: hong dongdong Add hudi-env.sh for hudi-cli module to set running environments. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-356) Sync translation and code in quickstart.cn and admin_guide.cn pages
hong dongdong created HUDI-356: -- Summary: Sync translation and code in quickstart.cn and admin_guide.cn pages Key: HUDI-356 URL: https://issues.apache.org/jira/browse/HUDI-356 Project: Apache Hudi (incubating) Issue Type: Improvement Components: Docs Reporter: hong dongdong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-345) Fix used deprecated function
hong dongdong created HUDI-345: -- Summary: Fix used deprecated function Key: HUDI-345 URL: https://issues.apache.org/jira/browse/HUDI-345 Project: Apache Hudi (incubating) Issue Type: Improvement Components: Common Core Reporter: hong dongdong Fix used deprecated function to be compatible with higher version of hadoop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
