[jira] [Created] (CARBONDATA-4234) Alter change datatype at nested levels
Akshay created CARBONDATA-4234: -- Summary: Alter change datatype at nested levels Key: CARBONDATA-4234 URL: https://issues.apache.org/jira/browse/CARBONDATA-4234 Project: CarbonData Issue Type: Sub-task Reporter: Akshay -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4199) Support renaming of map columns including nested levels
[ https://issues.apache.org/jira/browse/CARBONDATA-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4199: --- Summary: Support renaming of map columns including nested levels (was: Support renaming of map columns) > Support renaming of map columns including nested levels > --- > > Key: CARBONDATA-4199 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4199 > Project: CarbonData > Issue Type: Sub-task >Reporter: Akshay >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4233) Altering complex structure like add and drop at nested level
[ https://issues.apache.org/jira/browse/CARBONDATA-4233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4233: --- Parent: CARBONDATA-4160 Issue Type: Sub-task (was: Bug) > Altering complex structure like add and drop at nested level > > > Key: CARBONDATA-4233 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4233 > Project: CarbonData > Issue Type: Sub-task >Reporter: Akshay >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4233) Altering complex structure like add and drop at nested level
Akshay created CARBONDATA-4233: -- Summary: Altering complex structure like add and drop at nested level Key: CARBONDATA-4233 URL: https://issues.apache.org/jira/browse/CARBONDATA-4233 Project: CarbonData Issue Type: Bug Reporter: Akshay -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4179) Support renaming of complex columns including nested levels(array/struct)
[ https://issues.apache.org/jira/browse/CARBONDATA-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4179: --- Summary: Support renaming of complex columns including nested levels(array/struct) (was: Support renaming of complex columns(array/struct)) > Support renaming of complex columns including nested levels(array/struct) > - > > Key: CARBONDATA-4179 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4179 > Project: CarbonData > Issue Type: Sub-task >Reporter: Akshay >Priority: Major > Fix For: 2.2.0 > > Time Spent: 21.5h > Remaining Estimate: 0h > > Complex columns supported for - *array and struct.* > For schema lets say- *str struct* the following points will be > considered - > 1) Rename parent name *str to str2* > alter table test_rename change str str2 struct > > 2) Rename child name from *a to b* > alter table test_rename change str str struct > > *NOTE-* > Rename operation fails if the structure of the complex column has been > altered. This check ensures the old and new columns are compatible with each > other. Meaning the *number of children* and *complex levels* should be > unaltered while attempting to rename. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4170) Support dropping of parent complex columns(for both single and multi-level array/struct/map)
[ https://issues.apache.org/jira/browse/CARBONDATA-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4170: --- Summary: Support dropping of parent complex columns(for both single and multi-level array/struct/map) (was: Support dropping of parent complex columns(array/struct/map)) > Support dropping of parent complex columns(for both single and multi-level > array/struct/map) > > > Key: CARBONDATA-4170 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4170 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Akshay >Priority: Major > Fix For: 2.2.0 > > Time Spent: 10h 40m > Remaining Estimate: 0h > > Drop complex columns(array/struct/map) from carbon table. For example - > arr1 array, struct1 struct, map1 map > Command - > ALTER TABLE DROP COLUMNS(arr1, struct1, map1) > Design document - > [https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4216) Exception during alter add struct enabling local dictionary
Akshay created CARBONDATA-4216: -- Summary: Exception during alter add struct enabling local dictionary Key: CARBONDATA-4216 URL: https://issues.apache.org/jira/browse/CARBONDATA-4216 Project: CarbonData Issue Type: Bug Components: spark-integration Reporter: Akshay Array Index Out Of Bounds exception while adding struct when local dictionary is enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4198) Support adding of single-level and multi-level map columns
Akshay created CARBONDATA-4198: -- Summary: Support adding of single-level and multi-level map columns Key: CARBONDATA-4198 URL: https://issues.apache.org/jira/browse/CARBONDATA-4198 Project: CarbonData Issue Type: Sub-task Reporter: Akshay -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4199) Support renaming of map columns
Akshay created CARBONDATA-4199: -- Summary: Support renaming of map columns Key: CARBONDATA-4199 URL: https://issues.apache.org/jira/browse/CARBONDATA-4199 Project: CarbonData Issue Type: Sub-task Reporter: Akshay -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4179) Support renaming of complex columns(array/struct)
[ https://issues.apache.org/jira/browse/CARBONDATA-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4179: --- Summary: Support renaming of complex columns(array/struct) (was: Support renaming of complex columns(struct/array)) > Support renaming of complex columns(array/struct) > - > > Key: CARBONDATA-4179 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4179 > Project: CarbonData > Issue Type: Sub-task >Reporter: Akshay >Priority: Major > Time Spent: 13h > Remaining Estimate: 0h > > Complex columns supported for - *array and struct.* > For schema lets say- *str struct* the following points will be > considered - > 1) Rename parent name *str to str2* > alter table test_rename change str str2 struct > > 2) Rename child name from *a to b* > alter table test_rename change str str struct > > *NOTE-* > Rename operation fails if the structure of the complex column has been > altered. This check ensures the old and new columns are compatible with each > other. Meaning the *number of children* and *complex levels* should be > unaltered while attempting to rename. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4179) Support renaming of complex columns(struct/array)
[ https://issues.apache.org/jira/browse/CARBONDATA-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4179: --- Summary: Support renaming of complex columns(struct/array) (was: Support renaming of complex columns) > Support renaming of complex columns(struct/array) > - > > Key: CARBONDATA-4179 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4179 > Project: CarbonData > Issue Type: Sub-task >Reporter: Akshay >Priority: Major > Time Spent: 13h > Remaining Estimate: 0h > > Complex columns supported for - *array and struct.* > For schema lets say- *str struct* the following points will be > considered - > 1) Rename parent name *str to str2* > alter table test_rename change str str2 struct > > 2) Rename child name from *a to b* > alter table test_rename change str str struct > > *NOTE-* > Rename operation fails if the structure of the complex column has been > altered. This check ensures the old and new columns are compatible with each > other. Meaning the *number of children* and *complex levels* should be > unaltered while attempting to rename. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4179) Support renaming of complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4179: --- Description: Complex columns supported for - *array and struct.* For schema lets say- *str struct* the following points will be considered - 1) Rename parent name *str to str2* alter table test_rename change str str2 struct 2) Rename child name from *a to b* alter table test_rename change str str struct *NOTE-* Rename operation fails if the structure of the complex column has been altered. This check ensures the old and new columns are compatible with each other. Meaning the *number of children* and *complex levels* should be unaltered while attempting to rename. was: Complex columns supported for - *array and struct.* For schema lets say- *str struct* the following points will be considered - 1) Rename parent name *str to str2* alter table test_rename change str str2 struct 2) Rename child name from *a to b* alter table test_rename change str str struct 3) Change datatype of *a* from *int to long* alter table test_rename change str str struct *NOTE*- Renaming and changing datatype both operations should be done without changing the levels of the complex column. For example : 1) int cannot be converted to an array or struct. 2) array cannot be converted to struct and vice-versa > Support renaming of complex columns > --- > > Key: CARBONDATA-4179 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4179 > Project: CarbonData > Issue Type: Sub-task >Reporter: Akshay >Priority: Major > > Complex columns supported for - *array and struct.* > For schema lets say- *str struct* the following points will be > considered - > 1) Rename parent name *str to str2* > alter table test_rename change str str2 struct > > 2) Rename child name from *a to b* > alter table test_rename change str str struct > > *NOTE-* > Rename operation fails if the structure of the complex column has been > altered. This check ensures the old and new columns are compatible with each > other. Meaning the *number of children* and *complex levels* should be > unaltered while attempting to rename. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4179) Support renaming of complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4179: --- Summary: Support renaming of complex columns (was: Support rename/change datatype for complex columns) > Support renaming of complex columns > --- > > Key: CARBONDATA-4179 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4179 > Project: CarbonData > Issue Type: Sub-task >Reporter: Akshay >Priority: Major > > Complex columns supported for - *array and struct.* > For schema lets say- *str struct* the following points will be > considered - > 1) Rename parent name *str to str2* > alter table test_rename change str str2 struct > > 2) Rename child name from *a to b* > alter table test_rename change str str struct > > 3) Change datatype of *a* from *int to long* > alter table test_rename change str str struct > > *NOTE*- Renaming and changing datatype both operations should be done without > changing the levels of the complex column. For example : > 1) int cannot be converted to an array or struct. > 2) array cannot be converted to struct and vice-versa -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4179) Support rename/change datatype for complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4179: --- Description: Complex columns supported for - *array and struct.* For schema lets say- *str struct* the following points will be considered - 1) Rename parent name *str to str2* alter table test_rename change str str2 struct 2) Rename child name from *a to b* alter table test_rename change str str struct 3) Change datatype of *a* from *int to long* alter table test_rename change str str struct *NOTE*- Renaming and changing datatype both operations should be done without changing the levels of the complex column. For example : 1) int cannot be converted to an array or struct. 2) array cannot be converted to struct and vice-versa was: The following points will be considered for schema lets say- *str struct* - 1) Rename parent name *str to str2* alter table test_rename change str str2 struct 2) Rename child name from *a to b* alter table test_rename change str str struct 3) Change datatype of *a* from *int to long* alter table test_rename change str str struct *NOTE*- Renaming and changing datatype both operations should be done without changing the levels of the complex column. For example : 1) int cannot be converted to an array or struct. 2) array cannot be converted to struct and vice-versa > Support rename/change datatype for complex columns > -- > > Key: CARBONDATA-4179 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4179 > Project: CarbonData > Issue Type: Sub-task >Reporter: Akshay >Priority: Major > > Complex columns supported for - *array and struct.* > For schema lets say- *str struct* the following points will be > considered - > 1) Rename parent name *str to str2* > alter table test_rename change str str2 struct > > 2) Rename child name from *a to b* > alter table test_rename change str str struct > > 3) Change datatype of *a* from *int to long* > alter table test_rename change str str struct > > *NOTE*- Renaming and changing datatype both operations should be done without > changing the levels of the complex column. For example : > 1) int cannot be converted to an array or struct. > 2) array cannot be converted to struct and vice-versa -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4179) Support rename/change datatype for complex columns
Akshay created CARBONDATA-4179: -- Summary: Support rename/change datatype for complex columns Key: CARBONDATA-4179 URL: https://issues.apache.org/jira/browse/CARBONDATA-4179 Project: CarbonData Issue Type: Sub-task Reporter: Akshay The following points will be considered for schema lets say- *str struct* - 1) Rename parent name *str to str2* alter table test_rename change str str2 struct 2) Rename child name from *a to b* alter table test_rename change str str struct 3) Change datatype of *a* from *int to long* alter table test_rename change str str struct *NOTE*- Renaming and changing datatype both operations should be done without changing the levels of the complex column. For example : 1) int cannot be converted to an array or struct. 2) array cannot be converted to struct and vice-versa -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4175) Issue with array_contains after altering schema for array types
[ https://issues.apache.org/jira/browse/CARBONDATA-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4175: --- Description: NPE on executing filter query after adding array column to the carbon table > Issue with array_contains after altering schema for array types > --- > > Key: CARBONDATA-4175 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4175 > Project: CarbonData > Issue Type: Bug > Components: spark-integration >Reporter: Akshay >Priority: Major > > NPE on executing filter query after adding array column to the carbon table -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4175) Issue with array_contains after altering schema for array types
Akshay created CARBONDATA-4175: -- Summary: Issue with array_contains after altering schema for array types Key: CARBONDATA-4175 URL: https://issues.apache.org/jira/browse/CARBONDATA-4175 Project: CarbonData Issue Type: Bug Components: spark-integration Reporter: Akshay -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4170) Support dropping of parent complex columns(array/struct/map)
[ https://issues.apache.org/jira/browse/CARBONDATA-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4170: --- Description: Drop complex columns(array/struct/map) from carbon table. For example - arr1 array, struct1 struct, map1 map Command - ALTER TABLE DROP COLUMNS(arr1, struct1, map1) Design document - [https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit] was: Drop complex columns(array/struct/map) from carbon table. For example - arr1 array, struct1 struct, map1 map Command - ALTER TABLE DROP COLUMNS(arr1, struct1,map1) Design document - [https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit] > Support dropping of parent complex columns(array/struct/map) > > > Key: CARBONDATA-4170 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4170 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Akshay >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Drop complex columns(array/struct/map) from carbon table. For example - > arr1 array, struct1 struct, map1 map > Command - > ALTER TABLE DROP COLUMNS(arr1, struct1, map1) > Design document - > [https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4170) Support dropping of parent complex columns(array/struct/map)
[ https://issues.apache.org/jira/browse/CARBONDATA-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4170: --- Description: Drop complex columns(array/struct/map) from carbon table. For example - arr1 array, struct1 struct, map1 map Command - ALTER TABLE DROP COLUMNS(arr1, struct1,map1) Design document - [https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit] was: Drop complex columns(only array and struct) from carbon table. For example - arr1 array, struct1 struct Command - ALTER TABLE DROP COLUMNS(arr1, struct1) Design document - [https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit] > Support dropping of parent complex columns(array/struct/map) > > > Key: CARBONDATA-4170 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4170 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Akshay >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Drop complex columns(array/struct/map) from carbon table. For example - > arr1 array, struct1 struct, map1 map > Command - > ALTER TABLE DROP COLUMNS(arr1, struct1,map1) > Design document - > [https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4170) Support dropping of parent complex columns(array/struct/map)
[ https://issues.apache.org/jira/browse/CARBONDATA-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4170: --- Summary: Support dropping of parent complex columns(array/struct/map) (was: Support dropping of single & multi-level complex columns(array/struct)) > Support dropping of parent complex columns(array/struct/map) > > > Key: CARBONDATA-4170 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4170 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Akshay >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Drop complex columns(only array and struct) from carbon table. For example - > arr1 array, struct1 struct > Command - > ALTER TABLE DROP COLUMNS(arr1, struct1) > Design document - > [https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4160) Alter carbon schema related to complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4160: --- Summary: Alter carbon schema related to complex columns(was: Alter carbon schema related to complex columns(array/struct) ) > Alter carbon schema related to complex columns > > > Key: CARBONDATA-4160 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4160 > Project: CarbonData > Issue Type: New Feature > Components: spark-integration >Reporter: Akshay >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > Add complex columns(only array and struct) to carbon table. For example - > array, struct > Command - > ALTER TABLE ADD COLUMNS(arr array) > Design document - > [https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4170) Support dropping of single & multi-level complex columns(array/struct)
[ https://issues.apache.org/jira/browse/CARBONDATA-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4170: --- Description: Drop complex columns(only array and struct) from carbon table. For example - arr1 array, struct1 struct Command - ALTER TABLE DROP COLUMNS(arr1, struct1) Design document - [https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit] > Support dropping of single & multi-level complex columns(array/struct) > -- > > Key: CARBONDATA-4170 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4170 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Akshay >Priority: Major > > Drop complex columns(only array and struct) from carbon table. For example - > arr1 array, struct1 struct > Command - > ALTER TABLE DROP COLUMNS(arr1, struct1) > Design document - > [https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4170) Support dropping of single & multi-level complex columns(array/struct)
Akshay created CARBONDATA-4170: -- Summary: Support dropping of single & multi-level complex columns(array/struct) Key: CARBONDATA-4170 URL: https://issues.apache.org/jira/browse/CARBONDATA-4170 Project: CarbonData Issue Type: Sub-task Components: spark-integration Reporter: Akshay -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4164) Support adding of multi-level complex columns(array/struct)
[ https://issues.apache.org/jira/browse/CARBONDATA-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4164: --- Description: Add multi-level(upto 3 nested levels) complex columns(only array and struct) to carbon table. For example - Command - ALTER TABLE ADD COLUMNS(arr array >) > Support adding of multi-level complex columns(array/struct) > --- > > Key: CARBONDATA-4164 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4164 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Akshay >Priority: Major > > Add multi-level(upto 3 nested levels) complex columns(only array and struct) > to carbon table. For example - > Command - > ALTER TABLE ADD COLUMNS(arr array >) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4160) Alter carbon schema related to complex columns(array/struct)
[ https://issues.apache.org/jira/browse/CARBONDATA-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4160: --- Description: Add complex columns(only array and struct) to carbon table. For example - array, struct Command - ALTER TABLE ADD COLUMNS(arr array) Design document - [https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit] was: Add single-level complex columns(only array and struct) to carbon table. For example - array, struct Command - ALTER TABLE ADD COLUMNS(arr array) Design document - https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit > Alter carbon schema related to complex columns(array/struct) > -- > > Key: CARBONDATA-4160 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4160 > Project: CarbonData > Issue Type: New Feature > Components: spark-integration >Reporter: Akshay >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > Add complex columns(only array and struct) to carbon table. For example - > array, struct > Command - > ALTER TABLE ADD COLUMNS(arr array) > Design document - > [https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4163) Support adding of single-level complex columns(array/struct)
[ https://issues.apache.org/jira/browse/CARBONDATA-4163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4163: --- Description: Add single-level complex columns(only array and struct) to carbon table. For example - array, struct Command - ALTER TABLE ADD COLUMNS(arr array) > Support adding of single-level complex columns(array/struct) > -- > > Key: CARBONDATA-4163 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4163 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Akshay >Priority: Major > > Add single-level complex columns(only array and struct) to carbon table. For > example - > array, struct > Command - > ALTER TABLE ADD COLUMNS(arr array) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4164) Support adding of multi-level complex columns(array/struct)
[ https://issues.apache.org/jira/browse/CARBONDATA-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4164: --- Component/s: spark-integration > Support adding of multi-level complex columns(array/struct) > --- > > Key: CARBONDATA-4164 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4164 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Akshay >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4163) Support adding of single-level complex columns(array/struct)
Akshay created CARBONDATA-4163: -- Summary: Support adding of single-level complex columns(array/struct) Key: CARBONDATA-4163 URL: https://issues.apache.org/jira/browse/CARBONDATA-4163 Project: CarbonData Issue Type: Sub-task Components: spark-integration Reporter: Akshay -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4164) Support adding of multi-level complex columns(array/struct)
Akshay created CARBONDATA-4164: -- Summary: Support adding of multi-level complex columns(array/struct) Key: CARBONDATA-4164 URL: https://issues.apache.org/jira/browse/CARBONDATA-4164 Project: CarbonData Issue Type: Sub-task Reporter: Akshay -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4160) Alter carbon schema related to complex columns(array/struct)
[ https://issues.apache.org/jira/browse/CARBONDATA-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4160: --- Summary: Alter carbon schema related to complex columns(array/struct) (was: Support add complex columns(array/struct) ) > Alter carbon schema related to complex columns(array/struct) > -- > > Key: CARBONDATA-4160 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4160 > Project: CarbonData > Issue Type: New Feature > Components: spark-integration >Reporter: Akshay >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > Add single-level complex columns(only array and struct) to carbon table. For > example - > array, struct > Command - > ALTER TABLE ADD COLUMNS(arr array) > Design document - > https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4160) Support add complex columns(array/struct)
[ https://issues.apache.org/jira/browse/CARBONDATA-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4160: --- Summary: Support add complex columns(array/struct)(was: Alter carbon schema by adding single-level complex columns(array/struct) ) > Support add complex columns(array/struct) > --- > > Key: CARBONDATA-4160 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4160 > Project: CarbonData > Issue Type: New Feature > Components: spark-integration >Reporter: Akshay >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > Add single-level complex columns(only array and struct) to carbon table. For > example - > array, struct > Command - > ALTER TABLE ADD COLUMNS(arr array) > Design document - > https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4160) Alter carbon schema by adding single-level complex columns(array/struct)
[ https://issues.apache.org/jira/browse/CARBONDATA-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4160: --- Description: Add single-level complex columns(only array and struct) to carbon table. For example - array, struct Command - ALTER TABLE ADD COLUMNS(arr array) Design document - https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit was: Add single-level complex columns(only array and struct) to carbon table. For example - array, struct > Alter carbon schema by adding single-level complex columns(array/struct) > -- > > Key: CARBONDATA-4160 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4160 > Project: CarbonData > Issue Type: New Feature > Components: spark-integration >Reporter: Akshay >Priority: Major > > Add single-level complex columns(only array and struct) to carbon table. For > example - > array, struct > Command - > ALTER TABLE ADD COLUMNS(arr array) > Design document - > https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4160) Alter carbon schema by adding single-level complex columns(array/struct)
[ https://issues.apache.org/jira/browse/CARBONDATA-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4160: --- Description: Add single-level complex columns(only array and struct) to carbon table. For example - array, struct was: Add single-level complex columns to carbon table. For example - array, struct > Alter carbon schema by adding single-level complex columns(array/struct) > -- > > Key: CARBONDATA-4160 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4160 > Project: CarbonData > Issue Type: New Feature > Components: spark-integration >Reporter: Akshay >Priority: Major > > Add single-level complex columns(only array and struct) to carbon table. For > example - > array, struct -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4160) Alter carbon schema by adding single-level complex columns(array/struct)
Akshay created CARBONDATA-4160: -- Summary: Alter carbon schema by adding single-level complex columns(array/struct) Key: CARBONDATA-4160 URL: https://issues.apache.org/jira/browse/CARBONDATA-4160 Project: CarbonData Issue Type: New Feature Components: spark-integration Reporter: Akshay Add single-level complex columns to carbon table. For example - array, struct -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4109) Improve carbondata coverage for presto-integration code
Akshay created CARBONDATA-4109: -- Summary: Improve carbondata coverage for presto-integration code Key: CARBONDATA-4109 URL: https://issues.apache.org/jira/browse/CARBONDATA-4109 Project: CarbonData Issue Type: Improvement Components: core, presto-integration Reporter: Akshay Few scenarios had missing coverage in presto-integration code. This PR aims to improve it by considering all such scenarios. Dead code- ObjectStreamReader.java was created with an aim to query complex types. Instead ComplexTypeStreamReader was created. Making ObjectStreamreader obsolete. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4104) Vector filling for Primitive decimal type needs to be handled
Akshay created CARBONDATA-4104: -- Summary: Vector filling for Primitive decimal type needs to be handled Key: CARBONDATA-4104 URL: https://issues.apache.org/jira/browse/CARBONDATA-4104 Project: CarbonData Issue Type: Bug Components: core Reporter: Akshay Filling of vectors in case of complex primitive decimal type whose precision is greater than 18 is not handled properly. for ex array -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CARBONDATA-4049) Sometimes refresh table fails with error "table not found in database" error
[ https://issues.apache.org/jira/browse/CARBONDATA-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256956#comment-17256956 ] Akshay commented on CARBONDATA-4049: * Issue is on refresh table and we don't know what kind of operations were done prior. * On re-creating the session this error never comes. Meaning there is a possibility that the cache is cleared thereby removing the cause of this error. * This issue is not 100% reproducible * Also this issue is not impacting any other scenarios > Sometimes refresh table fails with error "table not found in database" error > > > Key: CARBONDATA-4049 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4049 > Project: CarbonData > Issue Type: Bug > Components: data-query >Affects Versions: 2.1.0 > Environment: Spark 2.4.5 >Reporter: Chetan Bhat >Priority: Minor > > In Carbon 2.1 version user creates a database. > user copies a old version store such as 1.6.1 to HDFS folder of the database > in the In Carbon 2.1 version > In Spark-SQL or beeline the user accesses the database using the use db > command. > Refresh table command is executed on the old version store table and then the > subsequent operations on the table are performed. > Next refresh table command is tried to be executed on another old version > store table . > > Issue : Sometimes refresh table fails with error "table not found in > database" error. > spark-sql> refresh table brinjal_deleteseg; > *Error in query: Table or view 'brinjal_deleteseg' not found in database > '1_6_1';* > > **Log - > 2020-11-12 18:55:46,922 | INFO | [main] | Created broadcast 171 from > broadCastHadoopConf at CarbonRDD.scala:58 | > org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)2020-11-12 > 18:55:46,922 | INFO | [main] | Created broadcast 171 from > broadCastHadoopConf at CarbonRDD.scala:58 | > org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)2020-11-12 > 18:55:46,924 | INFO | [main] | Pushed Filters: | > org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)2020-11-12 > 18:55:46,939 | INFO | [main] | Distributed Index server is enabled for > 1_6_1.brinjal_update | > org.apache.carbondata.core.util.CarbonProperties.isDistributedPruningEnabled(CarbonProperties.java:1742)2020-11-12 > 18:55:46,939 | INFO | [main] | Started block pruning ... | > org.apache.carbondata.hadoop.api.CarbonInputFormat.getPrunedBlocklets(CarbonInputFormat.java:526)2020-11-12 > 18:55:46,940 | INFO | [main] | Distributed Index server is enabled for > 1_6_1.brinjal_update | > org.apache.carbondata.core.util.CarbonProperties.isDistributedPruningEnabled(CarbonProperties.java:1742)2020-11-12 > 18:55:46,945 | INFO | [main] | Successfully Created directory: > hdfs://hacluster/tmp/indexservertmp/4b6353d4-65d7-4856-b3cd-b3bc11d15c55 | > org.apache.carbondata.core.util.CarbonUtil.createTempFolderForIndexServer(CarbonUtil.java:3273)2020-11-12 > 18:55:46,945 | INFO | [main] | Temp folder path for Query ID: > 4b6353d4-65d7-4856-b3cd-b3bc11d15c55 is > org.apache.carbondata.core.datastore.filesystem.HDFSCarbonFile@b8f2e1bf | > org.apache.carbondata.indexserver.DistributedIndexJob.execute(IndexJobs.scala:57)2020-11-12 > 18:55:46,946 | ERROR | [main] | Configured port for index server is not a > valid number | > org.apache.carbondata.core.util.CarbonProperties.getIndexServerPort(CarbonProperties.java:1779)java.lang.NumberFormatException: > null at java.lang.Integer.parseInt(Integer.java:542) at > java.lang.Integer.parseInt(Integer.java:615) at > org.apache.carbondata.core.util.CarbonProperties.getIndexServerPort(CarbonProperties.java:1777) > at > org.apache.carbondata.indexserver.IndexServer$.serverPort$lzycompute(IndexServer.scala:88) > at > org.apache.carbondata.indexserver.IndexServer$.serverPort(IndexServer.scala:88) > at > org.apache.carbondata.indexserver.IndexServer$.getClient(IndexServer.scala:312) > at > org.apache.carbondata.indexserver.IndexServer$.getClient(IndexServer.scala:301) > at > org.apache.carbondata.indexserver.DistributedIndexJob$$anonfun$1.apply(IndexJobs.scala:83) > at > org.apache.carbondata.indexserver.DistributedIndexJob$$anonfun$1.apply(IndexJobs.scala:59) > at > org.apache.carbondata.spark.util.CarbonScalaUtil$.logTime(CarbonScalaUtil.scala:769) > at > org.apache.carbondata.indexserver.DistributedIndexJob.execute(IndexJobs.scala:58) > at > org.apache.carbondata.core.index.IndexUtil.executeIndexJob(IndexUtil.java:304) > at > org.apache.carbondata.hadoop.api.CarbonInputFormat.getDistributedSplit(CarbonInputFormat.java:431) > at > org.apache.carbondata.hadoop.api.CarbonInputFormat.getPrunedBlocklets(CarbonInputFormat.java:532) > at > org.apache.carbondata.h
[jira] [Commented] (CARBONDATA-3905) When there are many segment files presto query fail
[
https://issues.apache.org/jira/browse/CARBONDATA-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17255411#comment-17255411
]
Akshay commented on CARBONDATA-3905:
Handled in PR -
https://github.com/apache/carbondata/pull/3967
> When there are many segment files presto query fail
> ---
>
> Key: CARBONDATA-3905
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3905
> Project: CarbonData
> Issue Type: Bug
> Components: presto-integration
>Affects Versions: 2.0.0
>Reporter: XiaoWen
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> test case1
> insert data in:
> {code:java}
> df.writeStream.foreachBatch{ (batchDF: DataFrame, batchId: Long) => {
> ...
> val cond = $"B.id".isin(df.select(col = "id").as[Int].collect: _*)
> target.as("A")
> .merge(df.as("B"), "A.id = B.id")
> .whenMatched(cond)
> .updateExpr(Map("name" -> "B.name", "city" -> "B.city", "age" ->
> "B.age"))
> .whenNotMatched(cond)
> .insertExpr(Map("id" -> "B.id", "name" -> "B.name", "city" -> "B.city",
> "age" -> "B.age"))
> .execute()
> ...
> }).outputMode("update").trigger(Trigger.ProcessingTime("3600
> seconds")).start()
> {code}
> a lot of segment files will be generated after a few hours
> when i try to use presto to query.
> single condition can be queried, but cannot be queried when there are
> multiple conditions.
> select name from test_table // ok
> select name from test_table where name = 'joe' // ok
> select name from test_table where name='joe' AND age > 25;// query failed
> select name from test_table where name='joe' AND age > 25 AND city
> ='shenzhen';// query failed
> i have also tried to compact 'major' the segment files to reduce the segment
> quantity, and I still cannot query successfully.
> presto server logs
> java.lang.IllegalArgumentException: Invalid position 0 in block with 0
> positions
> at io.prestosql.spi.block.BlockUtil.checkValidPosition(BlockUtil.java:62)
> at
> io.prestosql.spi.block.AbstractVariableWidthBlock.checkReadablePosition(AbstractVariableWidthBlock.java:160)
> at
> io.prestosql.spi.block.AbstractVariableWidthBlock.isNull(AbstractVariableWidthBlock.java:154)
> at io.prestosql.spi.block.LazyBlock.isNull(LazyBlock.java:248)
> at io.prestosql.$gen.PageFilter_20200703_084817_965.filter(Unknown Source)
> at io.prestosql.$gen.PageFilter_20200703_084817_965.filter(Unknown Source)
> at
> io.prestosql.operator.project.PageProcessor.createWorkProcessor(PageProcessor.java:115)
> at
> io.prestosql.operator.ScanFilterAndProjectOperator$SplitToPages.lambda$processPageSource$1(ScanFilterAndProjectOperator.java:254)
> at
> io.prestosql.operator.WorkProcessorUtils.lambda$flatMap$4(WorkProcessorUtils.java:246)
> at
> io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:320)
> at
> io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:373)
> at
> io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:307)
> at
> io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:373)
> at
> io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:307)
> at
> io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:373)
> at
> io.prestosql.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:221)
> at
> io.prestosql.operator.WorkProcessorUtils.lambda$processStateMonitor$2(WorkProcessorUtils.java:200)
> at
> io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:373)
> at
> io.prestosql.operator.WorkProcessorUtils.lambda$flatten$6(WorkProcessorUtils.java:278)
> at
> io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:320)
> at
> io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:373)
> at
> io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:307)
> at
> io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:373)
> at
> io.prestosql.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:221)
> at
> io.prestosql.operator.WorkProcessorUtils.lambda$processStateMonitor$2(WorkProcessorUtils.java:200)
> at
> io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:373)
> at
> io.prestosql.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:221)
> at
> io.prestosql.operator.WorkProcessorUtils.lambda$finishWhen$3(WorkProcessorUtils.java:215)
> at
> io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:373)
> at
> io.
[jira] [Updated] (CARBONDATA-4087) Issue with huge data(exceeding 32K records) after enabling local dictionary
[ https://issues.apache.org/jira/browse/CARBONDATA-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4087: --- Description: For large data SELECT on array(varchar) throws exception- "Error in Reading Data from Carbondata" due to ArrayOutOfBounds https://github.com/apache/carbondata/pull/4055 was: For large data SELECT on array(varchar) throws exception- "Error in Reading Data from Carbondata" due to ArrayOutOfBounds > Issue with huge data(exceeding 32K records) after enabling local dictionary > --- > > Key: CARBONDATA-4087 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4087 > Project: CarbonData > Issue Type: Bug > Components: core, presto-integration >Reporter: Akshay >Priority: Major > > For large data SELECT on array(varchar) throws exception- > "Error in Reading Data from Carbondata" due to ArrayOutOfBounds > > https://github.com/apache/carbondata/pull/4055 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4087) Issue with huge data(exceeding 32K records) after enabling local dictionary
[ https://issues.apache.org/jira/browse/CARBONDATA-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4087: --- Description: For large data SELECT on array(varchar) throws exception- "Error in Reading Data from Carbondata" due to ArrayOutOfBounds > Issue with huge data(exceeding 32K records) after enabling local dictionary > --- > > Key: CARBONDATA-4087 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4087 > Project: CarbonData > Issue Type: Bug > Components: core, presto-integration >Reporter: Akshay >Priority: Major > > For large data SELECT on array(varchar) throws exception- > "Error in Reading Data from Carbondata" due to ArrayOutOfBounds -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4087) Issue with huge data(exceeding 32K records) after enabling local dictionary
Akshay created CARBONDATA-4087: -- Summary: Issue with huge data(exceeding 32K records) after enabling local dictionary Key: CARBONDATA-4087 URL: https://issues.apache.org/jira/browse/CARBONDATA-4087 Project: CarbonData Issue Type: Bug Components: core, presto-integration Reporter: Akshay -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4080) Wrong results for select count on invalid segments
Akshay created CARBONDATA-4080: -- Summary: Wrong results for select count on invalid segments Key: CARBONDATA-4080 URL: https://issues.apache.org/jira/browse/CARBONDATA-4080 Project: CarbonData Issue Type: Bug Components: spark-integration Reporter: Akshay Wrong results for * select count on marked for delete segment * select count on compacted segment Issue comes only when the user explicitly sets deleted/compacted segments using the property carbon.input.segments. As select * on such segments gives 0 rows as output, in order to maintain consistency, select count should also give 0 rows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4073) Added FT for missing scenarios in Presto
[ https://issues.apache.org/jira/browse/CARBONDATA-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4073: --- Description: FT for following has been added. update without local-dict delete operation minor, major, custom compaction add and delete segments test update with inverted index read with partition columns Filter on partition columns Bloom index test range columns read streaming data was: FT for following has been added. update without local-dict delete operation minor, major, custom compaction add and delete segments test with inverted index read with partition columns Filter on partition columns Bloom index test range columns read streaming data > Added FT for missing scenarios in Presto > > > Key: CARBONDATA-4073 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4073 > Project: CarbonData > Issue Type: Test > Components: presto-integration >Reporter: Akshay >Priority: Major > > FT for following has been added. > update without local-dict > delete operation > minor, major, custom compaction > add and delete segments > test update with inverted index > read with partition columns > Filter on partition columns > Bloom index > test range columns > read streaming data -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4073) Added FT for missing scenarios in Presto
[ https://issues.apache.org/jira/browse/CARBONDATA-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4073: --- Description: FT for following has been added. update without local-dict delete operation minor, major, custom compaction add and delete segments test with inverted index read with partition columns Filter on partition columns Bloom index test range columns read streaming data was: update without local-dict delete operation minor, major, custom compaction add and delete segments test with inverted index read with partition columns Filter on partition columns Bloom index test range columns read streaming data > Added FT for missing scenarios in Presto > > > Key: CARBONDATA-4073 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4073 > Project: CarbonData > Issue Type: Test > Components: presto-integration >Reporter: Akshay >Priority: Major > > FT for following has been added. > update without local-dict > delete operation > minor, major, custom compaction > add and delete segments > test with inverted index > read with partition columns > Filter on partition columns > Bloom index > test range columns > read streaming data -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4073) Added FT for missing scenarios in Presto
Akshay created CARBONDATA-4073: -- Summary: Added FT for missing scenarios in Presto Key: CARBONDATA-4073 URL: https://issues.apache.org/jira/browse/CARBONDATA-4073 Project: CarbonData Issue Type: Test Components: presto-integration Reporter: Akshay update without local-dict delete operation minor, major, custom compaction add and delete segments test with inverted index read with partition columns Filter on partition columns Bloom index test range columns read streaming data -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CARBONDATA-3917) The rows of data loading is not accurate, more rows has been loaded
[ https://issues.apache.org/jira/browse/CARBONDATA-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17219223#comment-17219223 ] Akshay commented on CARBONDATA-3917: please provide further details for this issue. > The rows of data loading is not accurate, more rows has been loaded > --- > > Key: CARBONDATA-3917 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3917 > Project: CarbonData > Issue Type: Bug > Components: data-load >Affects Versions: 2.0.0 >Reporter: Taoli >Priority: Blocker > > 2020-07-18 18:46:23,856 | INFO | [Executor task launch worker for task 28380] > | Total rows processed in step Data Writer: 1277745 | > org.apache.carbondata.processing.loading.AbstractDataLoadProcessorStep.close(AbstractDataLoadProcessorStep.java:138) > 2020-07-18 18:46:23,857 | INFO | [Executor task launch worker for task 28380] > | Total rows processed in step Sort Processor: 1189959 | > org.apache.carbondata.processing.loading.AbstractDataLoadProcessorStep.close(AbstractDataLoadProcessorStep.java:138) > 2020-07-18 18:46:23,856 | DEBUG | > [LocalFolderDeletionPool:detail_cdr_s1mme_18461_1595087183856] | > PrivilegedAction as:omm (auth:SIMPLE) > from:org.apache.carbondata.core.util.CarbonUtil.deleteFoldersAndFiles(CarbonUtil.java:298) > | > org.apache.hadoop.security.UserGroupInformation.logPrivilegedAction(UserGroupInformation.java:1756) > 2020-07-18 18:46:23,857 | INFO | [Executor task launch worker for task 28380] > | Total rows processed in step Data Converter: 1189959 | > org.apache.carbondata.processing.loading.AbstractDataLoadProcessorStep.close(AbstractDataLoadProcessorStep.java:138) > 2020-07-18 18:46:23,857 | INFO | [Executor task launch worker for task 28380] > | Total rows processed in step Input Processor: 1189959 | > org.apache.carbondata.processing.loading.AbstractDataLoadProcessorStep.close(AbstractDataLoadProcessorStep.java:138) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (CARBONDATA-4039) Support Local dictionary for presto complex datatypes
[ https://issues.apache.org/jira/browse/CARBONDATA-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay reopened CARBONDATA-4039: > Support Local dictionary for presto complex datatypes > - > > Key: CARBONDATA-4039 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4039 > Project: CarbonData > Issue Type: New Feature > Components: core, presto-integration >Reporter: Akshay >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > > Support Local dictionary for presto complex datatypes - > Presto complex datatypes - array and struct only. > [https://github.com/apache/carbondata/pull/3987] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (CARBONDATA-4004) Wrong result in Presto select query after executing update
[ https://issues.apache.org/jira/browse/CARBONDATA-4004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay closed CARBONDATA-4004. -- > Wrong result in Presto select query after executing update > -- > > Key: CARBONDATA-4004 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4004 > Project: CarbonData > Issue Type: Bug > Components: core, presto-integration >Reporter: Akshay >Priority: Major > Fix For: 2.1.0 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > Presto select query after update operation returns different number of rows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (CARBONDATA-4039) Support Local dictionary for presto complex datatypes
[ https://issues.apache.org/jira/browse/CARBONDATA-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay closed CARBONDATA-4039. -- Resolution: Fixed > Support Local dictionary for presto complex datatypes > - > > Key: CARBONDATA-4039 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4039 > Project: CarbonData > Issue Type: New Feature > Components: core, presto-integration >Reporter: Akshay >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > > Support Local dictionary for presto complex datatypes - > Presto complex datatypes - array and struct only. > [https://github.com/apache/carbondata/pull/3987] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CARBONDATA-3953) Dead lock when doing dataframe persist and loading
[ https://issues.apache.org/jira/browse/CARBONDATA-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17218983#comment-17218983 ] Akshay commented on CARBONDATA-3953: Please provide proper test case to show the dataframe creation > Dead lock when doing dataframe persist and loading > -- > > Key: CARBONDATA-3953 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3953 > Project: CarbonData > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: ChenKai >Priority: Major > Attachments: image-2020-08-18-15-59-46-108.png, > image-2020-08-18-16-03-33-370.png > > > Thread-1 > !image-2020-08-18-15-59-46-108.png! > Thread-2 > !image-2020-08-18-16-03-33-370.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4039) Support Local dictionary for presto complex datatypes
[ https://issues.apache.org/jira/browse/CARBONDATA-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4039: --- Description: Support Local dictionary for presto complex datatypes - Presto complex datatypes - array and struct only. [https://github.com/apache/carbondata/pull/3987] was: Support Local dictionary for presto complex datatypes - Local dictionary support for presto complex datatypes - array and struct only. https://github.com/apache/carbondata/pull/3987 > Support Local dictionary for presto complex datatypes > - > > Key: CARBONDATA-4039 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4039 > Project: CarbonData > Issue Type: New Feature > Components: core, presto-integration >Reporter: Akshay >Priority: Major > > Support Local dictionary for presto complex datatypes - > Presto complex datatypes - array and struct only. > [https://github.com/apache/carbondata/pull/3987] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4039) Support Local dictionary for presto complex datatypes
Akshay created CARBONDATA-4039: -- Summary: Support Local dictionary for presto complex datatypes Key: CARBONDATA-4039 URL: https://issues.apache.org/jira/browse/CARBONDATA-4039 Project: CarbonData Issue Type: Improvement Components: core, presto-integration Reporter: Akshay Support Local dictionary for presto complex datatypes - Local dictionary support for presto complex datatypes - array and struct only. https://github.com/apache/carbondata/pull/3987 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4039) Support Local dictionary for presto complex datatypes
[ https://issues.apache.org/jira/browse/CARBONDATA-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4039: --- Issue Type: New Feature (was: Improvement) > Support Local dictionary for presto complex datatypes > - > > Key: CARBONDATA-4039 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4039 > Project: CarbonData > Issue Type: New Feature > Components: core, presto-integration >Reporter: Akshay >Priority: Major > > Support Local dictionary for presto complex datatypes - > Local dictionary support for presto complex datatypes - array and struct only. > https://github.com/apache/carbondata/pull/3987 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3830) Presto read support for complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3830: --- Attachment: Presto Read Support.pdf > Presto read support for complex columns > --- > > Key: CARBONDATA-3830 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3830 > Project: CarbonData > Issue Type: New Feature > Components: core, presto-integration >Reporter: Akshay >Assignee: Ajantha Bhat >Priority: Minor > Fix For: 2.1.0 > > Attachments: Presto Read Support.pdf > > Time Spent: 33h 40m > Remaining Estimate: 0h > > This feature is to enable Presto to read complex columns from carbon file. > Complex columns include - array, map and struct. > This design document handles only for array and struct type. > Map type will be handled later. > > PR - [https://github.com/apache/carbondata/pull/3887] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (CARBONDATA-3850) Presto read support for Array datatype
[ https://issues.apache.org/jira/browse/CARBONDATA-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay closed CARBONDATA-3850. -- Resolution: Fixed > Presto read support for Array datatype > -- > > Key: CARBONDATA-3850 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3850 > Project: CarbonData > Issue Type: Sub-task > Components: core, presto-integration >Reporter: Akshay >Priority: Minor > Attachments: Presto Read Support.pdf > > > Handles both single-level and multi-level reading of array data type form > Presto. > Attached is the design doc for the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3830) Presto read support for complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3830: --- Attachment: (was: Presto Read Support.pdf) > Presto read support for complex columns > --- > > Key: CARBONDATA-3830 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3830 > Project: CarbonData > Issue Type: New Feature > Components: core, presto-integration >Reporter: Akshay >Assignee: Ajantha Bhat >Priority: Minor > Fix For: 2.1.0 > > Time Spent: 33h 40m > Remaining Estimate: 0h > > This feature is to enable Presto to read complex columns from carbon file. > Complex columns include - array, map and struct. > This design document handles only for array and struct type. > Map type will be handled later. > > PR - [https://github.com/apache/carbondata/pull/3887] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CARBONDATA-3905) When there are many segment files presto query fail
[
https://issues.apache.org/jira/browse/CARBONDATA-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210030#comment-17210030
]
Akshay commented on CARBONDATA-3905:
Please provide the complete code to generate the segments
> When there are many segment files presto query fail
> ---
>
> Key: CARBONDATA-3905
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3905
> Project: CarbonData
> Issue Type: Bug
> Components: presto-integration
>Affects Versions: 2.0.0
>Reporter: XiaoWen
>Priority: Major
>
> test case1
> insert data in:
> {code:java}
> df.writeStream.foreachBatch{ (batchDF: DataFrame, batchId: Long) => {
> ...
> val cond = $"B.id".isin(df.select(col = "id").as[Int].collect: _*)
> target.as("A")
> .merge(df.as("B"), "A.id = B.id")
> .whenMatched(cond)
> .updateExpr(Map("name" -> "B.name", "city" -> "B.city", "age" ->
> "B.age"))
> .whenNotMatched(cond)
> .insertExpr(Map("id" -> "B.id", "name" -> "B.name", "city" -> "B.city",
> "age" -> "B.age"))
> .execute()
> ...
> }).outputMode("update").trigger(Trigger.ProcessingTime("3600
> seconds")).start()
> {code}
> a lot of segment files will be generated after a few hours
> when i try to use presto to query.
> single condition can be queried, but cannot be queried when there are
> multiple conditions.
> select name from test_table // ok
> select name from test_table where name = 'joe' // ok
> select name from test_table where name='joe' AND age > 25;// query failed
> select name from test_table where name='joe' AND age > 25 AND city
> ='shenzhen';// query failed
> i have also tried to compact 'major' the segment files to reduce the segment
> quantity, and I still cannot query successfully.
> presto server logs
> java.lang.IllegalArgumentException: Invalid position 0 in block with 0
> positions
> at io.prestosql.spi.block.BlockUtil.checkValidPosition(BlockUtil.java:62)
> at
> io.prestosql.spi.block.AbstractVariableWidthBlock.checkReadablePosition(AbstractVariableWidthBlock.java:160)
> at
> io.prestosql.spi.block.AbstractVariableWidthBlock.isNull(AbstractVariableWidthBlock.java:154)
> at io.prestosql.spi.block.LazyBlock.isNull(LazyBlock.java:248)
> at io.prestosql.$gen.PageFilter_20200703_084817_965.filter(Unknown Source)
> at io.prestosql.$gen.PageFilter_20200703_084817_965.filter(Unknown Source)
> at
> io.prestosql.operator.project.PageProcessor.createWorkProcessor(PageProcessor.java:115)
> at
> io.prestosql.operator.ScanFilterAndProjectOperator$SplitToPages.lambda$processPageSource$1(ScanFilterAndProjectOperator.java:254)
> at
> io.prestosql.operator.WorkProcessorUtils.lambda$flatMap$4(WorkProcessorUtils.java:246)
> at
> io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:320)
> at
> io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:373)
> at
> io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:307)
> at
> io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:373)
> at
> io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:307)
> at
> io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:373)
> at
> io.prestosql.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:221)
> at
> io.prestosql.operator.WorkProcessorUtils.lambda$processStateMonitor$2(WorkProcessorUtils.java:200)
> at
> io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:373)
> at
> io.prestosql.operator.WorkProcessorUtils.lambda$flatten$6(WorkProcessorUtils.java:278)
> at
> io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:320)
> at
> io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:373)
> at
> io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:307)
> at
> io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:373)
> at
> io.prestosql.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:221)
> at
> io.prestosql.operator.WorkProcessorUtils.lambda$processStateMonitor$2(WorkProcessorUtils.java:200)
> at
> io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:373)
> at
> io.prestosql.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:221)
> at
> io.prestosql.operator.WorkProcessorUtils.lambda$finishWhen$3(WorkProcessorUtils.java:215)
> at
> io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:373)
> at
> io.prestosql.operator.WorkProcessorSourceOperatorAdapter.getOu
[jira] [Closed] (CARBONDATA-3921) SI load fails with unable to get filestatus error in concurrent scenario
[ https://issues.apache.org/jira/browse/CARBONDATA-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay closed CARBONDATA-3921. -- Resolution: Fixed > SI load fails with unable to get filestatus error in concurrent scenario > > > Key: CARBONDATA-3921 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3921 > Project: CarbonData > Issue Type: Bug >Reporter: Akshay >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > The clean up mechanism in SI finally block causes to delete the ongoing load > files which caused the issue. > Solution - Remove calling of the API > TableProcessingOperations.deletePartialLoadDataIfExist, as it's not required > for the scenario and clean up happens with clean files and > deleteLoadsAndUpdateMetadata API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CARBONDATA-3872) IndexOutOfBoundsException in ResizableArray
[ https://issues.apache.org/jira/browse/CARBONDATA-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209293#comment-17209293 ] Akshay commented on CARBONDATA-3872: Please provide the full query that is causing this failure. > IndexOutOfBoundsException in ResizableArray > --- > > Key: CARBONDATA-3872 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3872 > Project: CarbonData > Issue Type: Bug > Components: sql >Affects Versions: 1.5.1 >Reporter: SeaAndHill >Priority: Critical > Attachments: carbondata.png > > > carbondata 在通过in 查询时，in 是子查询，运行报数组越界， 对应spark 版本是2.2.1， hadoop 版本是 2.7.2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (CARBONDATA-3872) IndexOutOfBoundsException in ResizableArray
[ https://issues.apache.org/jira/browse/CARBONDATA-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3872: --- Comment: was deleted (was: Please provide the query that is failing.) > IndexOutOfBoundsException in ResizableArray > --- > > Key: CARBONDATA-3872 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3872 > Project: CarbonData > Issue Type: Bug > Components: sql >Affects Versions: 1.5.1 >Reporter: SeaAndHill >Priority: Critical > Attachments: carbondata.png > > > carbondata 在通过in 查询时，in 是子查询，运行报数组越界， 对应spark 版本是2.2.1， hadoop 版本是 2.7.2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CARBONDATA-3872) IndexOutOfBoundsException in ResizableArray
[ https://issues.apache.org/jira/browse/CARBONDATA-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209292#comment-17209292 ] Akshay commented on CARBONDATA-3872: Please provide the query that is failing. > IndexOutOfBoundsException in ResizableArray > --- > > Key: CARBONDATA-3872 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3872 > Project: CarbonData > Issue Type: Bug > Components: sql >Affects Versions: 1.5.1 >Reporter: SeaAndHill >Priority: Critical > Attachments: carbondata.png > > > carbondata 在通过in 查询时，in 是子查询，运行报数组越界， 对应spark 版本是2.2.1， hadoop 版本是 2.7.2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4004) Wrong result in Presto select query after executing update
[ https://issues.apache.org/jira/browse/CARBONDATA-4004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4004: --- Summary: Wrong result in Presto select query after executing update (was: Wrong result in Presto select query in case of inverted index) > Wrong result in Presto select query after executing update > -- > > Key: CARBONDATA-4004 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4004 > Project: CarbonData > Issue Type: Bug > Components: core, presto-integration >Reporter: Akshay >Priority: Major > > Presto select query after update operation with inverted index enabled always > returns 0 rows as result. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4004) Wrong result in Presto select query after executing update
[ https://issues.apache.org/jira/browse/CARBONDATA-4004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4004: --- Description: Presto select query after update operation returns different number of rows. (was: Presto select query after update operation with inverted index enabled always returns 0 rows as result.) > Wrong result in Presto select query after executing update > -- > > Key: CARBONDATA-4004 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4004 > Project: CarbonData > Issue Type: Bug > Components: core, presto-integration >Reporter: Akshay >Priority: Major > > Presto select query after update operation returns different number of rows. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3830) Presto read support for complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3830: --- Description: This feature is to enable Presto to read complex columns from carbon file. Complex columns include - array, map and struct. This design document handles only for array type. Map and Struct types will be handled later. PR - https://github.com/apache/carbondata/pull/3887 was: This feature is to enable Presto to read complex columns from carbon file. Complex columns include - array, map and struct. This design document handles only for array type. Map and Struct types will be handled later. PR - [https://github.com/apache/carbondata/pull/3773] > Presto read support for complex columns > --- > > Key: CARBONDATA-3830 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3830 > Project: CarbonData > Issue Type: New Feature > Components: core, presto-integration >Reporter: Akshay >Assignee: Ajantha Bhat >Priority: Minor > Fix For: 2.1.0 > > Attachments: Presto Read Support.pdf > > Time Spent: 33h 40m > Remaining Estimate: 0h > > This feature is to enable Presto to read complex columns from carbon file. > Complex columns include - array, map and struct. > This design document handles only for array type. > Map and Struct types will be handled later. > > PR - https://github.com/apache/carbondata/pull/3887 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3830) Presto read support for complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3830: --- Description: This feature is to enable Presto to read complex columns from carbon file. Complex columns include - array, map and struct. This design document handles only for array and struct type. Map type will be handled later. PR - [https://github.com/apache/carbondata/pull/3887] was: This feature is to enable Presto to read complex columns from carbon file. Complex columns include - array, map and struct. This design document handles only for array type. Map and Struct types will be handled later. PR - https://github.com/apache/carbondata/pull/3887 > Presto read support for complex columns > --- > > Key: CARBONDATA-3830 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3830 > Project: CarbonData > Issue Type: New Feature > Components: core, presto-integration >Reporter: Akshay >Assignee: Ajantha Bhat >Priority: Minor > Fix For: 2.1.0 > > Attachments: Presto Read Support.pdf > > Time Spent: 33h 40m > Remaining Estimate: 0h > > This feature is to enable Presto to read complex columns from carbon file. > Complex columns include - array, map and struct. > This design document handles only for array and struct type. > Map type will be handled later. > > PR - [https://github.com/apache/carbondata/pull/3887] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4004) Wrong result in Presto select query in case of inverted index
Akshay created CARBONDATA-4004: -- Summary: Wrong result in Presto select query in case of inverted index Key: CARBONDATA-4004 URL: https://issues.apache.org/jira/browse/CARBONDATA-4004 Project: CarbonData Issue Type: Bug Components: core, presto-integration Reporter: Akshay Presto select query after update operation with inverted index enabled always returns 0 rows as result. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4000) Presto select query failure right after firing update query
[ https://issues.apache.org/jira/browse/CARBONDATA-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4000: --- Description: The putObject() method implementation was missing due to which select query on *struct* complex data type after firing *update* queries were failing. Solution- Added implementation in ColumnarVectorWrapperDirectWithDeleteDelta. Earlier without this implementation control went to its parent class AbstractCarbonColumnarVector,java which was unsupported. was: The putObject() method implementation was missing due to which select query on *struct* complex data type after *update* queries were failing. Solution- Added implementation in ColumnarVectorWrapperDirectWithDeleteDelta. Earlier without this implementation control went to its parent class AbstractCarbonColumnarVector,java which was unsupported. > Presto select query failure right after firing update query > --- > > Key: CARBONDATA-4000 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4000 > Project: CarbonData > Issue Type: Bug > Components: core >Reporter: Akshay >Priority: Major > > The putObject() method implementation was missing due to which select query > on *struct* complex data type after firing *update* queries were failing. > Solution- > Added implementation in ColumnarVectorWrapperDirectWithDeleteDelta. Earlier > without this implementation control went to its parent class > AbstractCarbonColumnarVector,java which was unsupported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4000) Presto select query failure right after firing update query
[ https://issues.apache.org/jira/browse/CARBONDATA-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-4000: --- Description: The putObject() method implementation was missing due to which select query on *struct* complex data type after *update* queries were failing. Solution- Added implementation in ColumnarVectorWrapperDirectWithDeleteDelta. Earlier without this implementation control went to its parent class AbstractCarbonColumnarVector,java which was unsupported. > Presto select query failure right after firing update query > --- > > Key: CARBONDATA-4000 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4000 > Project: CarbonData > Issue Type: Bug > Components: core >Reporter: Akshay >Priority: Major > > The putObject() method implementation was missing due to which select query > on *struct* complex data type after *update* queries were failing. > > Solution- > Added implementation in ColumnarVectorWrapperDirectWithDeleteDelta. Earlier > without this implementation control went to its parent class > AbstractCarbonColumnarVector,java which was unsupported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4000) Presto select query failure right after firing update query
Akshay created CARBONDATA-4000: -- Summary: Presto select query failure right after firing update query Key: CARBONDATA-4000 URL: https://issues.apache.org/jira/browse/CARBONDATA-4000 Project: CarbonData Issue Type: Bug Components: core Reporter: Akshay -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3981) Presto filter check on binary datatype
[
https://issues.apache.org/jira/browse/CARBONDATA-3981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akshay updated CARBONDATA-3981:
---
Description:
Due to the absence of binary datatype check, there was a problem during object
serialisation in presto filter queries.
"select * from table where bin = cast('abc' as varbinary)" threw - error during
serialisation.
So have added required check in prestoFIlterUtil.java
was:
Due to the absence of binary datatype check, there was a problem during object
serialisation in presto filter queries.
"select * from table where bin = cast('abc' as varbinary)" threw - error during
serialisation.
> Presto filter check on binary datatype
> --
>
> Key: CARBONDATA-3981
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3981
> Project: CarbonData
> Issue Type: Bug
> Components: presto-integration
>Reporter: Akshay
>Priority: Major
>
> Due to the absence of binary datatype check, there was a problem during
> object serialisation in presto filter queries.
> "select * from table where bin = cast('abc' as varbinary)" threw - error
> during serialisation.
> So have added required check in prestoFIlterUtil.java
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CARBONDATA-3981) Presto filter check on binary datatype
Akshay created CARBONDATA-3981:
--
Summary: Presto filter check on binary datatype
Key: CARBONDATA-3981
URL: https://issues.apache.org/jira/browse/CARBONDATA-3981
Project: CarbonData
Issue Type: Bug
Components: presto-integration
Reporter: Akshay
Due to the absence of binary datatype check, there was a problem during object
serialisation in presto filter queries.
"select * from table where bin = cast('abc' as varbinary)" threw - error during
serialisation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3968) Hive read complex types issues
[ https://issues.apache.org/jira/browse/CARBONDATA-3968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3968: --- Description: # Issues in reading array/map/struct of byte, varchar and decimal types. # Map of primitive type with only one row inserted has issues. was: # Issues in reading of byte, varchar and decimal types. # Map of primitive type with only one row inserted has issues. > Hive read complex types issues > -- > > Key: CARBONDATA-3968 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3968 > Project: CarbonData > Issue Type: Bug > Components: hive-integration >Reporter: Akshay >Priority: Major > > # Issues in reading array/map/struct of byte, varchar and decimal types. > # Map of primitive type with only one row inserted has issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3968) Hive read complex types issues
Akshay created CARBONDATA-3968: -- Summary: Hive read complex types issues Key: CARBONDATA-3968 URL: https://issues.apache.org/jira/browse/CARBONDATA-3968 Project: CarbonData Issue Type: Bug Components: hive-integration Reporter: Akshay # Issues in reading of byte, varchar and decimal types. # Map of primitive type with only one row inserted has issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3939) Exception added for index creation on long string columns
Akshay created CARBONDATA-3939: -- Summary: Exception added for index creation on long string columns Key: CARBONDATA-3939 URL: https://issues.apache.org/jira/browse/CARBONDATA-3939 Project: CarbonData Issue Type: Bug Reporter: Akshay Index creation for long string columns are not yet supported. User understandable exceptions are thrown if user tries to create the same. https://github.com/apache/carbondata/pull/3869 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3921) SI load fails with unable to get filestatus error in concurrent scenario
[ https://issues.apache.org/jira/browse/CARBONDATA-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3921: --- Description: The clean up mechanism in SI finally block causes to delete the ongoing load files which caused the issue. Solution - Remove calling of the API TableProcessingOperations.deletePartialLoadDataIfExist, as it's not required for the scenario and clean up happens with clean files and deleteLoadsAndUpdateMetadata API. was: The clean up mechanism in SI finally block causes to delete the ongoing load files which caused the issue. Solution - Remove calling of the API TableProcessingOperations.deletePartialLoadDataIfExist, as it's not required for he scenario and clean up happens with clean files and deleteLoadsAndUpdateMetadata API. > SI load fails with unable to get filestatus error in concurrent scenario > > > Key: CARBONDATA-3921 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3921 > Project: CarbonData > Issue Type: Bug >Reporter: Akshay >Priority: Major > > The clean up mechanism in SI finally block causes to delete the ongoing load > files which caused the issue. > Solution - Remove calling of the API > TableProcessingOperations.deletePartialLoadDataIfExist, as it's not required > for the scenario and clean up happens with clean files and > deleteLoadsAndUpdateMetadata API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3921) SI load fails with unable to get filestatus error in concurrent scenario
Akshay created CARBONDATA-3921: -- Summary: SI load fails with unable to get filestatus error in concurrent scenario Key: CARBONDATA-3921 URL: https://issues.apache.org/jira/browse/CARBONDATA-3921 Project: CarbonData Issue Type: Bug Reporter: Akshay The clean up mechanism in SI finally block causes to delete the ongoing load files which caused the issue. Solution - Remove calling of the API TableProcessingOperations.deletePartialLoadDataIfExist, as it's not required for he scenario and clean up happens with clean files and deleteLoadsAndUpdateMetadata API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3850) Presto read support for Array datatype
[ https://issues.apache.org/jira/browse/CARBONDATA-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3850: --- Attachment: Presto Read Support.pdf > Presto read support for Array datatype > -- > > Key: CARBONDATA-3850 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3850 > Project: CarbonData > Issue Type: Sub-task > Components: core, presto-integration >Reporter: Akshay >Priority: Minor > Attachments: Presto Read Support.pdf > > > Handles both single-level and multi-level reading of array data type form > Presto. > Attached is the design doc for the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3850) Presto read support for Array datatype
[ https://issues.apache.org/jira/browse/CARBONDATA-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3850: --- Attachment: (was: Presto Read Support.pdf) > Presto read support for Array datatype > -- > > Key: CARBONDATA-3850 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3850 > Project: CarbonData > Issue Type: Sub-task > Components: core, presto-integration >Reporter: Akshay >Priority: Minor > Attachments: Presto Read Support.pdf > > > Handles both single-level and multi-level reading of array data type form > Presto. > Attached is the design doc for the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3830) Presto read support for complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3830: --- Attachment: Presto Read Support.pdf > Presto read support for complex columns > --- > > Key: CARBONDATA-3830 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3830 > Project: CarbonData > Issue Type: New Feature > Components: core, presto-integration >Reporter: Akshay >Priority: Minor > Attachments: Presto Read Support.pdf > > Time Spent: 5h 20m > Remaining Estimate: 0h > > This feature is to enable Presto to read complex columns from carbon file. > Complex columns include - array, map and struct. > This design document handles only for array type. > Map and Struct types will be handled later. > > PR - [https://github.com/apache/carbondata/pull/3773] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3830) Presto read support for complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3830: --- Attachment: (was: Presto Read Support (1).pdf) > Presto read support for complex columns > --- > > Key: CARBONDATA-3830 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3830 > Project: CarbonData > Issue Type: New Feature > Components: core, presto-integration >Reporter: Akshay >Priority: Minor > Time Spent: 5h 20m > Remaining Estimate: 0h > > This feature is to enable Presto to read complex columns from carbon file. > Complex columns include - array, map and struct. > This design document handles only for array type. > Map and Struct types will be handled later. > > PR - [https://github.com/apache/carbondata/pull/3773] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3830) Presto read support for complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3830: --- Attachment: Presto Read Support (1).pdf > Presto read support for complex columns > --- > > Key: CARBONDATA-3830 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3830 > Project: CarbonData > Issue Type: New Feature > Components: core, presto-integration >Reporter: Akshay >Priority: Minor > Attachments: Presto Read Support (1).pdf > > Time Spent: 1h 40m > Remaining Estimate: 0h > > This feature is to enable Presto to read complex columns from carbon file. > Complex columns include - array, map and struct. > This design document handles only for array type. > Map and Struct types will be handled later. > > PR - [https://github.com/apache/carbondata/pull/3773] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3830) Presto read support for complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3830: --- Attachment: (was: Presto Read Support.pdf) > Presto read support for complex columns > --- > > Key: CARBONDATA-3830 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3830 > Project: CarbonData > Issue Type: New Feature > Components: core, presto-integration >Reporter: Akshay >Priority: Minor > Time Spent: 1h 40m > Remaining Estimate: 0h > > This feature is to enable Presto to read complex columns from carbon file. > Complex columns include - array, map and struct. > This design document handles only for array type. > Map and Struct types will be handled later. > > PR - [https://github.com/apache/carbondata/pull/3773] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3850) Presto read support for Array datatype
[ https://issues.apache.org/jira/browse/CARBONDATA-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3850: --- Component/s: presto-integration > Presto read support for Array datatype > -- > > Key: CARBONDATA-3850 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3850 > Project: CarbonData > Issue Type: Sub-task > Components: core, presto-integration >Reporter: Akshay >Priority: Minor > Attachments: Presto Read Support.pdf > > > Handles both single-level and multi-level reading of array data type form > Presto. > Attached is the design doc for the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3850) Presto read support for Array datatype
[ https://issues.apache.org/jira/browse/CARBONDATA-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3850: --- Attachment: Presto Read Support.pdf > Presto read support for Array datatype > -- > > Key: CARBONDATA-3850 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3850 > Project: CarbonData > Issue Type: Sub-task > Components: core >Reporter: Akshay >Priority: Minor > Attachments: Presto Read Support.pdf > > > Handles both single-level and multi-level reading of array data type form > Presto. > Attached is the design doc for the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3850) Presto read support for Array datatype
Akshay created CARBONDATA-3850: -- Summary: Presto read support for Array datatype Key: CARBONDATA-3850 URL: https://issues.apache.org/jira/browse/CARBONDATA-3850 Project: CarbonData Issue Type: Sub-task Components: core Reporter: Akshay Handles both single-level and multi-level reading of array data type form Presto. Attached is the design doc for the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3830) Presto read support for complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3830: --- Attachment: Presto Read Support.pdf > Presto read support for complex columns > --- > > Key: CARBONDATA-3830 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3830 > Project: CarbonData > Issue Type: New Feature > Components: core, presto-integration >Reporter: Akshay >Priority: Minor > Attachments: Presto Read Support.pdf > > Time Spent: 1h 40m > Remaining Estimate: 0h > > This feature is to enable Presto to read complex columns from carbon file. > Complex columns include - array, map and struct. > This design document handles only for array type. > Map and Struct types will be handled later. > > PR - [https://github.com/apache/carbondata/pull/3773] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3830) Presto read support for complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3830: --- Attachment: (was: Presto Read Support.pdf) > Presto read support for complex columns > --- > > Key: CARBONDATA-3830 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3830 > Project: CarbonData > Issue Type: New Feature > Components: core, presto-integration >Reporter: Akshay >Priority: Minor > Attachments: Presto Read Support.pdf > > Time Spent: 1h 40m > Remaining Estimate: 0h > > This feature is to enable Presto to read complex columns from carbon file. > Complex columns include - array, map and struct. > This design document handles only for array type. > Map and Struct types will be handled later. > > PR - [https://github.com/apache/carbondata/pull/3773] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3830) Presto read support for complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3830: --- Description: This feature is to enable Presto to read complex columns from carbon file. Complex columns include - array, map and struct. This design document handles only for array type. Map and Struct types will be handled later. PR - [https://github.com/apache/carbondata/pull/3773] was: This feature is to enable Presto to read complex columns from carbon file. Complex columns include - array, map and struct. This design document handles only for array type. Map and Struct types will be handled later. > Presto read support for complex columns > --- > > Key: CARBONDATA-3830 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3830 > Project: CarbonData > Issue Type: New Feature > Components: core, presto-integration >Reporter: Akshay >Priority: Minor > Attachments: Presto Read Support.pdf > > Time Spent: 1h 40m > Remaining Estimate: 0h > > This feature is to enable Presto to read complex columns from carbon file. > Complex columns include - array, map and struct. > This design document handles only for array type. > Map and Struct types will be handled later. > > PR - [https://github.com/apache/carbondata/pull/3773] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3830) Presto read support for complex columns
Akshay created CARBONDATA-3830: -- Summary: Presto read support for complex columns Key: CARBONDATA-3830 URL: https://issues.apache.org/jira/browse/CARBONDATA-3830 Project: CarbonData Issue Type: New Feature Components: core, presto-integration Reporter: Akshay Attachments: Presto Read Support.pdf This feature is to enable Presto to read complex columns from carbon file. Complex columns include - array, map and struct. This design document handles only for array type. Map and Struct types will be handled later. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3782) Changes to reflect default behaviour in case of invalid configuration
[ https://issues.apache.org/jira/browse/CARBONDATA-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3782: --- Description: Invalid configuration values in carbon.properties causes operation failure instead of reflecting default configuration value behaviour. [https://github.com/apache/carbondata/pull/3722] was:Invalid configuration values in carbon.properties causes operation failure instead of reflecting default configuration value behaviour. > Changes to reflect default behaviour in case of invalid configuration > - > > Key: CARBONDATA-3782 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3782 > Project: CarbonData > Issue Type: Bug > Components: data-load >Reporter: Akshay >Priority: Minor > > Invalid configuration values in carbon.properties causes operation failure > instead of reflecting default configuration value behaviour. > [https://github.com/apache/carbondata/pull/3722] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3782) Changes to reflect default behaviour in case of invalid configuration
Akshay created CARBONDATA-3782: -- Summary: Changes to reflect default behaviour in case of invalid configuration Key: CARBONDATA-3782 URL: https://issues.apache.org/jira/browse/CARBONDATA-3782 Project: CarbonData Issue Type: Bug Components: data-load Reporter: Akshay Invalid configuration values in carbon.properties causes operation failure instead of reflecting default configuration value behaviour. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3716) Fixed spark 2.4 UT failures
Akshay created CARBONDATA-3716: -- Summary: Fixed spark 2.4 UT failures Key: CARBONDATA-3716 URL: https://issues.apache.org/jira/browse/CARBONDATA-3716 Project: CarbonData Issue Type: Test Reporter: Akshay -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3716) Fixed spark 2.4 UT failures
[ https://issues.apache.org/jira/browse/CARBONDATA-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3716: --- Description: Issue: a) In 2.4 subquery alias produces an output set with alias identifier(that is both database_name and table_name) which causes failures. b) In 2.4 a new variable was introduced "spark.sql.legacy.allowCreatingManagedTableUsingNonemptyLocation" which is by default set to "false". While refreshing table In 2.4 we need to set this variable to "true" in order to create table in non empty location. > Fixed spark 2.4 UT failures > > > Key: CARBONDATA-3716 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3716 > Project: CarbonData > Issue Type: Test >Reporter: Akshay >Priority: Minor > > Issue: > a) In 2.4 subquery alias produces an output set with alias identifier(that is > both database_name and table_name) which causes failures. > b) In 2.4 a new variable was introduced > "spark.sql.legacy.allowCreatingManagedTableUsingNonemptyLocation" which is by > default set to "false". > While refreshing table In 2.4 we need to set this variable to "true" in order > to create table in non empty location. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3623) Error due to timestamp data type confliction.
Akshay created CARBONDATA-3623: -- Summary: Error due to timestamp data type confliction. Key: CARBONDATA-3623 URL: https://issues.apache.org/jira/browse/CARBONDATA-3623 Project: CarbonData Issue Type: Bug Reporter: Akshay In Carbondata the timestamp is converted into long and stored in RDD. While collect(action) is applied on dataframe it gives an error due to the confliction between data types that is - long in carbonScanRDD(from which dataframe is created) and Timestamp in schema. -- This message was sent by Atlassian Jira (v8.3.4#803005)
