[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/3021 Thank you..let's use your pull request and close this...so that we can have a single.place to comment. ---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/3021 This commit breaks the cloudera 2.2.0cloudera2 ---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/3021 https://github.com/apache/carbondata/commit/b0733ecbf380d7956dee57a9048dd7537620744e ---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/3021 We get error from a commit related to altertableschema.. which is not there in cloudera spark ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/2161 @xubo245 yes will do ---
[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user chandrasaripaka commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r244182374
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/filesystem/AlluxioCarbonFile.java
---
@@ -90,21 +97,82 @@ public CarbonFile getParentFile() {
return null == parent ? null : new AlluxioCarbonFile(parent);
}
+ /**
+ * RenameForce of the fileName for the AlluxioFileSystem
Implementation.
+ * Involves by opening a {@link FSDataInputStream} from the existing
path and copy
+ * bytes to {@link FSDataOutputStream}.
+ *
+ * Close the output and input streams only after the files have been
written
+ * Also check for the existence of the changed path and then delete the
previous Path.
+ * The No of Bytes that can be read is controlled by {@literal
io.file.buffer.size},
+ * where the default value is 4096.
+ * @param changeToName
+ * @return
+ */
@Override
public boolean renameForce(String changeToName) {
-FileSystem fs;
+FileSystem fs = null;
+FSDataOutputStream fsdos = null;
+FSDataInputStream fsdis = null;
try {
- fs =
fileStatus.getPath().getFileSystem(FileFactory.getConfiguration());
- if (fs instanceof DistributedFileSystem) {
-((DistributedFileSystem) fs).rename(fileStatus.getPath(), new
Path(changeToName),
-org.apache.hadoop.fs.Options.Rename.OVERWRITE);
+ Path actualPath = fileStatus.getPath();
+ Path changedPath = new Path(changeToName);
+ fs = actualPath.getFileSystem(hadoopConf);
+ fsdos = fs.create(changedPath, true);
+ fsdis = fs.open(actualPath);
+ if (null != fsdis && null != fsdos) {
+IOUtils.copyBytes(fsdis, fsdos, hadoopConf, true);
return true;
- } else {
-return false;
}
+ return false;
} catch (IOException e) {
LOGGER.error("Exception occured: " + e.getMessage());
return false;
+} finally {
+ try {
+if (null != fsdis && null != fsdos) {
+ if (fs.exists(new Path(changeToName))) {
+fs.delete(fileStatus.getPath(), true);
--- End diff --
Reassigned the fileStatus to the changedPath
---
[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user chandrasaripaka commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r244182177
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/filesystem/AlluxioCarbonFile.java
---
@@ -90,21 +97,82 @@ public CarbonFile getParentFile() {
return null == parent ? null : new AlluxioCarbonFile(parent);
}
+ /**
+ * RenameForce of the fileName for the AlluxioFileSystem
Implementation.
+ * Involves by opening a {@link FSDataInputStream} from the existing
path and copy
+ * bytes to {@link FSDataOutputStream}.
+ *
+ * Close the output and input streams only after the files have been
written
+ * Also check for the existence of the changed path and then delete the
previous Path.
+ * The No of Bytes that can be read is controlled by {@literal
io.file.buffer.size},
+ * where the default value is 4096.
+ * @param changeToName
+ * @return
+ */
@Override
public boolean renameForce(String changeToName) {
-FileSystem fs;
+FileSystem fs = null;
+FSDataOutputStream fsdos = null;
+FSDataInputStream fsdis = null;
try {
- fs =
fileStatus.getPath().getFileSystem(FileFactory.getConfiguration());
- if (fs instanceof DistributedFileSystem) {
-((DistributedFileSystem) fs).rename(fileStatus.getPath(), new
Path(changeToName),
-org.apache.hadoop.fs.Options.Rename.OVERWRITE);
+ Path actualPath = fileStatus.getPath();
+ Path changedPath = new Path(changeToName);
+ fs = actualPath.getFileSystem(hadoopConf);
+ fsdos = fs.create(changedPath, true);
+ fsdis = fs.open(actualPath);
+ if (null != fsdis && null != fsdos) {
+IOUtils.copyBytes(fsdis, fsdos, hadoopConf, true);
return true;
--- End diff --
ThankYou Changed it
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/3021 @chenliang613 , I am ok with it..as in the corporate we still need to livw with it.. we can update the jira and close it. ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/2161 @ravipesala , @jackylk can we merge the PR? ---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/3021 @ravipesala , Yes you can let me know how you want to inherit..I think defining a interface for the wrapper would be a good fit.. In our local fork we did it using spark-2.2.0 normal thrift server, it is ok if we dont run thrift server from cloudera distribution, that works. ---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/3021 Please try this: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-sql_2.11/2.2.0.cloudera2/ ---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/3021 Please check the repo here. https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh5_maven_repo_514x.html#maven_5142 ---
[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user chandrasaripaka commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r243806487
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -960,7 +960,7 @@ private CarbonCommonConstants() {
* If set to GLOBAL_SORT, the sorting scope is bigger and one index tree
per task will be
* created, thus loading is slower but query is faster.
*/
- public static final String LOAD_SORT_SCOPE_DEFAULT = "LOCAL_SORT";
+ public static final String LOAD_SORT_SCOPE_DEFAULT = "NO_SORT";
--- End diff --
Actually thats the change coming from Master branch..when I rebased it.
Please confirm..and I have squashed into one pull request.
---
[GitHub] carbondata issue #3021: [CARBONDATA-3193] Cdh5.14.2 spark2.2.0 support
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/3021 @jackylk , @ravipesala Can somebody validate this against the CDH5.14.2 and could you please create a profile for build validation for CDH Libraries.. Please let me know where to add..if I need to trigger a different profile build. ---
[GitHub] carbondata pull request #3021: Cdh support
GitHub user chandrasaripaka opened a pull request: https://github.com/apache/carbondata/pull/3021 Cdh support Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ No] Any interfaces changed? - [ No] Any backward compatibility impacted? - [ Yes] Document update required? - [ Yes] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ No Large Changes] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. This is just only the way we fix for CDH5.14.2, and the Spark2.2.0, the way how parquet treats the data. You can merge this pull request into a Git repository by running: $ git pull https://github.com/DataHeaps/carbondata cdh-support Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/3021.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3021 commit 746657e7e47fc69a62dbf14b3a7449108b92342c Author: Chandrasekhar Saripaka Date: 2018-07-25T17:10:20Z Merge pull request #1 from apache/master merge back commit 0c4c5f5d43c779c62864ed7ab99cf9ef91b082f2 Author: Chandrasekhar Saripaka Date: 2018-11-23T12:24:19Z Merge pull request #3 from apache/master Merge Carbondata master commit 91d4ee54bfac7bad56dede27059896c766b5f42f Author: Chandrasekhar Saripaka Date: 2018-12-13T07:42:30Z Merge pull request #4 from apache/master Updating the fork master commit 834e702d0e2e8211987c465e36596d78028ea8b2 Author: Chandrasekhar Saripaka Date: 2018-12-14T13:36:26Z Merge pull request #5 from apache/master Merge Master to DataHeaps Master commit 2ebfdf3142ee3f22e7938d735ca5a88c117f0848 Author: Chandrasekhar Saripaka Date: 2018-12-21T08:51:13Z Merge pull request #6 from apache/master Merge Upstream Master commit 0c777259d54d3b99acd9d62b75cddde7e726afc1 Author: ChandraSaripaka Date: 2018-12-24T07:19:30Z CARBONDATA-3193: Build support for CDH 5.14.2 Spark 2.2.0 ---
[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user chandrasaripaka commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r243773875
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -960,7 +960,7 @@ private CarbonCommonConstants() {
* If set to GLOBAL_SORT, the sorting scope is bigger and one index tree
per task will be
* created, thus loading is slower but query is faster.
*/
- public static final String LOAD_SORT_SCOPE_DEFAULT = "LOCAL_SORT";
+ public static final String LOAD_SORT_SCOPE_DEFAULT = "NO_SORT";
--- End diff --
resolved
---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/2161 I am changing the implementation for the same to read and copy the data from file system and change the same as suggested by @ravipesala , I get the similar issue with the previous code..as we need a force rename here, we have to delete the file again..and do the previous run.. ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/2161 > @chandrasaripaka you can put the CarbonFile test with Alluxio Mini Cluster but make sure it does not go beyond a few seconds to finish the test as it impacts the build time @ravipesala , MiniClusters are multiprocess tests, they go beyond few seconds. Will check if I can make it in the timeline, ---
[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user chandrasaripaka commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r241983388
--- Diff:
core/src/main/java/org/apache/carbondata/core/locks/AlluxioFileLock.java ---
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.locks;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.datastore.filesystem.AlluxioCarbonFile;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier;
+import org.apache.carbondata.core.util.path.CarbonTablePath;
+
+import org.apache.log4j.Logger;
+
+/**
+ * This class is used to handle the S3 File locking.
+ * This is acheived using the concept of acquiring the data out stream
using Append option.
+ */
+public class AlluxioFileLock extends AbstractCarbonLock {
--- End diff --
Understand , will just extend from that.
---
[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user chandrasaripaka commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r241983377
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/filesystem/AlluxioCarbonFile.java
---
@@ -94,14 +93,9 @@ public CarbonFile getParentFile() {
public boolean renameForce(String changeToName) {
FileSystem fs;
try {
- fs =
fileStatus.getPath().getFileSystem(FileFactory.getConfiguration());
- if (fs instanceof DistributedFileSystem) {
-((DistributedFileSystem) fs).rename(fileStatus.getPath(), new
Path(changeToName),
-org.apache.hadoop.fs.Options.Rename.OVERWRITE);
-return true;
- } else {
-return false;
- }
+ fs = fileStatus.getPath().getFileSystem(hadoopConf);
+ fs.delete(new Path(changeToName), true);
+ return fs.rename(fileStatus.getPath(), new Path(changeToName));
--- End diff --
Actually no I mean..it checks for the existence of the file and renames
only ( else it will return false), but will not do a force rename. But I am
thinking we can use the same alluxio fs copy, programatically to copy the same
contents from one file to another file(changedName under the same directory).
---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/2161 @ravipesala , I want to put a CarbonFile Test based on Alluxio Mini Cluster, will that be ok ? ---
[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user chandrasaripaka commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r241980998
--- Diff:
core/src/main/java/org/apache/carbondata/core/locks/AlluxioFileLock.java ---
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.locks;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.datastore.filesystem.AlluxioCarbonFile;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier;
+import org.apache.carbondata.core.util.path.CarbonTablePath;
+
+import org.apache.log4j.Logger;
+
+/**
+ * This class is used to handle the S3 File locking.
+ * This is acheived using the concept of acquiring the data out stream
using Append option.
+ */
+public class AlluxioFileLock extends AbstractCarbonLock {
--- End diff --
We cannot use HDFSFileLock..the reason is we cannot go back to HDFS, for
file locking. I understand although the structure is copied from HDFS
File..when we run our jobs on alluxio, we may not be using HDFS at all, because
the default file system (fs.defaultFS) falls back to Alluxio..
---
[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user chandrasaripaka commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2161#discussion_r241980433 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java --- @@ -43,7 +44,7 @@ * LOGGER */ private static final Logger LOGGER = - LogServiceFactory.getLogService(FileFactory.class.getName()); + LogServiceFactory.getLogService(FileFactory.class.getName()); --- End diff -- Removing the unnecessary formats..thanks for bringing up this. ---
[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user chandrasaripaka commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r241980383
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/filesystem/AlluxioCarbonFile.java
---
@@ -94,14 +93,9 @@ public CarbonFile getParentFile() {
public boolean renameForce(String changeToName) {
FileSystem fs;
try {
- fs =
fileStatus.getPath().getFileSystem(FileFactory.getConfiguration());
- if (fs instanceof DistributedFileSystem) {
-((DistributedFileSystem) fs).rename(fileStatus.getPath(), new
Path(changeToName),
-org.apache.hadoop.fs.Options.Rename.OVERWRITE);
-return true;
- } else {
-return false;
- }
+ fs = fileStatus.getPath().getFileSystem(hadoopConf);
+ fs.delete(new Path(changeToName), true);
+ return fs.rename(fileStatus.getPath(), new Path(changeToName));
--- End diff --
I am removing the delete..as kindle refer to the rename operation in
Alluxio..it conflicts the way delete is present..which is handled in the
alluxio version from 1.7.1
---
[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user chandrasaripaka commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r241980353
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/filesystem/AbstractDFSCarbonFile.java
---
@@ -550,12 +550,10 @@ public DataOutputStream
getDataOutputStreamUsingAppend(String path, FileFactory.
if (null != fileStatus && fileStatus.isDirectory()) {
--- End diff --
Missed in rebase.. Resolving this now.
---
[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user chandrasaripaka commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r241939242
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java
---
@@ -432,7 +437,7 @@ public static long getDirectorySize(String filePath)
throws IOException {
case VIEWFS:
case S3:
Path path = new Path(filePath);
-FileSystem fs = path.getFileSystem(getConfiguration());
+FileSystem fs = path.getFileSystem(configuration);
--- End diff --
Resolved.
---
[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user chandrasaripaka commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2161#discussion_r241290215 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java --- @@ -111,14 +112,9 @@ public static DataInputStream getDataInputStream(String path, FileType fileType) return getDataInputStream(path, fileType, -1); } - public static DataInputStream getDataInputStream(String path, FileType fileType, --- End diff -- Oh thats a rebase mistake..I think need to rebase again. ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/2161 @xubo245 , so may I know if the PR can be merged or it needs some rework..we can attach some test logs. ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/2161 @xubo245 , please merge the PR, after a review ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/2161 > @chandrasaripaka Can you fix the CI error? @xubo245 , just committed please review and let me know. ---
[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user chandrasaripaka commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r238211698
--- Diff:
core/src/test/java/org/apache/carbondata/core/datastore/filesystem/AlluxioCarbonFileTest.java
---
@@ -108,12 +121,12 @@ public void testListFilesForNullListStatus() {
alluxioCarbonFile = new
AlluxioCarbonFile(fileStatusWithOutDirectoryPermission);
new MockUp() {
@Mock
-public FileSystem getFileSystem(Configuration conf) throws
IOException {
-return new DistributedFileSystem();
+public FileSystem get(FileSystemContext context) throws
IOException {
--- End diff --
Fixed the test in the latest push
---
[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user chandrasaripaka commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r200066306
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java
---
@@ -365,10 +366,11 @@ public static boolean createNewLockFile(String
filePath, FileType fileType) thro
public static String getUpdatedFilePath(String filePath, FileType
fileType) {
switch (fileType) {
case HDFS:
- case ALLUXIO:
case VIEWFS:
case S3:
return filePath;
+ case ALLUXIO:
+return StringUtils.containsAny(filePath,"alluxio")?filePath:
"alluxio://"+filePath;
--- End diff --
Alluxio does the path translation in the FileSystem , that s why we have
this issue.
https://github.com/Alluxio/alluxio/blob/master/core/client/hdfs/src/main/java/alluxio/hadoop/AbstractFileSystem.java
---
[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user chandrasaripaka commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r200064110
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/filesystem/AlluxioCarbonFile.java
---
@@ -105,18 +103,21 @@ public CarbonFile getParentFile() {
return null == parent ? null : new AlluxioCarbonFile(parent);
}
+ /**
+ * CARBON-2218 Adopting to {@link FileSystem} , in accordance
+ * with the AlluxioFileSystem Implementation.
+ * Implicit expection of the AlluxioFileSystem when using this method.
+ * If success, returns true else false when an Exception is raised.
+ * @param changetoName
+ * @return
+ */
@Override
public boolean renameForce(String changetoName) {
FileSystem fs;
try {
fs =
fileStatus.getPath().getFileSystem(FileFactory.getConfiguration());
- if (fs instanceof DistributedFileSystem) {
-((DistributedFileSystem) fs).rename(fileStatus.getPath(), new
Path(changetoName),
-org.apache.hadoop.fs.Options.Rename.OVERWRITE);
-return true;
- } else {
-return false;
- }
+ fs.delete(new Path(changetoName), true);
--- End diff --
Any update on this. Can you suggest on a alternative way.
---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/2161 @xubo245 how do we merge this ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/2161 @xubo2545 looks like there is a permission issue for locating the file to build for my I'd. Actually the build is passing. ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/2161 I guess the build passed, but at the time of locating the jar , it failed. Can somebody look into the issue with Spark2.1.0 CI build. ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user chandrasaripaka commented on the issue:
https://github.com/apache/carbondata/pull/2161
@CarbonDataQA FileNotFoundException means that the credentials Jenkins is
using is probably wrong. Or the user account does not have write access to the
repo. org.kohsuke.github.GHFileNotFoundException: {"message":"Not
Found","documentation_url":"https://developer.github.com/v3/repos/statuses/#create-a-status"}
at
---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/2161 @CarbonDataQA May I know if this has to be fixed from my side..as a part of the pull request, Kindly advise. @xubo245 Also, I dont have access to resolve the conflicts and recommit. Please advise. ---
[GitHub] carbondata issue #2161: Adding Alluxio Support CARBONDATA-2218
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/2161 @xubo245 CARBONDATA-2218 , which is the jira I raised before. ---
[GitHub] carbondata issue #2161: Adding Alluxio Supporty
Github user chandrasaripaka commented on the issue: https://github.com/apache/carbondata/pull/2161 @chenliang613 This PR is for supporting the Alluxio as a File System for supporting the CarbonFileFormat. Though there was a AlluxioCarbonFile in the past, this was failing with Alluxio. Since Alluxio does not inherit DistributedFileSystem, it has to be a HadoopFileSystem that will be the change, as technically it does not support the appends on the File. Also, adopting to a AlluxioLock, however one can also use the ZookeeperLock, and also navigate. ---
[GitHub] carbondata pull request #2161: Adding Alluxio Supporty
GitHub user chandrasaripaka opened a pull request: https://github.com/apache/carbondata/pull/2161 Adding Alluxio Supporty Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [No ] Any interfaces changed? - [No ] Any backward compatibility impacted? - [Yes ] Document update required? - [Partially ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? It is the support for alluxio that has been fixed. - How it is tested? Please attach test report. It is tested with external alluxio. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. Added Alluxio Support , as it is not reflecting the correct FileSystem. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/DataHeaps/carbondata alluxio-support Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2161.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2161 commit 35ce7550954fbc8d9211352b79876c1ee148b609 Author: pour Date: 2018-03-22T08:00:38Z Adding Alluxio Supporty ---
