[jira] [Commented] (CARBONDATA-283) Improve the test cases for concurrent scenarios
[ https://issues.apache.org/jira/browse/CARBONDATA-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15635994#comment-15635994 ] ASF GitHub Bot commented on CARBONDATA-283: --- Github user ManoharVanam commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/207#discussion_r86523287 --- Diff: processing/src/main/java/org/apache/carbondata/processing/util/CarbonTableStatusUtil.java --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.carbondata.processing.util; + +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Date; +import java.util.List; + +import org.apache.carbondata.common.logging.LogService; +import org.apache.carbondata.common.logging.LogServiceFactory; +import org.apache.carbondata.core.constants.CarbonCommonConstants; +import org.apache.carbondata.core.load.LoadMetadataDetails; + +/** + * This class contains all table status file utilities + */ +public final class CarbonTableStatusUtil { + private static final LogService LOGGER = + LogServiceFactory.getLogService(CarbonTableStatusUtil.class.getName()); + + private CarbonTableStatusUtil() { + + } + + /** + * updates table status details using latest metadata + * + * @param oldMetadata + * @param newMetadata + * @return + */ + + public static List updateLatestTableStatusDetails( --- End diff -- ok > Improve the test cases for concurrent scenarios > --- > > Key: CARBONDATA-283 > URL: https://issues.apache.org/jira/browse/CARBONDATA-283 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam >Priority: Minor > > Improve test cases for data retention concurrent scenarios -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-283) Improve the test cases for concurrent scenarios
[ https://issues.apache.org/jira/browse/CARBONDATA-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15635992#comment-15635992 ] ASF GitHub Bot commented on CARBONDATA-283: --- Github user ManoharVanam commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/207#discussion_r86523222 --- Diff: processing/src/main/java/org/apache/carbondata/processing/util/CarbonTableStatusUtil.java --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.carbondata.processing.util; + +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Date; +import java.util.List; + +import org.apache.carbondata.common.logging.LogService; +import org.apache.carbondata.common.logging.LogServiceFactory; +import org.apache.carbondata.core.constants.CarbonCommonConstants; +import org.apache.carbondata.core.load.LoadMetadataDetails; + +/** + * This class contains all table status file utilities + */ +public final class CarbonTableStatusUtil { + private static final LogService LOGGER = + LogServiceFactory.getLogService(CarbonTableStatusUtil.class.getName()); + + private CarbonTableStatusUtil() { + + } + + /** + * updates table status details using latest metadata + * + * @param oldMetadata + * @param newMetadata + * @return + */ + + public static List updateLatestTableStatusDetails( + LoadMetadataDetails[] oldMetadata, LoadMetadataDetails[] newMetadata) { + +List newListMetadata = +new ArrayList(Arrays.asList(newMetadata)); +for (LoadMetadataDetails oldSegment : oldMetadata) { + if (CarbonCommonConstants.MARKED_FOR_DELETE.equalsIgnoreCase(oldSegment.getLoadStatus())) { + updateSegmentMetadataDetails(newListMetadata.get(newListMetadata.indexOf(oldSegment))); + } +} +return newListMetadata; + } + + /** + * returns current time + * + * @return + */ + private static String readCurrentTime() { +SimpleDateFormat sdf = new SimpleDateFormat(CarbonCommonConstants.CARBON_TIMESTAMP); +String date = null; + +date = sdf.format(new Date()); + +return date; + } + + /** + * updates segment status and modificaton time details + * + * @param loadMetadata + */ + public static void updateSegmentMetadataDetails(LoadMetadataDetails loadMetadata) { --- End diff -- ok > Improve the test cases for concurrent scenarios > --- > > Key: CARBONDATA-283 > URL: https://issues.apache.org/jira/browse/CARBONDATA-283 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam >Priority: Minor > > Improve test cases for data retention concurrent scenarios -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-283) Improve the test cases for concurrent scenarios
[ https://issues.apache.org/jira/browse/CARBONDATA-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15633010#comment-15633010 ] ASF GitHub Bot commented on CARBONDATA-283: --- Github user gvramana commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/207#discussion_r86347851 --- Diff: processing/src/main/java/org/apache/carbondata/processing/util/CarbonTableStatusUtil.java --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.carbondata.processing.util; + +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Date; +import java.util.List; + +import org.apache.carbondata.common.logging.LogService; +import org.apache.carbondata.common.logging.LogServiceFactory; +import org.apache.carbondata.core.constants.CarbonCommonConstants; +import org.apache.carbondata.core.load.LoadMetadataDetails; + +/** + * This class contains all table status file utilities + */ +public final class CarbonTableStatusUtil { + private static final LogService LOGGER = + LogServiceFactory.getLogService(CarbonTableStatusUtil.class.getName()); + + private CarbonTableStatusUtil() { + + } + + /** + * updates table status details using latest metadata + * + * @param oldMetadata + * @param newMetadata + * @return + */ + + public static List updateLatestTableStatusDetails( + LoadMetadataDetails[] oldMetadata, LoadMetadataDetails[] newMetadata) { + +List newListMetadata = +new ArrayList(Arrays.asList(newMetadata)); +for (LoadMetadataDetails oldSegment : oldMetadata) { + if (CarbonCommonConstants.MARKED_FOR_DELETE.equalsIgnoreCase(oldSegment.getLoadStatus())) { + updateSegmentMetadataDetails(newListMetadata.get(newListMetadata.indexOf(oldSegment))); + } +} +return newListMetadata; + } + + /** + * returns current time + * + * @return + */ + private static String readCurrentTime() { +SimpleDateFormat sdf = new SimpleDateFormat(CarbonCommonConstants.CARBON_TIMESTAMP); +String date = null; + +date = sdf.format(new Date()); + +return date; + } + + /** + * updates segment status and modificaton time details + * + * @param loadMetadata + */ + public static void updateSegmentMetadataDetails(LoadMetadataDetails loadMetadata) { --- End diff -- Move these function to status Manager > Improve the test cases for concurrent scenarios > --- > > Key: CARBONDATA-283 > URL: https://issues.apache.org/jira/browse/CARBONDATA-283 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam >Priority: Minor > > Improve test cases for data retention concurrent scenarios -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-283) Improve the test cases for concurrent scenarios
[ https://issues.apache.org/jira/browse/CARBONDATA-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15568981#comment-15568981 ] ASF GitHub Bot commented on CARBONDATA-283: --- Github user jackylk commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/207#discussion_r83025827 --- Diff: processing/src/main/java/org/apache/carbondata/processing/util/CarbonTableStatusUtil.java --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.carbondata.processing.util; + +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Date; +import java.util.List; + +import org.apache.carbondata.common.logging.LogService; +import org.apache.carbondata.common.logging.LogServiceFactory; +import org.apache.carbondata.core.constants.CarbonCommonConstants; +import org.apache.carbondata.core.load.LoadMetadataDetails; + +/** + * This class contains all table status file utilities + */ +public final class CarbonTableStatusUtil { + private static final LogService LOGGER = + LogServiceFactory.getLogService(CarbonTableStatusUtil.class.getName()); + + private CarbonTableStatusUtil() { + + } + + /** + * updates table status details using latest metadata + * + * @param oldMetadata + * @param newMetadata + * @return + */ + + public static List updateLatestTableStatusDetails( + LoadMetadataDetails[] oldMetadata, LoadMetadataDetails[] newMetadata) { + +List newListMetadata = +new ArrayList(Arrays.asList(newMetadata)); +for (LoadMetadataDetails oldSegment : oldMetadata) { + if (CarbonCommonConstants.MARKED_FOR_DELETE.equalsIgnoreCase(oldSegment.getLoadStatus())) { + updateSegmentMetadataDetails(newListMetadata.get(newListMetadata.indexOf(oldSegment))); + } +} +return newListMetadata; + } + + /** + * returns current time + * + * @return + */ + private static String readCurrentTime() { +SimpleDateFormat sdf = new SimpleDateFormat(CarbonCommonConstants.CARBON_TIMESTAMP); +String date = null; + +date = sdf.format(new Date()); + +return date; + } + + /** + * updates segment status and modificaton time details + * + * @param loadMetadata + */ + public static void updateSegmentMetadataDetails(LoadMetadataDetails loadMetadata) { --- End diff -- Can you improve on the function name to depict the behavior of this function? > Improve the test cases for concurrent scenarios > --- > > Key: CARBONDATA-283 > URL: https://issues.apache.org/jira/browse/CARBONDATA-283 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam >Priority: Minor > > Improve test cases for data retention concurrent scenarios -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-283) Improve the test cases for concurrent scenarios
[ https://issues.apache.org/jira/browse/CARBONDATA-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15568980#comment-15568980 ] ASF GitHub Bot commented on CARBONDATA-283: --- Github user jackylk commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/207#discussion_r83026458 --- Diff: processing/src/main/java/org/apache/carbondata/processing/util/CarbonTableStatusUtil.java --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.carbondata.processing.util; + +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Date; +import java.util.List; + +import org.apache.carbondata.common.logging.LogService; +import org.apache.carbondata.common.logging.LogServiceFactory; +import org.apache.carbondata.core.constants.CarbonCommonConstants; +import org.apache.carbondata.core.load.LoadMetadataDetails; + +/** + * This class contains all table status file utilities + */ +public final class CarbonTableStatusUtil { + private static final LogService LOGGER = + LogServiceFactory.getLogService(CarbonTableStatusUtil.class.getName()); + + private CarbonTableStatusUtil() { + + } + + /** + * updates table status details using latest metadata + * + * @param oldMetadata + * @param newMetadata + * @return + */ + + public static List updateLatestTableStatusDetails( --- End diff -- I think these should not be utility functions, but should be member function of LoadMetadataDetails > Improve the test cases for concurrent scenarios > --- > > Key: CARBONDATA-283 > URL: https://issues.apache.org/jira/browse/CARBONDATA-283 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam >Priority: Minor > > Improve test cases for data retention concurrent scenarios -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-283) Improve the test cases for concurrent scenarios
[ https://issues.apache.org/jira/browse/CARBONDATA-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15542319#comment-15542319 ] ASF GitHub Bot commented on CARBONDATA-283: --- GitHub user ManoharVanam opened a pull request: https://github.com/apache/incubator-carbondata/pull/207 [CARBONDATA-283] VT enhancement for data retention concurrent scenarios why raise this PR ? 1 . Handling concurrency scenarios for retention feature 2. Added test cases for concurrent scenarios like load, delete segment by date/ id & clean files operations. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ManoharVanam/incubator-carbondata Concurrency Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/207.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #207 commit 1af96b9bfc258fba4110f39dd0a5ca5e119a8189 Author: Manohar Date: 2016-09-30T10:01:34Z Handle delete segment concurrent scenarios > Improve the test cases for concurrent scenarios > --- > > Key: CARBONDATA-283 > URL: https://issues.apache.org/jira/browse/CARBONDATA-283 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam >Priority: Minor > > Improve test cases for data retention concurrent scenarios -- This message was sent by Atlassian JIRA (v6.3.4#6332)